The logs issue is mostly separate from the activation records.
RE activation records:
Can we handle these in same way as user events? Maybe exactly like user events,
as in use a single service to process both topics.
RE logging:
We deal with logs this way (collect from container via fluent), but the
problems we've seen are:
- forcing structure isn't easy to enforce consistently on all runtimes (we get
around this by mostly only supporting nodejs, java may also not be too bad, if
you support a few common logging libs)
- there are separate issues with concurrency support - for nodejs we use
cls-hooked to manage this context; using environment variables instead of a
different main signature is also a problem for concurrency, and also can be
worked around with cls-hooked. Not sure other operators care about this so
maybe it is not an issue except to be documented per runtime.
- many tests currently assume that the logs are immediately available, and this
cannot be the case with decoupled log collection (I've started work to add some
retries/delay tolerance, but it's incomplete)
Thanks
Tyson
On 6/21/19, 10:29 AM, "David P Grove" <gro...@us.ibm.com> wrote:
Rodric Rabbah <rod...@gmail.com> wrote on 06/20/2019 09:37:38 PM:
>
> Overflowing to Kafka (option b) is better. Actually I would dump all
> the activations there and have a separate process to drain that
> Kafka topic to the datastore or logstore.
I agree. Spilling to Kafka is desirable to avoid OOMs in the invoker.
> There is another approach of routing the logs directly to a logstore
> without going through the invoker at all. IBM may have experimented
> with this maybe someone else can comment on that.
In the Kubernetes world (especially with the KubernetesContainerFactory),
this is the only really good way of doing it.
To really do this well, our actions should be required to implement
structured logging. If every log line had the activationId and namespace
info in it, then the logs could stream from the container through an
efficient OpenWhisk-specific logging agent (I had prototyped an agent using
fluent bit last year) to the platform logging service.
If you don't have structured logging from the actions, you can try to
kludge this flow together in various ways but it gets messier.
--dave
