Hi,
we have showed in recent dev calls that the amount of logs the
controllers and invoker write out to the log files does harm the
performance under high load severely.
To reduce that burden we introduced the new metric infrastructure based
on Kamon
which is already delivered (but switched off by default).
Additionally we now pushed PR
https://github.com/apache/incubator-openwhisk/pull/2960
which reassigns the level of the remaining log messages.
Both changes reduce the amount of messages logged at INFO level (which
is the default) to about a 5th of the original size. Our tests show that
the performance under load improves by at least a factor of two.
In the PR we aimed for better error logging and to still ensure
maintainability of the system.
We would like to emphasize that with that change it is more important to
have a monitoring infrastructure in place (as a starter the document
‘metrics.md’ in the repo explains how to test the Kamon and statsd based
metrics in your local environment).
Please leave your feedback as comment to the PR or as answer to this
message.
Kind regards,
Vadim and Martin
