Hi OpenWhisk community, while doing our profiling tests on the controller we have found that the way we log and the logging per se make a huge impact on the controller thoughput. In a short, the more parallel activations you have, the more blocking threads waiting for the logger will be there. After doing several experiments we found a couple of action items: * replace the logging library (e.g. to logback), use an async appender and increase the size of the LogAppender buffer. * reduce the number of logs that we generate.
Since the first action item will only provide a partial relief and postpone the point when we start observing the degradation, reducing the number of logs seems to be a more plausible way to go. One of the first points I’d like to address are the metrics that we append to log messages in a form of markers. The initial idea is to send them directly to statsd or some other backend. For example Kamon library is a good candidate: http://kamon.io Here is the issue I’ve opened some time ago. https://github.com/apache/incubator-openwhisk/issues/2780 Do you have any suggestion/concerns regarding that? Thank you. regards, Vadim.
