I took a brief look at the PR - it looks like “the prior approach” of sending to couchdb is still enabled, is that correct?
If so, it may be worthwhile to make the reference impl store to couchdb, and remove the activation persistence from controller/invoker? This would also imply that the “polling” that is controller would also need to be replaced? Thanks Tyson On Apr 17, 2018, at 12:02 PM, Rodric Rabbah <rod...@gmail.com<mailto:rod...@gmail.com>> wrote: It would be useful to provide a reference implementation for consuming this data. Can you also capture the goals in an issue (I scanned the PR quickly but there’s no corresponding issue). Further now I think there are multiple ways of recording/reporting some of the metrics (log markers which are largely silenced by default, kamon metrics, and now kafka). Is that right? I think we’ll need to also document these and a guide for when to use which - I caution that we are proliferating multiple ways of doing similar things with no consistency or articulated long term vision. -r On Apr 17, 2018, at 12:01 PM, Vadim Raskin <raskinva...@gmail.com<mailto:raskinva...@gmail.com>> wrote: Hi Chetan, Can you share some details on how this is currently being done with CouchDB. Do we have any analytics view configured which computes these numbers currently? To my knowledge we don't have the views that are shared anywhere in open repos. May be we also include a basic default implementation out of the box which collect aggregated stats using Kamon metrics already being used Not a bad idea, I'll consider sharing the peace of code after finishing the development (separate from the original PR), it might require some post-processing to strip away the ibm specific parts, so it might bide some time. regards, Vadim. On Tue, Apr 17, 2018 at 8:29 AM Chetan Mehrotra <chetan.mehro...@gmail.com<mailto:chetan.mehro...@gmail.com>> wrote: Hi Vadim, This looks helpful to get better insight in runtime operational stats! It has some advantages over to the prior approach (sending them to CouchDB). Can you share some details on how this is currently being done with CouchDB. Do we have any analytics view configured which computes these numbers currently? Now it would be possible to simply connect a custom micro service to Kafka and consume the activations in real-time. May be we also include a basic default implementation out of the box which collect aggregated stats using Kamon metrics already being used Chetan Mehrotra On Mon, Apr 16, 2018 at 9:51 PM, Vadim Raskin <raskinva...@gmail.com<mailto:raskinva...@gmail.com>> wrote: Hi everyone, I’ve just opened a PR that enables sending activation metadata to Kafka. It has some advantages over to the prior approach (sending them to CouchDB). Now it would be possible to simply connect a custom micro service to Kafka and consume the activations in real-time. Some of the use cases it might cover: activation metrics - collect the data and push them into a custom time-series database; user activity audit; activation analytics - potentially get some insights with KSQL. At the moment I’ve created a new kafka topic called events, which will include messages from Controllers and Invokers. It encompasses the following data collected from a single activation: concurrentActivations throttledActivations statusCode initTime waitTime duration kind Probably some more metadata will affiliate this list soon. Just wanted to give a short heads up here. The PR I mentioned: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk%2Fpull%2F3552&data=02%7C01%7Ctnorris%40adobe.com%7Cc908a394ad184dffaefe08d5a495c08c%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636595885445950512&sdata=seMpqJ1pUTYUt2a4XDUcBPTleDNCjxruf3HSjx%2BhDk4%3D&reserved=0 Thank you, Vadim.
