Regarding where to use what. I think we need to distinguish between system and user metrics here. First are dedicated for platform operations and they are mostly component specific. For those the right approach would be to emit them using Kamon. Second are aimed be used by the users of the platform. Given that the actions could be scattered across the components and the multi-tenant nature of the platform - we need to aggregate the metrics at some place, potentially process them and send them to the tenant-specific backend. Thus - Kafka.
Eventually we might get rid of activation metadata in Cloudant, but at the moment I consider sending metadata to both Kafka and CouchDB as a less disruptive approach. Otherwise we need to overhaul too many corners. regards, Vadim. On Wed, Apr 18, 2018 at 7:21 AM Dascalita Dragos <ddrag...@gmail.com> wrote: > Continuing what Rodric was saying I'd also like to bring into this picture > Distributed Tracing from this older PR [1]. > A holistic approach of "what to use, when, and for what purpose" would be > useful. > > [1] - https://github.com/apache/incubator-openwhisk/pull/2282 > > On Tue, Apr 17, 2018 at 2:59 PM Tyson Norris <tnor...@adobe.com.invalid> > wrote: > > > I took a brief look at the PR - it looks like “the prior approach” of > > sending to couchdb is still enabled, is that correct? > > > > If so, it may be worthwhile to make the reference impl store to couchdb, > > and remove the activation persistence from controller/invoker? > > > > This would also imply that the “polling” that is controller would also > > need to be replaced? > > Thanks > > Tyson > > > > On Apr 17, 2018, at 12:02 PM, Rodric Rabbah <rod...@gmail.com<mailto: > > rod...@gmail.com>> wrote: > > > > It would be useful to provide a reference implementation for consuming > > this data. Can you also capture the goals in an issue (I scanned the PR > > quickly but there’s no corresponding issue). > > > > Further now I think there are multiple ways of recording/reporting some > of > > the metrics (log markers which are largely silenced by default, kamon > > metrics, and now kafka). Is that right? > > > > I think we’ll need to also document these and a guide for when to use > > which - I caution that we are proliferating multiple ways of doing > similar > > things with no consistency or articulated long term vision. > > > > -r > > > > On Apr 17, 2018, at 12:01 PM, Vadim Raskin <raskinva...@gmail.com > <mailto: > > raskinva...@gmail.com>> wrote: > > > > Hi Chetan, > > > > Can you share some details on how this is currently being done with > > CouchDB. Do we have any analytics view configured which computes these > > numbers currently? > > > > To my knowledge we don't have the views that are shared anywhere in open > > repos. > > > > May be we also include a basic default implementation out of the box > > which collect aggregated stats using Kamon metrics already being used > > > > Not a bad idea, I'll consider sharing the peace of code after finishing > the > > development (separate from the original PR), it might require some > > post-processing to strip away the ibm specific parts, so it might bide > some > > time. > > > > regards, > > Vadim. > > > > > > > > On Tue, Apr 17, 2018 at 8:29 AM Chetan Mehrotra < > chetan.mehro...@gmail.com > > <mailto:chetan.mehro...@gmail.com>> > > wrote: > > > > Hi Vadim, > > > > This looks helpful to get better insight in runtime operational stats! > > > > It has some advantages over to the prior approach (sending them to > > CouchDB). > > > > Can you share some details on how this is currently being done with > > CouchDB. Do we have any analytics view configured which computes these > > numbers currently? > > > > Now it would be possible to simply connect a custom micro service > > to Kafka and consume the activations in real-time. > > > > May be we also include a basic default implementation out of the box > > which collect aggregated stats using Kamon metrics already being used > > Chetan Mehrotra > > > > > > On Mon, Apr 16, 2018 at 9:51 PM, Vadim Raskin <raskinva...@gmail.com > > <mailto:raskinva...@gmail.com>> > > wrote: > > Hi everyone, > > > > > > I’ve just opened a PR that enables sending activation metadata to Kafka. > > It has some advantages over to the prior approach (sending them to > > CouchDB). Now it would be possible to simply connect a custom micro > > service > > to Kafka and consume the activations in real-time. Some of the use cases > > it > > might cover: activation metrics - collect the data and push them into a > > custom time-series database; user activity audit; activation analytics - > > potentially get some insights with KSQL. > > > > > > At the moment I’ve created a new kafka topic called events, which will > > include messages from Controllers and Invokers. It encompasses the > > following data collected from a single activation: > > > > > > concurrentActivations > > throttledActivations > > statusCode > > initTime > > waitTime > > duration > > kind > > > > > > Probably some more metadata will affiliate this list soon. > > > > > > Just wanted to give a short heads up here. The PR I mentioned: > > > > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk%2Fpull%2F3552&data=02%7C01%7Ctnorris%40adobe.com%7Cc908a394ad184dffaefe08d5a495c08c%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636595885445950512&sdata=seMpqJ1pUTYUt2a4XDUcBPTleDNCjxruf3HSjx%2BhDk4%3D&reserved=0 > > > > > > Thank you, > > > > > > Vadim. > > > > >
