Hi Fazlan, Yes, would reduce some events 3 times, but still, I think in the new approach we need to send at least one event per gateway request to analytics server. Which means, based on the TPS values (load) of APIM server, we need to scale up analytics nodes as well. Basically, our requirement is calculating request/response/fault count based on different bucket values (group by). Therefore, based on the proposed model, if we calculate local statistics summary, then we don't need to scale analytics nodes based on APIM TPS, because only one event will be pushed per group by bucket per min.
Thanks, Sinthuja On Wed, Jun 20, 2018 at 10:23 AM Fazlan Nazeem <[email protected]> wrote: > Hi Sinthuja, > > There is an ongoing effort to combine request, response and execution time > event streams into a single stream and publish a single event instead of 3 > events to the Stream processor. This is targeted for the Q3 release and can > bring down the traffic by 3 times to the analytics server. > > Mail Subject: Moving APIM Analytics to SP > > On Tue, Jun 19, 2018 at 5:05 PM Sinthuja Rajendran <[email protected]> > wrote: > >> Hi, >> >> With my recent work with metrics and other monitoring systems, I'm >> thinking whether our model of sending everything and calculate in the >> analytics server side is correct. >> >> Basically IMHO, the majority of the product analytics use cases are >> statistics calculation. For example, in APIM, we are calculating API >> statistics, Subscription statistics, etc. And for this, we are sending >> events for every request/response/fault to analytics server, and analytics >> server is performing the actual statistics calculation on the events. >> Therefore in the high traffic scenario, there are a lot of events needs to >> be published to analytics server, and we are getting issues like "Event >> Queue Full" in the gateway. >> >> My proposal is, what if we calculate the statistics on the local node >> itself (similar to the edge analytics by running Siddhi within the >> products), and only send the minute statistics summary to analytics server >> to do the global (across all nodes) calculation. With this way, the traffic >> to analytics server will be reduced drastically as only the final >> calculated value for each group by combination will be reported. Therefore, >> the analytics server can focus on global summarization and other global >> monitoring aspects. >> >> Thanks, >> Sinthuja. >> >> -- >> *Sinthuja Rajendran* >> Senior Technical Lead >> WSO2, Inc.:http://wso2.com >> >> Blog: http://sinthu-rajan.blogspot.com/ >> Mobile: +94774273955 >> >> >> > > -- > Thanks & Regards, > > *Fazlan Nazeem* > Senior Software Engineer > WSO2 Inc > Mobile : +94772338839 > [email protected] > -- *Sinthuja Rajendran* Senior Technical Lead WSO2, Inc.:http://wso2.com Blog: http://sinthu-rajan.blogspot.com/ Mobile: +94774273955
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
