Hi Mohan/Sajith, Can you please explain what this particular scenario was in IS? Is this a counting scenario on a window? This would provide input for others who are writing real time analytics on CEP.
Cheers~ On Mon, Jun 27, 2016 at 5:50 PM, Mohanadarshan Vivekanandalingam < [email protected]> wrote: > > > On Mon, Jun 27, 2016 at 8:03 PM, Seshika Fernando <[email protected]> > wrote: > >> Iranga, >> The optimization applies to siddhi queries that were written for IS >> analytics. Other product analytics would have different queries so this is >> not applicable there. >> This is not a siddhi level change, rather an optimization of the >> previously written queries. >> > Yes, above per-second and per-minute design is applied to some other > analytics product as well but it is not generic and too specific for the > usecase and query as mentioned by Seshi.. > > Thanks, > Mohan > > >> Seshi >> On 27 Jun 2016 16:48, "Iranga Muthuthanthri" <[email protected]> wrote: >> >>> >>> >>> On Mon, Jun 27, 2016 at 4:35 PM, Mohanadarshan Vivekanandalingam < >>> [email protected]> wrote: >>> >>>> >>>> >>>> On Mon, Jun 27, 2016 at 4:30 PM, Seshika Fernando <[email protected]> >>>> wrote: >>>> >>>>> Hey saith, >>>>> >>>>> This is great. So when you removed the duplicate windows, there were >>>>> no OOM issues? >>>>> >>>> Yes, we have introduced per-second windows and eventually per-minute >>>> event count get reduced. Then, there will not be an OOM.. >>>> >>> >>> is this specific for IS-Analytics or common to all other 'product >>> analytics'. >>> >>> >>>> Latest number is also seems good.. SajithR, can share those numbers.. >>>> >>>> Thanks, >>>> Mohan >>>> >>>> >>>>> Seshi >>>>> On 24 Jun 2016 14:28, "Sajith Ravindra" <[email protected]> wrote: >>>>> >>>>>> Hi Malith, >>>>>> >>>>>> Thanks for the explanation. >>>>>>> >>>>>>> I would expect some variation in the throughput. The aim should be >>>>>>> minimize the variation in the throughput (while maintaining the >>>>>>> throughput >>>>>>> at its highest level). >>>>>>> >>>>>> Agreed. Actually, our expectation is to minimize the fluctuation and >>>>>> to increase the throughput. >>>>>> >>>>>>> >>>>>>> It is possible to measure the latency as well? >>>>>>> >>>>>> In IS-Analytics we don't generate any output events or alerts >>>>>> currently. Therefore, we don't calculate the latency. What we do is >>>>>> summarize events using siddhi queries and persist them in a DB and >>>>>> further >>>>>> summarize persisted data using Spark and then displayed through >>>>>> dashbaords. >>>>>> >>>>>> >>>>>> Thanks >>>>>> *,Sajith Ravindra* >>>>>> Senior Software Engineer >>>>>> WSO2 Inc.; http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> >>>>>> mobile: +94 77 2273550 >>>>>> blog: http://sajithr.blogspot.com/ >>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>>> >>>>>> On Fri, Jun 24, 2016 at 1:57 PM, Malith Jayasinghe <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Sajith, >>>>>>> >>>>>>> Thanks for the explanation. >>>>>>> >>>>>>> I would expect some variation in the throughput. The aim should be >>>>>>> minimize the variation in the throughput (while maintaining the >>>>>>> throughput >>>>>>> at its highest level). >>>>>>> >>>>>>> It is possible to measure the latency as well? >>>>>>> >>>>>>> regards >>>>>>> >>>>>>> Malith >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 23, 2016 at 11:13 PM, Sajith Ravindra <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Malith, >>>>>>>> >>>>>>>> The deployment is just a standalone DAS server, and we are planning >>>>>>>> to do a test for HA deployment in recent future. >>>>>>>> >>>>>>>> The workload is generated by a .csv data file which has 100K sample >>>>>>>> events, 10M events are generated by iterating through the same data >>>>>>>> set 100 >>>>>>>> times. But we keep increasing the timestamp. A simple thrift client is >>>>>>>> used >>>>>>>> to publish data. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> *,Sajith Ravindra* >>>>>>>> Senior Software Engineer >>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>> lean.enterprise.middleware >>>>>>>> >>>>>>>> mobile: +94 77 2273550 >>>>>>>> blog: http://sajithr.blogspot.com/ >>>>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>>>>> >>>>>>>> On Fri, Jun 24, 2016 at 10:34 AM, Malith Jayasinghe < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Sajith, >>>>>>>>> >>>>>>>>> Could you please provide some details about how you are actually >>>>>>>>> doing these performance tests. For example, what is deployment model? >>>>>>>>> How >>>>>>>>> are you generating these workloads/events? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> Malith >>>>>>>>> >>>>>>>>> On Thu, Jun 23, 2016 at 11:28 AM, Sajith Ravindra < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> This is to give an update on the performance study we conducted >>>>>>>>>> on is-analytics server on last. The idea of this test round was to >>>>>>>>>> evaluate >>>>>>>>>> the performance of Siddhi queries used for is-analytics, therefore we >>>>>>>>>> disabled event stream persistence and spark for this test. >>>>>>>>>> >>>>>>>>>> This test was conducted on a standalone DAS server with Xms2g and >>>>>>>>>> Xmx4g. >>>>>>>>>> >>>>>>>>>> On the initial round when input TPS reaches ~20K, the server went >>>>>>>>>> OOM after consuming around 1M events. The reason for this was the >>>>>>>>>> events >>>>>>>>>> accumulated inside 7 1min time batch windows used inside. To >>>>>>>>>> overcome this >>>>>>>>>> we implemented an extension to siddhi which allows us to avoid >>>>>>>>>> duplicating >>>>>>>>>> the window. >>>>>>>>>> >>>>>>>>>> After removing duplicate windows the server was able to consume >>>>>>>>>> events at a rate of ~22K, but there were fluctuations (see the graph >>>>>>>>>> bellow) of the throughput. With analysis, we found that intense GC >>>>>>>>>> causes >>>>>>>>>> this. We suspect that this intense GC is caused when expiring a >>>>>>>>>> large >>>>>>>>>> number of events accumulated inside 1-minute window. To overcome >>>>>>>>>> this we >>>>>>>>>> are planning to batch events in 1-second windows and then accumulate >>>>>>>>>> 1second batches in 1 min window in order to stop accumulating a large >>>>>>>>>> number of events. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> As the next steps, we are planning to test the performance with >>>>>>>>>> event stream persistence and then move on to check the performance >>>>>>>>>> in DAS >>>>>>>>>> minimum HA mode. >>>>>>>>>> >>>>>>>>>> We will keep updating this thread with our findings. >>>>>>>>>> >>>>>>>>>> Please share your thought, suggestions on this. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> *,Sajith Ravindra* >>>>>>>>>> Senior Software Engineer >>>>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>>>> lean.enterprise.middleware >>>>>>>>>> >>>>>>>>>> mobile: +94 77 2273550 >>>>>>>>>> blog: http://sajithr.blogspot.com/ >>>>>>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Malith Jayasinghe >>>>>>>>> >>>>>>>>> >>>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>>> Email : [email protected] >>>>>>>>> Mobile : 0770704040 >>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Malith Jayasinghe >>>>>>> >>>>>>> >>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>> Email : [email protected] >>>>>>> Mobile : 0770704040 >>>>>>> Lean . Enterprise . Middleware >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> *V. Mohanadarshan* >>>> *Associate Tech Lead,* >>>> *Data Technologies Team,* >>>> *WSO2, Inc. http://wso2.com <http://wso2.com> * >>>> *lean.enterprise.middleware.* >>>> >>>> email: [email protected] >>>> phone:(+94) 771117673 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Thanks & Regards >>> >>> Iranga Muthuthanthri >>> (M) -0777-255773 >>> Team Product Management >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *V. Mohanadarshan* > *Associate Tech Lead,* > *Data Technologies Team,* > *WSO2, Inc. http://wso2.com <http://wso2.com> * > *lean.enterprise.middleware.* > > email: [email protected] > phone:(+94) 771117673 > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Dulitha Wijewantha (Chan) Software Engineer - Mobile Development WSO2 Inc Lean.Enterprise.Middleware * ~Email [email protected] <[email protected]>* * ~Mobile +94712112165* * ~Website dulitha.me <http://dulitha.me>* * ~Twitter @dulitharw <https://twitter.com/dulitharw>* *~Github @dulichan <https://github.com/dulichan>* *~SO @chan <http://stackoverflow.com/users/813471/chan>*
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
