Hey saith, This is great. So when you removed the duplicate windows, there were no OOM issues?
Seshi On 24 Jun 2016 14:28, "Sajith Ravindra" <[email protected]> wrote: > Hi Malith, > > Thanks for the explanation. >> >> I would expect some variation in the throughput. The aim should be >> minimize the variation in the throughput (while maintaining the throughput >> at its highest level). >> > Agreed. Actually, our expectation is to minimize the fluctuation and to > increase the throughput. > >> >> It is possible to measure the latency as well? >> > In IS-Analytics we don't generate any output events or alerts currently. > Therefore, we don't calculate the latency. What we do is summarize events > using siddhi queries and persist them in a DB and further summarize > persisted data using Spark and then displayed through dashbaords. > > > Thanks > *,Sajith Ravindra* > Senior Software Engineer > WSO2 Inc.; http://wso2.com > lean.enterprise.middleware > > mobile: +94 77 2273550 > blog: http://sajithr.blogspot.com/ > <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> > > On Fri, Jun 24, 2016 at 1:57 PM, Malith Jayasinghe <[email protected]> > wrote: > >> Hi Sajith, >> >> Thanks for the explanation. >> >> I would expect some variation in the throughput. The aim should be >> minimize the variation in the throughput (while maintaining the throughput >> at its highest level). >> >> It is possible to measure the latency as well? >> >> regards >> >> Malith >> >> >> >> On Thu, Jun 23, 2016 at 11:13 PM, Sajith Ravindra <[email protected]> >> wrote: >> >>> Hi Malith, >>> >>> The deployment is just a standalone DAS server, and we are planning to >>> do a test for HA deployment in recent future. >>> >>> The workload is generated by a .csv data file which has 100K sample >>> events, 10M events are generated by iterating through the same data set 100 >>> times. But we keep increasing the timestamp. A simple thrift client is used >>> to publish data. >>> >>> >>> >>> Thanks >>> *,Sajith Ravindra* >>> Senior Software Engineer >>> WSO2 Inc.; http://wso2.com >>> lean.enterprise.middleware >>> >>> mobile: +94 77 2273550 >>> blog: http://sajithr.blogspot.com/ >>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>> >>> On Fri, Jun 24, 2016 at 10:34 AM, Malith Jayasinghe <[email protected]> >>> wrote: >>> >>>> Hi Sajith, >>>> >>>> Could you please provide some details about how you are actually doing >>>> these performance tests. For example, what is deployment model? How are >>>> you generating these workloads/events? >>>> >>>> Thanks >>>> >>>> Malith >>>> >>>> On Thu, Jun 23, 2016 at 11:28 AM, Sajith Ravindra <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> This is to give an update on the performance study we conducted on >>>>> is-analytics server on last. The idea of this test round was to evaluate >>>>> the performance of Siddhi queries used for is-analytics, therefore we >>>>> disabled event stream persistence and spark for this test. >>>>> >>>>> This test was conducted on a standalone DAS server with Xms2g and >>>>> Xmx4g. >>>>> >>>>> On the initial round when input TPS reaches ~20K, the server went OOM >>>>> after consuming around 1M events. The reason for this was the events >>>>> accumulated inside 7 1min time batch windows used inside. To overcome >>>>> this >>>>> we implemented an extension to siddhi which allows us to avoid duplicating >>>>> the window. >>>>> >>>>> After removing duplicate windows the server was able to consume events >>>>> at a rate of ~22K, but there were fluctuations (see the graph bellow) of >>>>> the throughput. With analysis, we found that intense GC causes this. We >>>>> suspect that this intense GC is caused when expiring a large number of >>>>> events accumulated inside 1-minute window. To overcome this we are >>>>> planning to batch events in 1-second windows and then accumulate 1second >>>>> batches in 1 min window in order to stop accumulating a large number of >>>>> events. >>>>> >>>>> >>>>> As the next steps, we are planning to test the performance with event >>>>> stream persistence and then move on to check the performance in DAS >>>>> minimum >>>>> HA mode. >>>>> >>>>> We will keep updating this thread with our findings. >>>>> >>>>> Please share your thought, suggestions on this. >>>>> >>>>> Thanks >>>>> *,Sajith Ravindra* >>>>> Senior Software Engineer >>>>> WSO2 Inc.; http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>>> mobile: +94 77 2273550 >>>>> blog: http://sajithr.blogspot.com/ >>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Malith Jayasinghe >>>> >>>> >>>> WSO2, Inc. (http://wso2.com) >>>> Email : [email protected] >>>> Mobile : 0770704040 >>>> Lean . Enterprise . Middleware >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Malith Jayasinghe >> >> >> WSO2, Inc. (http://wso2.com) >> Email : [email protected] >> Mobile : 0770704040 >> Lean . Enterprise . Middleware >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
