On Mon, Jun 27, 2016 at 8:03 PM, Seshika Fernando <[email protected]> wrote:
> Iranga, > The optimization applies to siddhi queries that were written for IS > analytics. Other product analytics would have different queries so this is > not applicable there. > This is not a siddhi level change, rather an optimization of the > previously written queries. > Yes, above per-second and per-minute design is applied to some other analytics product as well but it is not generic and too specific for the usecase and query as mentioned by Seshi.. Thanks, Mohan > Seshi > On 27 Jun 2016 16:48, "Iranga Muthuthanthri" <[email protected]> wrote: > >> >> >> On Mon, Jun 27, 2016 at 4:35 PM, Mohanadarshan Vivekanandalingam < >> [email protected]> wrote: >> >>> >>> >>> On Mon, Jun 27, 2016 at 4:30 PM, Seshika Fernando <[email protected]> >>> wrote: >>> >>>> Hey saith, >>>> >>>> This is great. So when you removed the duplicate windows, there were no >>>> OOM issues? >>>> >>> Yes, we have introduced per-second windows and eventually per-minute >>> event count get reduced. Then, there will not be an OOM.. >>> >> >> is this specific for IS-Analytics or common to all other 'product >> analytics'. >> >> >>> Latest number is also seems good.. SajithR, can share those numbers.. >>> >>> Thanks, >>> Mohan >>> >>> >>>> Seshi >>>> On 24 Jun 2016 14:28, "Sajith Ravindra" <[email protected]> wrote: >>>> >>>>> Hi Malith, >>>>> >>>>> Thanks for the explanation. >>>>>> >>>>>> I would expect some variation in the throughput. The aim should be >>>>>> minimize the variation in the throughput (while maintaining the >>>>>> throughput >>>>>> at its highest level). >>>>>> >>>>> Agreed. Actually, our expectation is to minimize the fluctuation and >>>>> to increase the throughput. >>>>> >>>>>> >>>>>> It is possible to measure the latency as well? >>>>>> >>>>> In IS-Analytics we don't generate any output events or alerts >>>>> currently. Therefore, we don't calculate the latency. What we do is >>>>> summarize events using siddhi queries and persist them in a DB and further >>>>> summarize persisted data using Spark and then displayed through >>>>> dashbaords. >>>>> >>>>> >>>>> Thanks >>>>> *,Sajith Ravindra* >>>>> Senior Software Engineer >>>>> WSO2 Inc.; http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>>> mobile: +94 77 2273550 >>>>> blog: http://sajithr.blogspot.com/ >>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>> >>>>> On Fri, Jun 24, 2016 at 1:57 PM, Malith Jayasinghe <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Sajith, >>>>>> >>>>>> Thanks for the explanation. >>>>>> >>>>>> I would expect some variation in the throughput. The aim should be >>>>>> minimize the variation in the throughput (while maintaining the >>>>>> throughput >>>>>> at its highest level). >>>>>> >>>>>> It is possible to measure the latency as well? >>>>>> >>>>>> regards >>>>>> >>>>>> Malith >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 23, 2016 at 11:13 PM, Sajith Ravindra <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Malith, >>>>>>> >>>>>>> The deployment is just a standalone DAS server, and we are planning >>>>>>> to do a test for HA deployment in recent future. >>>>>>> >>>>>>> The workload is generated by a .csv data file which has 100K sample >>>>>>> events, 10M events are generated by iterating through the same data set >>>>>>> 100 >>>>>>> times. But we keep increasing the timestamp. A simple thrift client is >>>>>>> used >>>>>>> to publish data. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> *,Sajith Ravindra* >>>>>>> Senior Software Engineer >>>>>>> WSO2 Inc.; http://wso2.com >>>>>>> lean.enterprise.middleware >>>>>>> >>>>>>> mobile: +94 77 2273550 >>>>>>> blog: http://sajithr.blogspot.com/ >>>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>>>> >>>>>>> On Fri, Jun 24, 2016 at 10:34 AM, Malith Jayasinghe < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Sajith, >>>>>>>> >>>>>>>> Could you please provide some details about how you are actually >>>>>>>> doing these performance tests. For example, what is deployment model? >>>>>>>> How >>>>>>>> are you generating these workloads/events? >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> Malith >>>>>>>> >>>>>>>> On Thu, Jun 23, 2016 at 11:28 AM, Sajith Ravindra <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> This is to give an update on the performance study we conducted on >>>>>>>>> is-analytics server on last. The idea of this test round was to >>>>>>>>> evaluate >>>>>>>>> the performance of Siddhi queries used for is-analytics, therefore we >>>>>>>>> disabled event stream persistence and spark for this test. >>>>>>>>> >>>>>>>>> This test was conducted on a standalone DAS server with Xms2g and >>>>>>>>> Xmx4g. >>>>>>>>> >>>>>>>>> On the initial round when input TPS reaches ~20K, the server went >>>>>>>>> OOM after consuming around 1M events. The reason for this was the >>>>>>>>> events >>>>>>>>> accumulated inside 7 1min time batch windows used inside. To >>>>>>>>> overcome this >>>>>>>>> we implemented an extension to siddhi which allows us to avoid >>>>>>>>> duplicating >>>>>>>>> the window. >>>>>>>>> >>>>>>>>> After removing duplicate windows the server was able to consume >>>>>>>>> events at a rate of ~22K, but there were fluctuations (see the graph >>>>>>>>> bellow) of the throughput. With analysis, we found that intense GC >>>>>>>>> causes >>>>>>>>> this. We suspect that this intense GC is caused when expiring a large >>>>>>>>> number of events accumulated inside 1-minute window. To overcome >>>>>>>>> this we >>>>>>>>> are planning to batch events in 1-second windows and then accumulate >>>>>>>>> 1second batches in 1 min window in order to stop accumulating a large >>>>>>>>> number of events. >>>>>>>>> >>>>>>>>> >>>>>>>>> As the next steps, we are planning to test the performance with >>>>>>>>> event stream persistence and then move on to check the performance in >>>>>>>>> DAS >>>>>>>>> minimum HA mode. >>>>>>>>> >>>>>>>>> We will keep updating this thread with our findings. >>>>>>>>> >>>>>>>>> Please share your thought, suggestions on this. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> *,Sajith Ravindra* >>>>>>>>> Senior Software Engineer >>>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>>> lean.enterprise.middleware >>>>>>>>> >>>>>>>>> mobile: +94 77 2273550 >>>>>>>>> blog: http://sajithr.blogspot.com/ >>>>>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Malith Jayasinghe >>>>>>>> >>>>>>>> >>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>> Email : [email protected] >>>>>>>> Mobile : 0770704040 >>>>>>>> Lean . Enterprise . Middleware >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Malith Jayasinghe >>>>>> >>>>>> >>>>>> WSO2, Inc. (http://wso2.com) >>>>>> Email : [email protected] >>>>>> Mobile : 0770704040 >>>>>> Lean . Enterprise . Middleware >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> *V. Mohanadarshan* >>> *Associate Tech Lead,* >>> *Data Technologies Team,* >>> *WSO2, Inc. http://wso2.com <http://wso2.com> * >>> *lean.enterprise.middleware.* >>> >>> email: [email protected] >>> phone:(+94) 771117673 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Thanks & Regards >> >> Iranga Muthuthanthri >> (M) -0777-255773 >> Team Product Management >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *V. Mohanadarshan* *Associate Tech Lead,* *Data Technologies Team,* *WSO2, Inc. http://wso2.com <http://wso2.com> * *lean.enterprise.middleware.* email: [email protected] phone:(+94) 771117673
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
