On Wed, Jun 29, 2016 at 9:23 PM, Dulitha Wijewantha <[email protected]> wrote:
> Hi Mohan/Sajith, > Can you please explain what this particular scenario was in IS? Is this a > counting scenario on a window? This would provide input for others who are > writing real time analytics on CEP. > Yes, Here we have mainly focused on some aggregations like Sum with external time batch windows.. Queries can be found in [1] [1] https://github.com/wso2/analytics-is/blob/master/features/org.wso2.carbon.analytics.is.feature/src/main/capp/AuthenticationAnalyticsExecutionPlan/IsAnalytics-ExecutionPlan-AuthenticationData.siddhiql Thanks, Mohan > > Cheers~ > > On Mon, Jun 27, 2016 at 5:50 PM, Mohanadarshan Vivekanandalingam < > [email protected]> wrote: > >> >> >> On Mon, Jun 27, 2016 at 8:03 PM, Seshika Fernando <[email protected]> >> wrote: >> >>> Iranga, >>> The optimization applies to siddhi queries that were written for IS >>> analytics. Other product analytics would have different queries so this is >>> not applicable there. >>> This is not a siddhi level change, rather an optimization of the >>> previously written queries. >>> >> Yes, above per-second and per-minute design is applied to some other >> analytics product as well but it is not generic and too specific for the >> usecase and query as mentioned by Seshi.. >> >> Thanks, >> Mohan >> >> >>> Seshi >>> On 27 Jun 2016 16:48, "Iranga Muthuthanthri" <[email protected]> wrote: >>> >>>> >>>> >>>> On Mon, Jun 27, 2016 at 4:35 PM, Mohanadarshan Vivekanandalingam < >>>> [email protected]> wrote: >>>> >>>>> >>>>> >>>>> On Mon, Jun 27, 2016 at 4:30 PM, Seshika Fernando <[email protected]> >>>>> wrote: >>>>> >>>>>> Hey saith, >>>>>> >>>>>> This is great. So when you removed the duplicate windows, there were >>>>>> no OOM issues? >>>>>> >>>>> Yes, we have introduced per-second windows and eventually per-minute >>>>> event count get reduced. Then, there will not be an OOM.. >>>>> >>>> >>>> is this specific for IS-Analytics or common to all other 'product >>>> analytics'. >>>> >>>> >>>>> Latest number is also seems good.. SajithR, can share those numbers.. >>>>> >>>>> Thanks, >>>>> Mohan >>>>> >>>>> >>>>>> Seshi >>>>>> On 24 Jun 2016 14:28, "Sajith Ravindra" <[email protected]> wrote: >>>>>> >>>>>>> Hi Malith, >>>>>>> >>>>>>> Thanks for the explanation. >>>>>>>> >>>>>>>> I would expect some variation in the throughput. The aim should be >>>>>>>> minimize the variation in the throughput (while maintaining the >>>>>>>> throughput >>>>>>>> at its highest level). >>>>>>>> >>>>>>> Agreed. Actually, our expectation is to minimize the fluctuation >>>>>>> and to increase the throughput. >>>>>>> >>>>>>>> >>>>>>>> It is possible to measure the latency as well? >>>>>>>> >>>>>>> In IS-Analytics we don't generate any output events or alerts >>>>>>> currently. Therefore, we don't calculate the latency. What we do is >>>>>>> summarize events using siddhi queries and persist them in a DB and >>>>>>> further >>>>>>> summarize persisted data using Spark and then displayed through >>>>>>> dashbaords. >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> *,Sajith Ravindra* >>>>>>> Senior Software Engineer >>>>>>> WSO2 Inc.; http://wso2.com >>>>>>> lean.enterprise.middleware >>>>>>> >>>>>>> mobile: +94 77 2273550 >>>>>>> blog: http://sajithr.blogspot.com/ >>>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>>>> >>>>>>> On Fri, Jun 24, 2016 at 1:57 PM, Malith Jayasinghe <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> Hi Sajith, >>>>>>>> >>>>>>>> Thanks for the explanation. >>>>>>>> >>>>>>>> I would expect some variation in the throughput. The aim should be >>>>>>>> minimize the variation in the throughput (while maintaining the >>>>>>>> throughput >>>>>>>> at its highest level). >>>>>>>> >>>>>>>> It is possible to measure the latency as well? >>>>>>>> >>>>>>>> regards >>>>>>>> >>>>>>>> Malith >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jun 23, 2016 at 11:13 PM, Sajith Ravindra <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi Malith, >>>>>>>>> >>>>>>>>> The deployment is just a standalone DAS server, and we are >>>>>>>>> planning to do a test for HA deployment in recent future. >>>>>>>>> >>>>>>>>> The workload is generated by a .csv data file which has 100K >>>>>>>>> sample events, 10M events are generated by iterating through the same >>>>>>>>> data >>>>>>>>> set 100 times. But we keep increasing the timestamp. A simple thrift >>>>>>>>> client >>>>>>>>> is used to publish data. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> *,Sajith Ravindra* >>>>>>>>> Senior Software Engineer >>>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>>> lean.enterprise.middleware >>>>>>>>> >>>>>>>>> mobile: +94 77 2273550 >>>>>>>>> blog: http://sajithr.blogspot.com/ >>>>>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>>>>>> >>>>>>>>> On Fri, Jun 24, 2016 at 10:34 AM, Malith Jayasinghe < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Sajith, >>>>>>>>>> >>>>>>>>>> Could you please provide some details about how you are actually >>>>>>>>>> doing these performance tests. For example, what is deployment >>>>>>>>>> model? How >>>>>>>>>> are you generating these workloads/events? >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> Malith >>>>>>>>>> >>>>>>>>>> On Thu, Jun 23, 2016 at 11:28 AM, Sajith Ravindra < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> This is to give an update on the performance study we conducted >>>>>>>>>>> on is-analytics server on last. The idea of this test round was to >>>>>>>>>>> evaluate >>>>>>>>>>> the performance of Siddhi queries used for is-analytics, therefore >>>>>>>>>>> we >>>>>>>>>>> disabled event stream persistence and spark for this test. >>>>>>>>>>> >>>>>>>>>>> This test was conducted on a standalone DAS server with Xms2g >>>>>>>>>>> and Xmx4g. >>>>>>>>>>> >>>>>>>>>>> On the initial round when input TPS reaches ~20K, the server >>>>>>>>>>> went OOM after consuming around 1M events. The reason for this was >>>>>>>>>>> the >>>>>>>>>>> events accumulated inside 7 1min time batch windows used inside. To >>>>>>>>>>> overcome this we implemented an extension to siddhi which allows us >>>>>>>>>>> to >>>>>>>>>>> avoid duplicating the window. >>>>>>>>>>> >>>>>>>>>>> After removing duplicate windows the server was able to consume >>>>>>>>>>> events at a rate of ~22K, but there were fluctuations (see the graph >>>>>>>>>>> bellow) of the throughput. With analysis, we found that intense GC >>>>>>>>>>> causes >>>>>>>>>>> this. We suspect that this intense GC is caused when expiring a >>>>>>>>>>> large >>>>>>>>>>> number of events accumulated inside 1-minute window. To overcome >>>>>>>>>>> this we >>>>>>>>>>> are planning to batch events in 1-second windows and then accumulate >>>>>>>>>>> 1second batches in 1 min window in order to stop accumulating a >>>>>>>>>>> large >>>>>>>>>>> number of events. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> As the next steps, we are planning to test the performance with >>>>>>>>>>> event stream persistence and then move on to check the performance >>>>>>>>>>> in DAS >>>>>>>>>>> minimum HA mode. >>>>>>>>>>> >>>>>>>>>>> We will keep updating this thread with our findings. >>>>>>>>>>> >>>>>>>>>>> Please share your thought, suggestions on this. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> *,Sajith Ravindra* >>>>>>>>>>> Senior Software Engineer >>>>>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>> >>>>>>>>>>> mobile: +94 77 2273550 >>>>>>>>>>> blog: http://sajithr.blogspot.com/ >>>>>>>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Malith Jayasinghe >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>>>> Email : [email protected] >>>>>>>>>> Mobile : 0770704040 >>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Malith Jayasinghe >>>>>>>> >>>>>>>> >>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>> Email : [email protected] >>>>>>>> Mobile : 0770704040 >>>>>>>> Lean . Enterprise . Middleware >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *V. Mohanadarshan* >>>>> *Associate Tech Lead,* >>>>> *Data Technologies Team,* >>>>> *WSO2, Inc. http://wso2.com <http://wso2.com> * >>>>> *lean.enterprise.middleware.* >>>>> >>>>> email: [email protected] >>>>> phone:(+94) 771117673 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks & Regards >>>> >>>> Iranga Muthuthanthri >>>> (M) -0777-255773 >>>> Team Product Management >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *V. Mohanadarshan* >> *Associate Tech Lead,* >> *Data Technologies Team,* >> *WSO2, Inc. http://wso2.com <http://wso2.com> * >> *lean.enterprise.middleware.* >> >> email: [email protected] >> phone:(+94) 771117673 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Dulitha Wijewantha (Chan) > Software Engineer - Mobile Development > WSO2 Inc > Lean.Enterprise.Middleware > * ~Email [email protected] <[email protected]>* > * ~Mobile +94712112165 <%2B94712112165>* > * ~Website dulitha.me <http://dulitha.me>* > * ~Twitter @dulitharw <https://twitter.com/dulitharw>* > *~Github @dulichan <https://github.com/dulichan>* > *~SO @chan <http://stackoverflow.com/users/813471/chan>* > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *V. Mohanadarshan* *Associate Tech Lead,* *Data Technologies Team,* *WSO2, Inc. http://wso2.com <http://wso2.com> * *lean.enterprise.middleware.* email: [email protected] phone:(+94) 771117673
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
