Hi Inosh, If I am to write queries, do I need to think about the underlying store etc.?
*"In the meantime we also have to analyze how the each analytics scenario is implemented and select the correct record store for streams and tables."* On Tue, May 31, 2016 at 10:38 AM, Inosh Goonewardena <[email protected]> wrote: > Hi Nirmal, > > > On Tue, May 31, 2016 at 9:36 AM, Nirmal Fernando <[email protected]> wrote: > >> Are we making the lives of our user's difficult by this change? >> > > Why do you think that this change makes the user's life difficult? Is the > concern is about the primary event store name change? For the current users > who depends on DAS (esb/apim/is/iot analytics) it is just only a name > change in event sink artifacts. For existing users who already wrote there > analytics using DAS 3.0.0/3.0.1, we have to provide this piece of > information on migration document. > > > >> >> On Mon, May 30, 2016 at 8:53 PM, Inosh Goonewardena <[email protected]> >> wrote: >> >>> Hi Nirmal an all, >>> >>> Immediate change we need to do is we have to change the default record >>> store name in all event sink artifacts to EVENT_STORE_RWO (this is the >>> new name of our primary record store) to avoid artifact deployement issues. >>> >>> In the meantime we also have to analyze how the each analytics scenario >>> is implemented and select the correct record store for streams and tables. >>> Please take the esb analyics scenario I have described above as reference. >>> For example, in esb analytics scenario we can use EVENT_STORE_WO record >>> store for persist events that are received from esb because sparks scripts >>> are not directly reading from those tables to do summarization. >>> >>> On Monday, May 30, 2016, Nirmal Fernando <[email protected]> wrote: >>> >>>> Hi Inosh, >>>> >>>> What's need to be changed? Can you please provide us with more info? >>>> Thanks. >>>> >>>> On Mon, May 30, 2016 at 4:48 PM, Inosh Goonewardena <[email protected]> >>>> wrote: >>>> >>>>> Above mentioned changes have been made to DAS (product-das). >>>>> >>>>> @ ESB/APIM/IS/IOT analytics teams, >>>>> We need to make the necessary changes to <product> analytics features >>>>> accordingly as per above. >>>>> >>>>> On Fri, May 27, 2016 at 11:45 AM, Gihan Anuruddha <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Shavantha, >>>>>> >>>>>> Still, we didn't make above suggest changes to our implementation. >>>>>> So current release or SNAPSHOT api-analytics server doesn't have this. >>>>>> But >>>>>> anyway, relevant product team has to come with a proper load test and see >>>>>> which configuration will cater their analytics requirement efficiently. >>>>>> >>>>>> Regards, >>>>>> Gihan >>>>>> >>>>>> On Fri, May 27, 2016 at 11:25 AM, Shavantha Weerasinghe < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Inosh >>>>>>> >>>>>>> So going forward are we to use this approach for setting up the >>>>>>> analytic server. We are currently working on the api manager analytics >>>>>>> setup >>>>>>> >>>>>>> regards, >>>>>>> >>>>>>> Shavantha Weerasinghe >>>>>>> Senior Software Engineer QA >>>>>>> WSO2, Inc. >>>>>>> lean.enterprise.middleware. >>>>>>> http://wso2.com >>>>>>> http://wso2.org >>>>>>> Tel : 94 11 214 5345 >>>>>>> Fax :94 11 2145300 >>>>>>> >>>>>>> >>>>>>> On Wed, May 25, 2016 at 8:10 PM, Inosh Goonewardena <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> At the moment DAS support both MyISAM and InnoDB, but configured to >>>>>>>> use MyISAM by default. >>>>>>>> >>>>>>>> There are several differences between MYISAM and InnoDB, but what >>>>>>>> is most relevant with regard to DAS is the difference in concurrency. >>>>>>>> Basically, MyISAM uses table-level locking and InnoDB uses row-level >>>>>>>> locking. So, with MyISAM, if we are running Spark queries while >>>>>>>> publishing >>>>>>>> data to DAS, in higher TPS it can lead to issues due to the inability >>>>>>>> of >>>>>>>> obtaining the table lock by DAL layer to insert data to the table while >>>>>>>> Spark reading from the same table. >>>>>>>> >>>>>>>> However, on the other hand, with InnoDB write speed is considerably >>>>>>>> slow (because it is designed to support transactions), so it will >>>>>>>> affect >>>>>>>> the receiver performance. >>>>>>>> >>>>>>>> One option we have in DAS is, we can use two DBs to to keep >>>>>>>> incoming records and processed records, i.e., EVENT_STORE and >>>>>>>> PROCESSED_DATA_STORE. >>>>>>>> >>>>>>>> For ESB Analytics, we can configure to use MyISAM for EVENT_STORE >>>>>>>> and InnoDB for PROCESSED_DATA_STORE. It is because in ESB analytics, >>>>>>>> summarizing up to minute level is done by real time analytics and Spark >>>>>>>> queries will read and process data using minutely (and higher) tables >>>>>>>> which >>>>>>>> we can keep in PROCESSED_DATA_STORE. Since raw table(which data >>>>>>>> receiver >>>>>>>> writes data) is not being used by Spark queries, the receiver >>>>>>>> performance >>>>>>>> will not be affected. >>>>>>>> >>>>>>>> However, in most cases, Spark queries may written to read data >>>>>>>> directly from raw tables. As mentioned above, with MyISAM this could >>>>>>>> lead >>>>>>>> to performance issues if data publishing and spark analytics happens in >>>>>>>> parallel. So considering that I think we should change the default >>>>>>>> configuration to use InnoDB. WDYT? >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks & Regards, >>>>>>>> >>>>>>>> Inosh Goonewardena >>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>> Mobile: +94779966317 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> W.G. Gihan Anuruddha >>>>>> Senior Software Engineer | WSO2, Inc. >>>>>> M: +94772272595 >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks & Regards, >>>>> >>>>> Inosh Goonewardena >>>>> Associate Technical Lead- WSO2 Inc. >>>>> Mobile: +94779966317 >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> -- >>> Thanks & Regards, >>> >>> Inosh Goonewardena >>> Associate Technical Lead- WSO2 Inc. >>> Mobile: +94779966317 >>> >>> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > Thanks & Regards, > > Inosh Goonewardena > Associate Technical Lead- WSO2 Inc. > Mobile: +94779966317 > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
