Hi Nirmal,
On Tue, May 31, 2016 at 9:36 AM, Nirmal Fernando <[email protected]> wrote: > Are we making the lives of our user's difficult by this change? > Why do you think that this change makes the user's life difficult? Is the concern is about the primary event store name change? For the current users who depends on DAS (esb/apim/is/iot analytics) it is just only a name change in event sink artifacts. For existing users who already wrote there analytics using DAS 3.0.0/3.0.1, we have to provide this piece of information on migration document. > > On Mon, May 30, 2016 at 8:53 PM, Inosh Goonewardena <[email protected]> > wrote: > >> Hi Nirmal an all, >> >> Immediate change we need to do is we have to change the default record >> store name in all event sink artifacts to EVENT_STORE_RWO (this is the >> new name of our primary record store) to avoid artifact deployement issues. >> >> In the meantime we also have to analyze how the each analytics scenario >> is implemented and select the correct record store for streams and tables. >> Please take the esb analyics scenario I have described above as reference. >> For example, in esb analytics scenario we can use EVENT_STORE_WO record >> store for persist events that are received from esb because sparks scripts >> are not directly reading from those tables to do summarization. >> >> On Monday, May 30, 2016, Nirmal Fernando <[email protected]> wrote: >> >>> Hi Inosh, >>> >>> What's need to be changed? Can you please provide us with more info? >>> Thanks. >>> >>> On Mon, May 30, 2016 at 4:48 PM, Inosh Goonewardena <[email protected]> >>> wrote: >>> >>>> Above mentioned changes have been made to DAS (product-das). >>>> >>>> @ ESB/APIM/IS/IOT analytics teams, >>>> We need to make the necessary changes to <product> analytics features >>>> accordingly as per above. >>>> >>>> On Fri, May 27, 2016 at 11:45 AM, Gihan Anuruddha <[email protected]> >>>> wrote: >>>> >>>>> Hi Shavantha, >>>>> >>>>> Still, we didn't make above suggest changes to our implementation. So >>>>> current release or SNAPSHOT api-analytics server doesn't have this. But >>>>> anyway, relevant product team has to come with a proper load test and see >>>>> which configuration will cater their analytics requirement efficiently. >>>>> >>>>> Regards, >>>>> Gihan >>>>> >>>>> On Fri, May 27, 2016 at 11:25 AM, Shavantha Weerasinghe < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Inosh >>>>>> >>>>>> So going forward are we to use this approach for setting up the >>>>>> analytic server. We are currently working on the api manager analytics >>>>>> setup >>>>>> >>>>>> regards, >>>>>> >>>>>> Shavantha Weerasinghe >>>>>> Senior Software Engineer QA >>>>>> WSO2, Inc. >>>>>> lean.enterprise.middleware. >>>>>> http://wso2.com >>>>>> http://wso2.org >>>>>> Tel : 94 11 214 5345 >>>>>> Fax :94 11 2145300 >>>>>> >>>>>> >>>>>> On Wed, May 25, 2016 at 8:10 PM, Inosh Goonewardena <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> At the moment DAS support both MyISAM and InnoDB, but configured to >>>>>>> use MyISAM by default. >>>>>>> >>>>>>> There are several differences between MYISAM and InnoDB, but what is >>>>>>> most relevant with regard to DAS is the difference in concurrency. >>>>>>> Basically, MyISAM uses table-level locking and InnoDB uses row-level >>>>>>> locking. So, with MyISAM, if we are running Spark queries while >>>>>>> publishing >>>>>>> data to DAS, in higher TPS it can lead to issues due to the inability of >>>>>>> obtaining the table lock by DAL layer to insert data to the table while >>>>>>> Spark reading from the same table. >>>>>>> >>>>>>> However, on the other hand, with InnoDB write speed is considerably >>>>>>> slow (because it is designed to support transactions), so it will affect >>>>>>> the receiver performance. >>>>>>> >>>>>>> One option we have in DAS is, we can use two DBs to to keep incoming >>>>>>> records and processed records, i.e., EVENT_STORE and >>>>>>> PROCESSED_DATA_STORE. >>>>>>> >>>>>>> For ESB Analytics, we can configure to use MyISAM for EVENT_STORE >>>>>>> and InnoDB for PROCESSED_DATA_STORE. It is because in ESB analytics, >>>>>>> summarizing up to minute level is done by real time analytics and Spark >>>>>>> queries will read and process data using minutely (and higher) tables >>>>>>> which >>>>>>> we can keep in PROCESSED_DATA_STORE. Since raw table(which data receiver >>>>>>> writes data) is not being used by Spark queries, the receiver >>>>>>> performance >>>>>>> will not be affected. >>>>>>> >>>>>>> However, in most cases, Spark queries may written to read data >>>>>>> directly from raw tables. As mentioned above, with MyISAM this could >>>>>>> lead >>>>>>> to performance issues if data publishing and spark analytics happens in >>>>>>> parallel. So considering that I think we should change the default >>>>>>> configuration to use InnoDB. WDYT? >>>>>>> >>>>>>> -- >>>>>>> Thanks & Regards, >>>>>>> >>>>>>> Inosh Goonewardena >>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>> Mobile: +94779966317 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> W.G. Gihan Anuruddha >>>>> Senior Software Engineer | WSO2, Inc. >>>>> M: +94772272595 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks & Regards, >>>> >>>> Inosh Goonewardena >>>> Associate Technical Lead- WSO2 Inc. >>>> Mobile: +94779966317 >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> -- >> Thanks & Regards, >> >> Inosh Goonewardena >> Associate Technical Lead- WSO2 Inc. >> Mobile: +94779966317 >> >> > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Thanks & Regards, Inosh Goonewardena Associate Technical Lead- WSO2 Inc. Mobile: +94779966317
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
