Are we making the lives of our user's difficult by this change? On Mon, May 30, 2016 at 8:53 PM, Inosh Goonewardena <[email protected]> wrote:
> Hi Nirmal an all, > > Immediate change we need to do is we have to change the default record > store name in all event sink artifacts to EVENT_STORE_RWO (this is the > new name of our primary record store) to avoid artifact deployement issues. > > In the meantime we also have to analyze how the each analytics scenario is > implemented and select the correct record store for streams and tables. > Please take the esb analyics scenario I have described above as reference. > For example, in esb analytics scenario we can use EVENT_STORE_WO record > store for persist events that are received from esb because sparks scripts > are not directly reading from those tables to do summarization. > > On Monday, May 30, 2016, Nirmal Fernando <[email protected]> wrote: > >> Hi Inosh, >> >> What's need to be changed? Can you please provide us with more info? >> Thanks. >> >> On Mon, May 30, 2016 at 4:48 PM, Inosh Goonewardena <[email protected]> >> wrote: >> >>> Above mentioned changes have been made to DAS (product-das). >>> >>> @ ESB/APIM/IS/IOT analytics teams, >>> We need to make the necessary changes to <product> analytics features >>> accordingly as per above. >>> >>> On Fri, May 27, 2016 at 11:45 AM, Gihan Anuruddha <[email protected]> >>> wrote: >>> >>>> Hi Shavantha, >>>> >>>> Still, we didn't make above suggest changes to our implementation. So >>>> current release or SNAPSHOT api-analytics server doesn't have this. But >>>> anyway, relevant product team has to come with a proper load test and see >>>> which configuration will cater their analytics requirement efficiently. >>>> >>>> Regards, >>>> Gihan >>>> >>>> On Fri, May 27, 2016 at 11:25 AM, Shavantha Weerasinghe < >>>> [email protected]> wrote: >>>> >>>>> Hi Inosh >>>>> >>>>> So going forward are we to use this approach for setting up the >>>>> analytic server. We are currently working on the api manager analytics >>>>> setup >>>>> >>>>> regards, >>>>> >>>>> Shavantha Weerasinghe >>>>> Senior Software Engineer QA >>>>> WSO2, Inc. >>>>> lean.enterprise.middleware. >>>>> http://wso2.com >>>>> http://wso2.org >>>>> Tel : 94 11 214 5345 >>>>> Fax :94 11 2145300 >>>>> >>>>> >>>>> On Wed, May 25, 2016 at 8:10 PM, Inosh Goonewardena <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> At the moment DAS support both MyISAM and InnoDB, but configured to >>>>>> use MyISAM by default. >>>>>> >>>>>> There are several differences between MYISAM and InnoDB, but what is >>>>>> most relevant with regard to DAS is the difference in concurrency. >>>>>> Basically, MyISAM uses table-level locking and InnoDB uses row-level >>>>>> locking. So, with MyISAM, if we are running Spark queries while >>>>>> publishing >>>>>> data to DAS, in higher TPS it can lead to issues due to the inability of >>>>>> obtaining the table lock by DAL layer to insert data to the table while >>>>>> Spark reading from the same table. >>>>>> >>>>>> However, on the other hand, with InnoDB write speed is considerably >>>>>> slow (because it is designed to support transactions), so it will affect >>>>>> the receiver performance. >>>>>> >>>>>> One option we have in DAS is, we can use two DBs to to keep incoming >>>>>> records and processed records, i.e., EVENT_STORE and >>>>>> PROCESSED_DATA_STORE. >>>>>> >>>>>> For ESB Analytics, we can configure to use MyISAM for EVENT_STORE and >>>>>> InnoDB for PROCESSED_DATA_STORE. It is because in ESB analytics, >>>>>> summarizing up to minute level is done by real time analytics and Spark >>>>>> queries will read and process data using minutely (and higher) tables >>>>>> which >>>>>> we can keep in PROCESSED_DATA_STORE. Since raw table(which data receiver >>>>>> writes data) is not being used by Spark queries, the receiver performance >>>>>> will not be affected. >>>>>> >>>>>> However, in most cases, Spark queries may written to read data >>>>>> directly from raw tables. As mentioned above, with MyISAM this could lead >>>>>> to performance issues if data publishing and spark analytics happens in >>>>>> parallel. So considering that I think we should change the default >>>>>> configuration to use InnoDB. WDYT? >>>>>> >>>>>> -- >>>>>> Thanks & Regards, >>>>>> >>>>>> Inosh Goonewardena >>>>>> Associate Technical Lead- WSO2 Inc. >>>>>> Mobile: +94779966317 >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> W.G. Gihan Anuruddha >>>> Senior Software Engineer | WSO2, Inc. >>>> M: +94772272595 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Thanks & Regards, >>> >>> Inosh Goonewardena >>> Associate Technical Lead- WSO2 Inc. >>> Mobile: +94779966317 >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > -- > Thanks & Regards, > > Inosh Goonewardena > Associate Technical Lead- WSO2 Inc. > Mobile: +94779966317 > > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
