This change is done. So we can use the same record store name (i.e. EVENT_STORE) which will represent read and write optimized record store.
On Wed, Jun 1, 2016 at 10:21 AM, Nirmal Fernando <[email protected]> wrote: > +1, for no changes to existing packs. Let us know once this is done. > > On Wed, Jun 1, 2016 at 10:18 AM, Inosh Goonewardena <[email protected]> > wrote: > >> Hi Anjana, >> >> On Wed, Jun 1, 2016 at 10:04 AM, Anjana Fernando <[email protected]> wrote: >> >>> Hi Inosh, >>> >>> Actually can we keep the record store name "EVENT_STORE" as it is, which >>> would resemble the suggest "EVENT_STORE_RWO" one here, and just add the >>> extra "EVENT_STORE_RO". So with this, this will not break any existing >>> solutions, which have explicitly already mentioned the record store as >>> "EVENT_STORE" in their configurations. Only required people in the future, >>> e.g. ESB Anallytics, can use the new "EVENT_STORE_RO" one. >>> >> >> Yes. We can name it that way. That was actually my first thought too. >> Will do the changes now. >> >> >>> >>> Cheers, >>> Anjana. >>> >>> On Tue, May 31, 2016 at 11:32 AM, Nirmal Fernando <[email protected]> >>> wrote: >>> >>>> Ok, great! thanks Gokul. >>>> >>>> On Tue, May 31, 2016 at 11:26 AM, Gokul Balakrishnan <[email protected]> >>>> wrote: >>>> >>>>> Hi Nirmal, >>>>> >>>>> We are not making the users' lives difficult nor are we exposing any >>>>> of these changes to the end user. You need not worry about any of it when >>>>> you write queries. >>>>> >>>>> If I explain the change a bit, tables that get created from spark >>>>> output are added to PROCESSED_DATA_STORE (R/W optimised) by default. There >>>>> have been no changes on this end. >>>>> >>>>> The primary record store DAS uses has been renamed to EVENT_STORE_RWO, >>>>> This record store will be used store raw incoming data. >>>>> >>>>> The only change is we now have a new records store called >>>>> EVENT_STORE_WO(optimised for writes) which can optionally be used if your >>>>> priority write performance over read (an example would be ESB analytics >>>>> events which are not required to be read by Spark). >>>>> >>>>> If a record store other than RDBMS (i.e. HBase/Cassandra), none of >>>>> this would make any impact, and the properties set for R/W optimisation >>>>> will automatically be ignored. >>>>> >>>>> As Inosh has explained, the change that needs to be done by the >>>>> *-analytics teams is to rename the store on the event sink artifacts to >>>>> ensure that there are no deployment issues with latest DAS packs. >>>>> >>>>> Thanks, >>>>> >>>>> On 31 May 2016 at 10:43, Nirmal Fernando <[email protected]> wrote: >>>>> >>>>>> Hi Inosh, >>>>>> >>>>>> If I am to write queries, do I need to think about the underlying >>>>>> store etc.? >>>>>> >>>>>> *"In the meantime we also have to analyze how the each analytics >>>>>> scenario is implemented and select the correct record store for streams >>>>>> and >>>>>> tables."* >>>>>> >>>>>> On Tue, May 31, 2016 at 10:38 AM, Inosh Goonewardena <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Nirmal, >>>>>>> >>>>>>> >>>>>>> On Tue, May 31, 2016 at 9:36 AM, Nirmal Fernando <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Are we making the lives of our user's difficult by this change? >>>>>>>> >>>>>>> >>>>>>> Why do you think that this change makes the user's life difficult? >>>>>>> Is the concern is about the primary event store name change? For the >>>>>>> current users who depends on DAS (esb/apim/is/iot analytics) it is just >>>>>>> only a name change in event sink artifacts. For existing users who >>>>>>> already >>>>>>> wrote there analytics using DAS 3.0.0/3.0.1, we have to provide this >>>>>>> piece >>>>>>> of information on migration document. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Mon, May 30, 2016 at 8:53 PM, Inosh Goonewardena <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi Nirmal an all, >>>>>>>>> >>>>>>>>> Immediate change we need to do is we have to change the default >>>>>>>>> record store name in all event sink artifacts to EVENT_STORE_RWO (this >>>>>>>>> is the new name of our primary record store) to avoid artifact >>>>>>>>> deployement >>>>>>>>> issues. >>>>>>>>> >>>>>>>>> In the meantime we also have to analyze how the each analytics >>>>>>>>> scenario is implemented and select the correct record store for >>>>>>>>> streams and >>>>>>>>> tables. Please take the esb analyics scenario I have described above >>>>>>>>> as >>>>>>>>> reference. For example, in esb analytics scenario we can use >>>>>>>>> EVENT_STORE_WO record store for persist events that are received >>>>>>>>> from esb because sparks scripts are not directly reading from those >>>>>>>>> tables >>>>>>>>> to do summarization. >>>>>>>>> >>>>>>>>> On Monday, May 30, 2016, Nirmal Fernando <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Inosh, >>>>>>>>>> >>>>>>>>>> What's need to be changed? Can you please provide us with more >>>>>>>>>> info? Thanks. >>>>>>>>>> >>>>>>>>>> On Mon, May 30, 2016 at 4:48 PM, Inosh Goonewardena < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Above mentioned changes have been made to DAS (product-das). >>>>>>>>>>> >>>>>>>>>>> @ ESB/APIM/IS/IOT analytics teams, >>>>>>>>>>> We need to make the necessary changes to <product> analytics >>>>>>>>>>> features accordingly as per above. >>>>>>>>>>> >>>>>>>>>>> On Fri, May 27, 2016 at 11:45 AM, Gihan Anuruddha < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Shavantha, >>>>>>>>>>>> >>>>>>>>>>>> Still, we didn't make above suggest changes to our >>>>>>>>>>>> implementation. So current release or SNAPSHOT api-analytics >>>>>>>>>>>> server >>>>>>>>>>>> doesn't have this. But anyway, relevant product team has to come >>>>>>>>>>>> with a >>>>>>>>>>>> proper load test and see which configuration will cater their >>>>>>>>>>>> analytics >>>>>>>>>>>> requirement efficiently. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Gihan >>>>>>>>>>>> >>>>>>>>>>>> On Fri, May 27, 2016 at 11:25 AM, Shavantha Weerasinghe < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Inosh >>>>>>>>>>>>> >>>>>>>>>>>>> So going forward are we to use this approach for setting up >>>>>>>>>>>>> the analytic server. We are currently working on the api manager >>>>>>>>>>>>> analytics >>>>>>>>>>>>> setup >>>>>>>>>>>>> >>>>>>>>>>>>> regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Shavantha Weerasinghe >>>>>>>>>>>>> Senior Software Engineer QA >>>>>>>>>>>>> WSO2, Inc. >>>>>>>>>>>>> lean.enterprise.middleware. >>>>>>>>>>>>> http://wso2.com >>>>>>>>>>>>> http://wso2.org >>>>>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, May 25, 2016 at 8:10 PM, Inosh Goonewardena < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> At the moment DAS support both MyISAM and InnoDB, but >>>>>>>>>>>>>> configured to use MyISAM by default. >>>>>>>>>>>>>> >>>>>>>>>>>>>> There are several differences between MYISAM and InnoDB, but >>>>>>>>>>>>>> what is most relevant with regard to DAS is the difference in >>>>>>>>>>>>>> concurrency. >>>>>>>>>>>>>> Basically, MyISAM uses table-level locking and InnoDB uses >>>>>>>>>>>>>> row-level >>>>>>>>>>>>>> locking. So, with MyISAM, if we are running Spark queries while >>>>>>>>>>>>>> publishing >>>>>>>>>>>>>> data to DAS, in higher TPS it can lead to issues due to the >>>>>>>>>>>>>> inability of >>>>>>>>>>>>>> obtaining the table lock by DAL layer to insert data to the >>>>>>>>>>>>>> table while >>>>>>>>>>>>>> Spark reading from the same table. >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, on the other hand, with InnoDB write speed is >>>>>>>>>>>>>> considerably slow (because it is designed to support >>>>>>>>>>>>>> transactions), so it >>>>>>>>>>>>>> will affect the receiver performance. >>>>>>>>>>>>>> >>>>>>>>>>>>>> One option we have in DAS is, we can use two DBs to to keep >>>>>>>>>>>>>> incoming records and processed records, i.e., EVENT_STORE and >>>>>>>>>>>>>> PROCESSED_DATA_STORE. >>>>>>>>>>>>>> >>>>>>>>>>>>>> For ESB Analytics, we can configure to use MyISAM for >>>>>>>>>>>>>> EVENT_STORE and InnoDB for PROCESSED_DATA_STORE. It is because >>>>>>>>>>>>>> in ESB >>>>>>>>>>>>>> analytics, summarizing up to minute level is done by real time >>>>>>>>>>>>>> analytics >>>>>>>>>>>>>> and Spark queries will read and process data using minutely (and >>>>>>>>>>>>>> higher) >>>>>>>>>>>>>> tables which we can keep in PROCESSED_DATA_STORE. Since raw >>>>>>>>>>>>>> table(which >>>>>>>>>>>>>> data receiver writes data) is not being used by Spark queries, >>>>>>>>>>>>>> the receiver >>>>>>>>>>>>>> performance will not be affected. >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, in most cases, Spark queries may written to read >>>>>>>>>>>>>> data directly from raw tables. As mentioned above, with MyISAM >>>>>>>>>>>>>> this could >>>>>>>>>>>>>> lead to performance issues if data publishing and spark >>>>>>>>>>>>>> analytics happens >>>>>>>>>>>>>> in parallel. So considering that I think we should change the >>>>>>>>>>>>>> default >>>>>>>>>>>>>> configuration to use InnoDB. WDYT? >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Thanks & Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Inosh Goonewardena >>>>>>>>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>>>>>>>> Mobile: +94779966317 >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> W.G. Gihan Anuruddha >>>>>>>>>>>> Senior Software Engineer | WSO2, Inc. >>>>>>>>>>>> M: +94772272595 >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Thanks & Regards, >>>>>>>>>>> >>>>>>>>>>> Inosh Goonewardena >>>>>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>>>>> Mobile: +94779966317 >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Thanks & regards, >>>>>>>>>> Nirmal >>>>>>>>>> >>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>> Mobile: +94715779733 >>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanks & Regards, >>>>>>>>> >>>>>>>>> Inosh Goonewardena >>>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>>> Mobile: +94779966317 >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks & regards, >>>>>>>> Nirmal >>>>>>>> >>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>> Mobile: +94715779733 >>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks & Regards, >>>>>>> >>>>>>> Inosh Goonewardena >>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>> Mobile: +94779966317 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks & regards, >>>>>> Nirmal >>>>>> >>>>>> Team Lead - WSO2 Machine Learner >>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>> Mobile: +94715779733 >>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Gokul Balakrishnan >>>>> Senior Software Engineer, >>>>> WSO2, Inc. http://wso2.com >>>>> M +94 77 5935 789 | +44 7563 570502 >>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> *Anjana Fernando* >>> Senior Technical Lead >>> WSO2 Inc. | http://wso2.com >>> lean . enterprise . middleware >>> >> >> >> >> -- >> Thanks & Regards, >> >> Inosh Goonewardena >> Associate Technical Lead- WSO2 Inc. >> Mobile: +94779966317 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Thanks & Regards, Inosh Goonewardena Associate Technical Lead- WSO2 Inc. Mobile: +94779966317
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
