Hi Inosh, Actually can we keep the record store name "EVENT_STORE" as it is, which would resemble the suggest "EVENT_STORE_RWO" one here, and just add the extra "EVENT_STORE_RO". So with this, this will not break any existing solutions, which have explicitly already mentioned the record store as "EVENT_STORE" in their configurations. Only required people in the future, e.g. ESB Anallytics, can use the new "EVENT_STORE_RO" one.
Cheers, Anjana. On Tue, May 31, 2016 at 11:32 AM, Nirmal Fernando <[email protected]> wrote: > Ok, great! thanks Gokul. > > On Tue, May 31, 2016 at 11:26 AM, Gokul Balakrishnan <[email protected]> > wrote: > >> Hi Nirmal, >> >> We are not making the users' lives difficult nor are we exposing any of >> these changes to the end user. You need not worry about any of it when you >> write queries. >> >> If I explain the change a bit, tables that get created from spark output >> are added to PROCESSED_DATA_STORE (R/W optimised) by default. There have >> been no changes on this end. >> >> The primary record store DAS uses has been renamed to EVENT_STORE_RWO, >> This record store will be used store raw incoming data. >> >> The only change is we now have a new records store called >> EVENT_STORE_WO(optimised for writes) which can optionally be used if your >> priority write performance over read (an example would be ESB analytics >> events which are not required to be read by Spark). >> >> If a record store other than RDBMS (i.e. HBase/Cassandra), none of this >> would make any impact, and the properties set for R/W optimisation will >> automatically be ignored. >> >> As Inosh has explained, the change that needs to be done by the >> *-analytics teams is to rename the store on the event sink artifacts to >> ensure that there are no deployment issues with latest DAS packs. >> >> Thanks, >> >> On 31 May 2016 at 10:43, Nirmal Fernando <[email protected]> wrote: >> >>> Hi Inosh, >>> >>> If I am to write queries, do I need to think about the underlying store >>> etc.? >>> >>> *"In the meantime we also have to analyze how the each analytics >>> scenario is implemented and select the correct record store for streams and >>> tables."* >>> >>> On Tue, May 31, 2016 at 10:38 AM, Inosh Goonewardena <[email protected]> >>> wrote: >>> >>>> Hi Nirmal, >>>> >>>> >>>> On Tue, May 31, 2016 at 9:36 AM, Nirmal Fernando <[email protected]> >>>> wrote: >>>> >>>>> Are we making the lives of our user's difficult by this change? >>>>> >>>> >>>> Why do you think that this change makes the user's life difficult? Is >>>> the concern is about the primary event store name change? For the current >>>> users who depends on DAS (esb/apim/is/iot analytics) it is just only a name >>>> change in event sink artifacts. For existing users who already wrote there >>>> analytics using DAS 3.0.0/3.0.1, we have to provide this piece of >>>> information on migration document. >>>> >>>> >>>> >>>>> >>>>> On Mon, May 30, 2016 at 8:53 PM, Inosh Goonewardena <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Nirmal an all, >>>>>> >>>>>> Immediate change we need to do is we have to change the default >>>>>> record store name in all event sink artifacts to EVENT_STORE_RWO (this >>>>>> is the new name of our primary record store) to avoid artifact >>>>>> deployement >>>>>> issues. >>>>>> >>>>>> In the meantime we also have to analyze how the each analytics >>>>>> scenario is implemented and select the correct record store for streams >>>>>> and >>>>>> tables. Please take the esb analyics scenario I have described above as >>>>>> reference. For example, in esb analytics scenario we can use >>>>>> EVENT_STORE_WO record store for persist events that are received >>>>>> from esb because sparks scripts are not directly reading from those >>>>>> tables >>>>>> to do summarization. >>>>>> >>>>>> On Monday, May 30, 2016, Nirmal Fernando <[email protected]> wrote: >>>>>> >>>>>>> Hi Inosh, >>>>>>> >>>>>>> What's need to be changed? Can you please provide us with more info? >>>>>>> Thanks. >>>>>>> >>>>>>> On Mon, May 30, 2016 at 4:48 PM, Inosh Goonewardena <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Above mentioned changes have been made to DAS (product-das). >>>>>>>> >>>>>>>> @ ESB/APIM/IS/IOT analytics teams, >>>>>>>> We need to make the necessary changes to <product> analytics >>>>>>>> features accordingly as per above. >>>>>>>> >>>>>>>> On Fri, May 27, 2016 at 11:45 AM, Gihan Anuruddha <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Shavantha, >>>>>>>>> >>>>>>>>> Still, we didn't make above suggest changes to our implementation. >>>>>>>>> So current release or SNAPSHOT api-analytics server doesn't have >>>>>>>>> this. But >>>>>>>>> anyway, relevant product team has to come with a proper load test and >>>>>>>>> see >>>>>>>>> which configuration will cater their analytics requirement >>>>>>>>> efficiently. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Gihan >>>>>>>>> >>>>>>>>> On Fri, May 27, 2016 at 11:25 AM, Shavantha Weerasinghe < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Inosh >>>>>>>>>> >>>>>>>>>> So going forward are we to use this approach for setting up the >>>>>>>>>> analytic server. We are currently working on the api manager >>>>>>>>>> analytics setup >>>>>>>>>> >>>>>>>>>> regards, >>>>>>>>>> >>>>>>>>>> Shavantha Weerasinghe >>>>>>>>>> Senior Software Engineer QA >>>>>>>>>> WSO2, Inc. >>>>>>>>>> lean.enterprise.middleware. >>>>>>>>>> http://wso2.com >>>>>>>>>> http://wso2.org >>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, May 25, 2016 at 8:10 PM, Inosh Goonewardena < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> At the moment DAS support both MyISAM and InnoDB, but configured >>>>>>>>>>> to use MyISAM by default. >>>>>>>>>>> >>>>>>>>>>> There are several differences between MYISAM and InnoDB, but >>>>>>>>>>> what is most relevant with regard to DAS is the difference in >>>>>>>>>>> concurrency. >>>>>>>>>>> Basically, MyISAM uses table-level locking and InnoDB uses row-level >>>>>>>>>>> locking. So, with MyISAM, if we are running Spark queries while >>>>>>>>>>> publishing >>>>>>>>>>> data to DAS, in higher TPS it can lead to issues due to the >>>>>>>>>>> inability of >>>>>>>>>>> obtaining the table lock by DAL layer to insert data to the table >>>>>>>>>>> while >>>>>>>>>>> Spark reading from the same table. >>>>>>>>>>> >>>>>>>>>>> However, on the other hand, with InnoDB write speed is >>>>>>>>>>> considerably slow (because it is designed to support transactions), >>>>>>>>>>> so it >>>>>>>>>>> will affect the receiver performance. >>>>>>>>>>> >>>>>>>>>>> One option we have in DAS is, we can use two DBs to to keep >>>>>>>>>>> incoming records and processed records, i.e., EVENT_STORE and >>>>>>>>>>> PROCESSED_DATA_STORE. >>>>>>>>>>> >>>>>>>>>>> For ESB Analytics, we can configure to use MyISAM for >>>>>>>>>>> EVENT_STORE and InnoDB for PROCESSED_DATA_STORE. It is because in >>>>>>>>>>> ESB >>>>>>>>>>> analytics, summarizing up to minute level is done by real time >>>>>>>>>>> analytics >>>>>>>>>>> and Spark queries will read and process data using minutely (and >>>>>>>>>>> higher) >>>>>>>>>>> tables which we can keep in PROCESSED_DATA_STORE. Since raw >>>>>>>>>>> table(which >>>>>>>>>>> data receiver writes data) is not being used by Spark queries, the >>>>>>>>>>> receiver >>>>>>>>>>> performance will not be affected. >>>>>>>>>>> >>>>>>>>>>> However, in most cases, Spark queries may written to read data >>>>>>>>>>> directly from raw tables. As mentioned above, with MyISAM this >>>>>>>>>>> could lead >>>>>>>>>>> to performance issues if data publishing and spark analytics >>>>>>>>>>> happens in >>>>>>>>>>> parallel. So considering that I think we should change the default >>>>>>>>>>> configuration to use InnoDB. WDYT? >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Thanks & Regards, >>>>>>>>>>> >>>>>>>>>>> Inosh Goonewardena >>>>>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>>>>> Mobile: +94779966317 >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> W.G. Gihan Anuruddha >>>>>>>>> Senior Software Engineer | WSO2, Inc. >>>>>>>>> M: +94772272595 >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks & Regards, >>>>>>>> >>>>>>>> Inosh Goonewardena >>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>> Mobile: +94779966317 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks & regards, >>>>>>> Nirmal >>>>>>> >>>>>>> Team Lead - WSO2 Machine Learner >>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>> Mobile: +94715779733 >>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Thanks & Regards, >>>>>> >>>>>> Inosh Goonewardena >>>>>> Associate Technical Lead- WSO2 Inc. >>>>>> Mobile: +94779966317 >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks & regards, >>>>> Nirmal >>>>> >>>>> Team Lead - WSO2 Machine Learner >>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>> Mobile: +94715779733 >>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks & Regards, >>>> >>>> Inosh Goonewardena >>>> Associate Technical Lead- WSO2 Inc. >>>> Mobile: +94779966317 >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Gokul Balakrishnan >> Senior Software Engineer, >> WSO2, Inc. http://wso2.com >> M +94 77 5935 789 | +44 7563 570502 >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Anjana Fernando* Senior Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
