Ok, great! thanks Gokul. On Tue, May 31, 2016 at 11:26 AM, Gokul Balakrishnan <[email protected]> wrote:
> Hi Nirmal, > > We are not making the users' lives difficult nor are we exposing any of > these changes to the end user. You need not worry about any of it when you > write queries. > > If I explain the change a bit, tables that get created from spark output > are added to PROCESSED_DATA_STORE (R/W optimised) by default. There have > been no changes on this end. > > The primary record store DAS uses has been renamed to EVENT_STORE_RWO, > This record store will be used store raw incoming data. > > The only change is we now have a new records store called > EVENT_STORE_WO(optimised for writes) which can optionally be used if your > priority write performance over read (an example would be ESB analytics > events which are not required to be read by Spark). > > If a record store other than RDBMS (i.e. HBase/Cassandra), none of this > would make any impact, and the properties set for R/W optimisation will > automatically be ignored. > > As Inosh has explained, the change that needs to be done by the > *-analytics teams is to rename the store on the event sink artifacts to > ensure that there are no deployment issues with latest DAS packs. > > Thanks, > > On 31 May 2016 at 10:43, Nirmal Fernando <[email protected]> wrote: > >> Hi Inosh, >> >> If I am to write queries, do I need to think about the underlying store >> etc.? >> >> *"In the meantime we also have to analyze how the each analytics scenario >> is implemented and select the correct record store for streams and tables."* >> >> On Tue, May 31, 2016 at 10:38 AM, Inosh Goonewardena <[email protected]> >> wrote: >> >>> Hi Nirmal, >>> >>> >>> On Tue, May 31, 2016 at 9:36 AM, Nirmal Fernando <[email protected]> >>> wrote: >>> >>>> Are we making the lives of our user's difficult by this change? >>>> >>> >>> Why do you think that this change makes the user's life difficult? Is >>> the concern is about the primary event store name change? For the current >>> users who depends on DAS (esb/apim/is/iot analytics) it is just only a name >>> change in event sink artifacts. For existing users who already wrote there >>> analytics using DAS 3.0.0/3.0.1, we have to provide this piece of >>> information on migration document. >>> >>> >>> >>>> >>>> On Mon, May 30, 2016 at 8:53 PM, Inosh Goonewardena <[email protected]> >>>> wrote: >>>> >>>>> Hi Nirmal an all, >>>>> >>>>> Immediate change we need to do is we have to change the default record >>>>> store name in all event sink artifacts to EVENT_STORE_RWO (this is >>>>> the new name of our primary record store) to avoid artifact deployement >>>>> issues. >>>>> >>>>> In the meantime we also have to analyze how the each analytics >>>>> scenario is implemented and select the correct record store for streams >>>>> and >>>>> tables. Please take the esb analyics scenario I have described above as >>>>> reference. For example, in esb analytics scenario we can use >>>>> EVENT_STORE_WO record store for persist events that are received from >>>>> esb because sparks scripts are not directly reading from those tables to >>>>> do >>>>> summarization. >>>>> >>>>> On Monday, May 30, 2016, Nirmal Fernando <[email protected]> wrote: >>>>> >>>>>> Hi Inosh, >>>>>> >>>>>> What's need to be changed? Can you please provide us with more info? >>>>>> Thanks. >>>>>> >>>>>> On Mon, May 30, 2016 at 4:48 PM, Inosh Goonewardena <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Above mentioned changes have been made to DAS (product-das). >>>>>>> >>>>>>> @ ESB/APIM/IS/IOT analytics teams, >>>>>>> We need to make the necessary changes to <product> analytics >>>>>>> features accordingly as per above. >>>>>>> >>>>>>> On Fri, May 27, 2016 at 11:45 AM, Gihan Anuruddha <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Shavantha, >>>>>>>> >>>>>>>> Still, we didn't make above suggest changes to our implementation. >>>>>>>> So current release or SNAPSHOT api-analytics server doesn't have >>>>>>>> this. But >>>>>>>> anyway, relevant product team has to come with a proper load test and >>>>>>>> see >>>>>>>> which configuration will cater their analytics requirement efficiently. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Gihan >>>>>>>> >>>>>>>> On Fri, May 27, 2016 at 11:25 AM, Shavantha Weerasinghe < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Inosh >>>>>>>>> >>>>>>>>> So going forward are we to use this approach for setting up the >>>>>>>>> analytic server. We are currently working on the api manager >>>>>>>>> analytics setup >>>>>>>>> >>>>>>>>> regards, >>>>>>>>> >>>>>>>>> Shavantha Weerasinghe >>>>>>>>> Senior Software Engineer QA >>>>>>>>> WSO2, Inc. >>>>>>>>> lean.enterprise.middleware. >>>>>>>>> http://wso2.com >>>>>>>>> http://wso2.org >>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>> Fax :94 11 2145300 >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, May 25, 2016 at 8:10 PM, Inosh Goonewardena < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> At the moment DAS support both MyISAM and InnoDB, but configured >>>>>>>>>> to use MyISAM by default. >>>>>>>>>> >>>>>>>>>> There are several differences between MYISAM and InnoDB, but what >>>>>>>>>> is most relevant with regard to DAS is the difference in concurrency. >>>>>>>>>> Basically, MyISAM uses table-level locking and InnoDB uses row-level >>>>>>>>>> locking. So, with MyISAM, if we are running Spark queries while >>>>>>>>>> publishing >>>>>>>>>> data to DAS, in higher TPS it can lead to issues due to the >>>>>>>>>> inability of >>>>>>>>>> obtaining the table lock by DAL layer to insert data to the table >>>>>>>>>> while >>>>>>>>>> Spark reading from the same table. >>>>>>>>>> >>>>>>>>>> However, on the other hand, with InnoDB write speed is >>>>>>>>>> considerably slow (because it is designed to support transactions), >>>>>>>>>> so it >>>>>>>>>> will affect the receiver performance. >>>>>>>>>> >>>>>>>>>> One option we have in DAS is, we can use two DBs to to keep >>>>>>>>>> incoming records and processed records, i.e., EVENT_STORE and >>>>>>>>>> PROCESSED_DATA_STORE. >>>>>>>>>> >>>>>>>>>> For ESB Analytics, we can configure to use MyISAM for EVENT_STORE >>>>>>>>>> and InnoDB for PROCESSED_DATA_STORE. It is because in ESB analytics, >>>>>>>>>> summarizing up to minute level is done by real time analytics and >>>>>>>>>> Spark >>>>>>>>>> queries will read and process data using minutely (and higher) >>>>>>>>>> tables which >>>>>>>>>> we can keep in PROCESSED_DATA_STORE. Since raw table(which data >>>>>>>>>> receiver >>>>>>>>>> writes data) is not being used by Spark queries, the receiver >>>>>>>>>> performance >>>>>>>>>> will not be affected. >>>>>>>>>> >>>>>>>>>> However, in most cases, Spark queries may written to read data >>>>>>>>>> directly from raw tables. As mentioned above, with MyISAM this could >>>>>>>>>> lead >>>>>>>>>> to performance issues if data publishing and spark analytics happens >>>>>>>>>> in >>>>>>>>>> parallel. So considering that I think we should change the default >>>>>>>>>> configuration to use InnoDB. WDYT? >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Thanks & Regards, >>>>>>>>>> >>>>>>>>>> Inosh Goonewardena >>>>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>>>> Mobile: +94779966317 >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> W.G. Gihan Anuruddha >>>>>>>> Senior Software Engineer | WSO2, Inc. >>>>>>>> M: +94772272595 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks & Regards, >>>>>>> >>>>>>> Inosh Goonewardena >>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>> Mobile: +94779966317 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks & regards, >>>>>> Nirmal >>>>>> >>>>>> Team Lead - WSO2 Machine Learner >>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>> Mobile: +94715779733 >>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Thanks & Regards, >>>>> >>>>> Inosh Goonewardena >>>>> Associate Technical Lead- WSO2 Inc. >>>>> Mobile: +94779966317 >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> Thanks & Regards, >>> >>> Inosh Goonewardena >>> Associate Technical Lead- WSO2 Inc. >>> Mobile: +94779966317 >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Gokul Balakrishnan > Senior Software Engineer, > WSO2, Inc. http://wso2.com > M +94 77 5935 789 | +44 7563 570502 > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
