Hi Anjana, On Wed, Jun 1, 2016 at 10:04 AM, Anjana Fernando <[email protected]> wrote:
> Hi Inosh, > > Actually can we keep the record store name "EVENT_STORE" as it is, which > would resemble the suggest "EVENT_STORE_RWO" one here, and just add the > extra "EVENT_STORE_RO". So with this, this will not break any existing > solutions, which have explicitly already mentioned the record store as > "EVENT_STORE" in their configurations. Only required people in the future, > e.g. ESB Anallytics, can use the new "EVENT_STORE_RO" one. > Yes. We can name it that way. That was actually my first thought too. Will do the changes now. > > Cheers, > Anjana. > > On Tue, May 31, 2016 at 11:32 AM, Nirmal Fernando <[email protected]> wrote: > >> Ok, great! thanks Gokul. >> >> On Tue, May 31, 2016 at 11:26 AM, Gokul Balakrishnan <[email protected]> >> wrote: >> >>> Hi Nirmal, >>> >>> We are not making the users' lives difficult nor are we exposing any of >>> these changes to the end user. You need not worry about any of it when you >>> write queries. >>> >>> If I explain the change a bit, tables that get created from spark output >>> are added to PROCESSED_DATA_STORE (R/W optimised) by default. There have >>> been no changes on this end. >>> >>> The primary record store DAS uses has been renamed to EVENT_STORE_RWO, >>> This record store will be used store raw incoming data. >>> >>> The only change is we now have a new records store called >>> EVENT_STORE_WO(optimised for writes) which can optionally be used if your >>> priority write performance over read (an example would be ESB analytics >>> events which are not required to be read by Spark). >>> >>> If a record store other than RDBMS (i.e. HBase/Cassandra), none of this >>> would make any impact, and the properties set for R/W optimisation will >>> automatically be ignored. >>> >>> As Inosh has explained, the change that needs to be done by the >>> *-analytics teams is to rename the store on the event sink artifacts to >>> ensure that there are no deployment issues with latest DAS packs. >>> >>> Thanks, >>> >>> On 31 May 2016 at 10:43, Nirmal Fernando <[email protected]> wrote: >>> >>>> Hi Inosh, >>>> >>>> If I am to write queries, do I need to think about the underlying store >>>> etc.? >>>> >>>> *"In the meantime we also have to analyze how the each analytics >>>> scenario is implemented and select the correct record store for streams and >>>> tables."* >>>> >>>> On Tue, May 31, 2016 at 10:38 AM, Inosh Goonewardena <[email protected]> >>>> wrote: >>>> >>>>> Hi Nirmal, >>>>> >>>>> >>>>> On Tue, May 31, 2016 at 9:36 AM, Nirmal Fernando <[email protected]> >>>>> wrote: >>>>> >>>>>> Are we making the lives of our user's difficult by this change? >>>>>> >>>>> >>>>> Why do you think that this change makes the user's life difficult? Is >>>>> the concern is about the primary event store name change? For the current >>>>> users who depends on DAS (esb/apim/is/iot analytics) it is just only a >>>>> name >>>>> change in event sink artifacts. For existing users who already wrote there >>>>> analytics using DAS 3.0.0/3.0.1, we have to provide this piece of >>>>> information on migration document. >>>>> >>>>> >>>>> >>>>>> >>>>>> On Mon, May 30, 2016 at 8:53 PM, Inosh Goonewardena <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Nirmal an all, >>>>>>> >>>>>>> Immediate change we need to do is we have to change the default >>>>>>> record store name in all event sink artifacts to EVENT_STORE_RWO (this >>>>>>> is the new name of our primary record store) to avoid artifact >>>>>>> deployement >>>>>>> issues. >>>>>>> >>>>>>> In the meantime we also have to analyze how the each analytics >>>>>>> scenario is implemented and select the correct record store for streams >>>>>>> and >>>>>>> tables. Please take the esb analyics scenario I have described above as >>>>>>> reference. For example, in esb analytics scenario we can use >>>>>>> EVENT_STORE_WO record store for persist events that are received >>>>>>> from esb because sparks scripts are not directly reading from those >>>>>>> tables >>>>>>> to do summarization. >>>>>>> >>>>>>> On Monday, May 30, 2016, Nirmal Fernando <[email protected]> wrote: >>>>>>> >>>>>>>> Hi Inosh, >>>>>>>> >>>>>>>> What's need to be changed? Can you please provide us with more >>>>>>>> info? Thanks. >>>>>>>> >>>>>>>> On Mon, May 30, 2016 at 4:48 PM, Inosh Goonewardena <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Above mentioned changes have been made to DAS (product-das). >>>>>>>>> >>>>>>>>> @ ESB/APIM/IS/IOT analytics teams, >>>>>>>>> We need to make the necessary changes to <product> analytics >>>>>>>>> features accordingly as per above. >>>>>>>>> >>>>>>>>> On Fri, May 27, 2016 at 11:45 AM, Gihan Anuruddha <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Shavantha, >>>>>>>>>> >>>>>>>>>> Still, we didn't make above suggest changes to our >>>>>>>>>> implementation. So current release or SNAPSHOT api-analytics server >>>>>>>>>> doesn't have this. But anyway, relevant product team has to come >>>>>>>>>> with a >>>>>>>>>> proper load test and see which configuration will cater their >>>>>>>>>> analytics >>>>>>>>>> requirement efficiently. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Gihan >>>>>>>>>> >>>>>>>>>> On Fri, May 27, 2016 at 11:25 AM, Shavantha Weerasinghe < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Inosh >>>>>>>>>>> >>>>>>>>>>> So going forward are we to use this approach for setting up the >>>>>>>>>>> analytic server. We are currently working on the api manager >>>>>>>>>>> analytics setup >>>>>>>>>>> >>>>>>>>>>> regards, >>>>>>>>>>> >>>>>>>>>>> Shavantha Weerasinghe >>>>>>>>>>> Senior Software Engineer QA >>>>>>>>>>> WSO2, Inc. >>>>>>>>>>> lean.enterprise.middleware. >>>>>>>>>>> http://wso2.com >>>>>>>>>>> http://wso2.org >>>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, May 25, 2016 at 8:10 PM, Inosh Goonewardena < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> At the moment DAS support both MyISAM and InnoDB, but >>>>>>>>>>>> configured to use MyISAM by default. >>>>>>>>>>>> >>>>>>>>>>>> There are several differences between MYISAM and InnoDB, but >>>>>>>>>>>> what is most relevant with regard to DAS is the difference in >>>>>>>>>>>> concurrency. >>>>>>>>>>>> Basically, MyISAM uses table-level locking and InnoDB uses >>>>>>>>>>>> row-level >>>>>>>>>>>> locking. So, with MyISAM, if we are running Spark queries while >>>>>>>>>>>> publishing >>>>>>>>>>>> data to DAS, in higher TPS it can lead to issues due to the >>>>>>>>>>>> inability of >>>>>>>>>>>> obtaining the table lock by DAL layer to insert data to the table >>>>>>>>>>>> while >>>>>>>>>>>> Spark reading from the same table. >>>>>>>>>>>> >>>>>>>>>>>> However, on the other hand, with InnoDB write speed is >>>>>>>>>>>> considerably slow (because it is designed to support >>>>>>>>>>>> transactions), so it >>>>>>>>>>>> will affect the receiver performance. >>>>>>>>>>>> >>>>>>>>>>>> One option we have in DAS is, we can use two DBs to to keep >>>>>>>>>>>> incoming records and processed records, i.e., EVENT_STORE and >>>>>>>>>>>> PROCESSED_DATA_STORE. >>>>>>>>>>>> >>>>>>>>>>>> For ESB Analytics, we can configure to use MyISAM for >>>>>>>>>>>> EVENT_STORE and InnoDB for PROCESSED_DATA_STORE. It is because in >>>>>>>>>>>> ESB >>>>>>>>>>>> analytics, summarizing up to minute level is done by real time >>>>>>>>>>>> analytics >>>>>>>>>>>> and Spark queries will read and process data using minutely (and >>>>>>>>>>>> higher) >>>>>>>>>>>> tables which we can keep in PROCESSED_DATA_STORE. Since raw >>>>>>>>>>>> table(which >>>>>>>>>>>> data receiver writes data) is not being used by Spark queries, the >>>>>>>>>>>> receiver >>>>>>>>>>>> performance will not be affected. >>>>>>>>>>>> >>>>>>>>>>>> However, in most cases, Spark queries may written to read data >>>>>>>>>>>> directly from raw tables. As mentioned above, with MyISAM this >>>>>>>>>>>> could lead >>>>>>>>>>>> to performance issues if data publishing and spark analytics >>>>>>>>>>>> happens in >>>>>>>>>>>> parallel. So considering that I think we should change the default >>>>>>>>>>>> configuration to use InnoDB. WDYT? >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Thanks & Regards, >>>>>>>>>>>> >>>>>>>>>>>> Inosh Goonewardena >>>>>>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>>>>>> Mobile: +94779966317 >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> W.G. Gihan Anuruddha >>>>>>>>>> Senior Software Engineer | WSO2, Inc. >>>>>>>>>> M: +94772272595 >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanks & Regards, >>>>>>>>> >>>>>>>>> Inosh Goonewardena >>>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>>> Mobile: +94779966317 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks & regards, >>>>>>>> Nirmal >>>>>>>> >>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>> Mobile: +94715779733 >>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks & Regards, >>>>>>> >>>>>>> Inosh Goonewardena >>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>> Mobile: +94779966317 >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks & regards, >>>>>> Nirmal >>>>>> >>>>>> Team Lead - WSO2 Machine Learner >>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>> Mobile: +94715779733 >>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks & Regards, >>>>> >>>>> Inosh Goonewardena >>>>> Associate Technical Lead- WSO2 Inc. >>>>> Mobile: +94779966317 >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Gokul Balakrishnan >>> Senior Software Engineer, >>> WSO2, Inc. http://wso2.com >>> M +94 77 5935 789 | +44 7563 570502 >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Anjana Fernando* > Senior Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- Thanks & Regards, Inosh Goonewardena Associate Technical Lead- WSO2 Inc. Mobile: +94779966317
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
