Great! .. Cheers, Anjana.
On Wed, Jun 1, 2016 at 10:18 AM, Inosh Goonewardena <[email protected]> wrote: > Hi Anjana, > > On Wed, Jun 1, 2016 at 10:04 AM, Anjana Fernando <[email protected]> wrote: > >> Hi Inosh, >> >> Actually can we keep the record store name "EVENT_STORE" as it is, which >> would resemble the suggest "EVENT_STORE_RWO" one here, and just add the >> extra "EVENT_STORE_RO". So with this, this will not break any existing >> solutions, which have explicitly already mentioned the record store as >> "EVENT_STORE" in their configurations. Only required people in the future, >> e.g. ESB Anallytics, can use the new "EVENT_STORE_RO" one. >> > > Yes. We can name it that way. That was actually my first thought too. Will > do the changes now. > > >> >> Cheers, >> Anjana. >> >> On Tue, May 31, 2016 at 11:32 AM, Nirmal Fernando <[email protected]> >> wrote: >> >>> Ok, great! thanks Gokul. >>> >>> On Tue, May 31, 2016 at 11:26 AM, Gokul Balakrishnan <[email protected]> >>> wrote: >>> >>>> Hi Nirmal, >>>> >>>> We are not making the users' lives difficult nor are we exposing any of >>>> these changes to the end user. You need not worry about any of it when you >>>> write queries. >>>> >>>> If I explain the change a bit, tables that get created from spark >>>> output are added to PROCESSED_DATA_STORE (R/W optimised) by default. There >>>> have been no changes on this end. >>>> >>>> The primary record store DAS uses has been renamed to EVENT_STORE_RWO, >>>> This record store will be used store raw incoming data. >>>> >>>> The only change is we now have a new records store called >>>> EVENT_STORE_WO(optimised for writes) which can optionally be used if your >>>> priority write performance over read (an example would be ESB analytics >>>> events which are not required to be read by Spark). >>>> >>>> If a record store other than RDBMS (i.e. HBase/Cassandra), none of this >>>> would make any impact, and the properties set for R/W optimisation will >>>> automatically be ignored. >>>> >>>> As Inosh has explained, the change that needs to be done by the >>>> *-analytics teams is to rename the store on the event sink artifacts to >>>> ensure that there are no deployment issues with latest DAS packs. >>>> >>>> Thanks, >>>> >>>> On 31 May 2016 at 10:43, Nirmal Fernando <[email protected]> wrote: >>>> >>>>> Hi Inosh, >>>>> >>>>> If I am to write queries, do I need to think about the underlying >>>>> store etc.? >>>>> >>>>> *"In the meantime we also have to analyze how the each analytics >>>>> scenario is implemented and select the correct record store for streams >>>>> and >>>>> tables."* >>>>> >>>>> On Tue, May 31, 2016 at 10:38 AM, Inosh Goonewardena <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Nirmal, >>>>>> >>>>>> >>>>>> On Tue, May 31, 2016 at 9:36 AM, Nirmal Fernando <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Are we making the lives of our user's difficult by this change? >>>>>>> >>>>>> >>>>>> Why do you think that this change makes the user's life difficult? Is >>>>>> the concern is about the primary event store name change? For the current >>>>>> users who depends on DAS (esb/apim/is/iot analytics) it is just only a >>>>>> name >>>>>> change in event sink artifacts. For existing users who already wrote >>>>>> there >>>>>> analytics using DAS 3.0.0/3.0.1, we have to provide this piece of >>>>>> information on migration document. >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> On Mon, May 30, 2016 at 8:53 PM, Inosh Goonewardena <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Nirmal an all, >>>>>>>> >>>>>>>> Immediate change we need to do is we have to change the default >>>>>>>> record store name in all event sink artifacts to EVENT_STORE_RWO (this >>>>>>>> is the new name of our primary record store) to avoid artifact >>>>>>>> deployement >>>>>>>> issues. >>>>>>>> >>>>>>>> In the meantime we also have to analyze how the each analytics >>>>>>>> scenario is implemented and select the correct record store for >>>>>>>> streams and >>>>>>>> tables. Please take the esb analyics scenario I have described above as >>>>>>>> reference. For example, in esb analytics scenario we can use >>>>>>>> EVENT_STORE_WO record store for persist events that are received >>>>>>>> from esb because sparks scripts are not directly reading from those >>>>>>>> tables >>>>>>>> to do summarization. >>>>>>>> >>>>>>>> On Monday, May 30, 2016, Nirmal Fernando <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Inosh, >>>>>>>>> >>>>>>>>> What's need to be changed? Can you please provide us with more >>>>>>>>> info? Thanks. >>>>>>>>> >>>>>>>>> On Mon, May 30, 2016 at 4:48 PM, Inosh Goonewardena < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Above mentioned changes have been made to DAS (product-das). >>>>>>>>>> >>>>>>>>>> @ ESB/APIM/IS/IOT analytics teams, >>>>>>>>>> We need to make the necessary changes to <product> analytics >>>>>>>>>> features accordingly as per above. >>>>>>>>>> >>>>>>>>>> On Fri, May 27, 2016 at 11:45 AM, Gihan Anuruddha <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Hi Shavantha, >>>>>>>>>>> >>>>>>>>>>> Still, we didn't make above suggest changes to our >>>>>>>>>>> implementation. So current release or SNAPSHOT api-analytics server >>>>>>>>>>> doesn't have this. But anyway, relevant product team has to come >>>>>>>>>>> with a >>>>>>>>>>> proper load test and see which configuration will cater their >>>>>>>>>>> analytics >>>>>>>>>>> requirement efficiently. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Gihan >>>>>>>>>>> >>>>>>>>>>> On Fri, May 27, 2016 at 11:25 AM, Shavantha Weerasinghe < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Inosh >>>>>>>>>>>> >>>>>>>>>>>> So going forward are we to use this approach for setting up the >>>>>>>>>>>> analytic server. We are currently working on the api manager >>>>>>>>>>>> analytics setup >>>>>>>>>>>> >>>>>>>>>>>> regards, >>>>>>>>>>>> >>>>>>>>>>>> Shavantha Weerasinghe >>>>>>>>>>>> Senior Software Engineer QA >>>>>>>>>>>> WSO2, Inc. >>>>>>>>>>>> lean.enterprise.middleware. >>>>>>>>>>>> http://wso2.com >>>>>>>>>>>> http://wso2.org >>>>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, May 25, 2016 at 8:10 PM, Inosh Goonewardena < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> At the moment DAS support both MyISAM and InnoDB, but >>>>>>>>>>>>> configured to use MyISAM by default. >>>>>>>>>>>>> >>>>>>>>>>>>> There are several differences between MYISAM and InnoDB, but >>>>>>>>>>>>> what is most relevant with regard to DAS is the difference in >>>>>>>>>>>>> concurrency. >>>>>>>>>>>>> Basically, MyISAM uses table-level locking and InnoDB uses >>>>>>>>>>>>> row-level >>>>>>>>>>>>> locking. So, with MyISAM, if we are running Spark queries while >>>>>>>>>>>>> publishing >>>>>>>>>>>>> data to DAS, in higher TPS it can lead to issues due to the >>>>>>>>>>>>> inability of >>>>>>>>>>>>> obtaining the table lock by DAL layer to insert data to the table >>>>>>>>>>>>> while >>>>>>>>>>>>> Spark reading from the same table. >>>>>>>>>>>>> >>>>>>>>>>>>> However, on the other hand, with InnoDB write speed is >>>>>>>>>>>>> considerably slow (because it is designed to support >>>>>>>>>>>>> transactions), so it >>>>>>>>>>>>> will affect the receiver performance. >>>>>>>>>>>>> >>>>>>>>>>>>> One option we have in DAS is, we can use two DBs to to keep >>>>>>>>>>>>> incoming records and processed records, i.e., EVENT_STORE and >>>>>>>>>>>>> PROCESSED_DATA_STORE. >>>>>>>>>>>>> >>>>>>>>>>>>> For ESB Analytics, we can configure to use MyISAM for >>>>>>>>>>>>> EVENT_STORE and InnoDB for PROCESSED_DATA_STORE. It is because in >>>>>>>>>>>>> ESB >>>>>>>>>>>>> analytics, summarizing up to minute level is done by real time >>>>>>>>>>>>> analytics >>>>>>>>>>>>> and Spark queries will read and process data using minutely (and >>>>>>>>>>>>> higher) >>>>>>>>>>>>> tables which we can keep in PROCESSED_DATA_STORE. Since raw >>>>>>>>>>>>> table(which >>>>>>>>>>>>> data receiver writes data) is not being used by Spark queries, >>>>>>>>>>>>> the receiver >>>>>>>>>>>>> performance will not be affected. >>>>>>>>>>>>> >>>>>>>>>>>>> However, in most cases, Spark queries may written to read data >>>>>>>>>>>>> directly from raw tables. As mentioned above, with MyISAM this >>>>>>>>>>>>> could lead >>>>>>>>>>>>> to performance issues if data publishing and spark analytics >>>>>>>>>>>>> happens in >>>>>>>>>>>>> parallel. So considering that I think we should change the default >>>>>>>>>>>>> configuration to use InnoDB. WDYT? >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Thanks & Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Inosh Goonewardena >>>>>>>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>>>>>>> Mobile: +94779966317 >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> W.G. Gihan Anuruddha >>>>>>>>>>> Senior Software Engineer | WSO2, Inc. >>>>>>>>>>> M: +94772272595 >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Thanks & Regards, >>>>>>>>>> >>>>>>>>>> Inosh Goonewardena >>>>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>>>> Mobile: +94779966317 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks & regards, >>>>>>>>> Nirmal >>>>>>>>> >>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>> Mobile: +94715779733 >>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks & Regards, >>>>>>>> >>>>>>>> Inosh Goonewardena >>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>> Mobile: +94779966317 >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks & regards, >>>>>>> Nirmal >>>>>>> >>>>>>> Team Lead - WSO2 Machine Learner >>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>> Mobile: +94715779733 >>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks & Regards, >>>>>> >>>>>> Inosh Goonewardena >>>>>> Associate Technical Lead- WSO2 Inc. >>>>>> Mobile: +94779966317 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks & regards, >>>>> Nirmal >>>>> >>>>> Team Lead - WSO2 Machine Learner >>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>> Mobile: +94715779733 >>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Gokul Balakrishnan >>>> Senior Software Engineer, >>>> WSO2, Inc. http://wso2.com >>>> M +94 77 5935 789 | +44 7563 570502 >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *Anjana Fernando* >> Senior Technical Lead >> WSO2 Inc. | http://wso2.com >> lean . enterprise . middleware >> > > > > -- > Thanks & Regards, > > Inosh Goonewardena > Associate Technical Lead- WSO2 Inc. > Mobile: +94779966317 > -- *Anjana Fernando* Senior Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
