Hi Inosh, What's need to be changed? Can you please provide us with more info? Thanks.
On Mon, May 30, 2016 at 4:48 PM, Inosh Goonewardena <[email protected]> wrote: > Above mentioned changes have been made to DAS (product-das). > > @ ESB/APIM/IS/IOT analytics teams, > We need to make the necessary changes to <product> analytics features > accordingly as per above. > > On Fri, May 27, 2016 at 11:45 AM, Gihan Anuruddha <[email protected]> wrote: > >> Hi Shavantha, >> >> Still, we didn't make above suggest changes to our implementation. So >> current release or SNAPSHOT api-analytics server doesn't have this. But >> anyway, relevant product team has to come with a proper load test and see >> which configuration will cater their analytics requirement efficiently. >> >> Regards, >> Gihan >> >> On Fri, May 27, 2016 at 11:25 AM, Shavantha Weerasinghe < >> [email protected]> wrote: >> >>> Hi Inosh >>> >>> So going forward are we to use this approach for setting up the analytic >>> server. We are currently working on the api manager analytics setup >>> >>> regards, >>> >>> Shavantha Weerasinghe >>> Senior Software Engineer QA >>> WSO2, Inc. >>> lean.enterprise.middleware. >>> http://wso2.com >>> http://wso2.org >>> Tel : 94 11 214 5345 >>> Fax :94 11 2145300 >>> >>> >>> On Wed, May 25, 2016 at 8:10 PM, Inosh Goonewardena <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> At the moment DAS support both MyISAM and InnoDB, but configured to use >>>> MyISAM by default. >>>> >>>> There are several differences between MYISAM and InnoDB, but what is >>>> most relevant with regard to DAS is the difference in concurrency. >>>> Basically, MyISAM uses table-level locking and InnoDB uses row-level >>>> locking. So, with MyISAM, if we are running Spark queries while publishing >>>> data to DAS, in higher TPS it can lead to issues due to the inability of >>>> obtaining the table lock by DAL layer to insert data to the table while >>>> Spark reading from the same table. >>>> >>>> However, on the other hand, with InnoDB write speed is considerably >>>> slow (because it is designed to support transactions), so it will affect >>>> the receiver performance. >>>> >>>> One option we have in DAS is, we can use two DBs to to keep incoming >>>> records and processed records, i.e., EVENT_STORE and PROCESSED_DATA_STORE. >>>> >>>> For ESB Analytics, we can configure to use MyISAM for EVENT_STORE and >>>> InnoDB for PROCESSED_DATA_STORE. It is because in ESB analytics, >>>> summarizing up to minute level is done by real time analytics and Spark >>>> queries will read and process data using minutely (and higher) tables which >>>> we can keep in PROCESSED_DATA_STORE. Since raw table(which data receiver >>>> writes data) is not being used by Spark queries, the receiver performance >>>> will not be affected. >>>> >>>> However, in most cases, Spark queries may written to read data directly >>>> from raw tables. As mentioned above, with MyISAM this could lead to >>>> performance issues if data publishing and spark analytics happens in >>>> parallel. So considering that I think we should change the default >>>> configuration to use InnoDB. WDYT? >>>> >>>> -- >>>> Thanks & Regards, >>>> >>>> Inosh Goonewardena >>>> Associate Technical Lead- WSO2 Inc. >>>> Mobile: +94779966317 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> W.G. Gihan Anuruddha >> Senior Software Engineer | WSO2, Inc. >> M: +94772272595 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Thanks & Regards, > > Inosh Goonewardena > Associate Technical Lead- WSO2 Inc. > Mobile: +94779966317 > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
