On Fri, Sep 18, 2015 at 10:29 AM, Iranga Muthuthanthri <[email protected]> wrote:
> > > On Wed, Sep 16, 2015 at 12:22 PM, Malith Dhanushka <[email protected]> > wrote: > >> Hi Sachith, >> >> +1 for using Hector and this will be a handy utility tool for DAS as >> well. Because anyone who has Cassandra raw data can use this tool to >> persist them in any file store facilitated by DAL. Given the fact, is it >> worth the effort implementing this as a generic tool? WDUT? >> > > for clarification on 'any file store facilitated by DAL ' means RDBMS, > HBase and Cassandra ? > That's correct Iranga. >> Thanks, >> Malith >> >> On Wed, Sep 16, 2015 at 10:54 AM, Sachith Withana <[email protected]> >> wrote: >> >>> Hi all, >>> >>> DAS migration tool is used to migrate the data from BAM deployments to >>> DAS. >>> >>> Basically how it works is, it reads the records from Cassandra column >>> families (of BAM) and inserts them to DAS analytics tables at the Data >>> Access Layer (DAL) level. >>> >>> BAM uses Cassandra 1.x versions and the previous iteration of the tool >>> was using CQL to get all the records from a given column family and insert >>> them to DAL. >>> >>> But for large amounts of records read from BAM caused CQL to throw an >>> OutOfMemory exception from the GC, since CQL is trying to load all the >>> records to memory ( using select * from *tableName* ). >>> >>> Therefore we had to introduce pagination support by rewriting the >>> migration tool using the Hector Driver. >>> >>> Now the hector based implementation reads in records batch-wise ( the >>> batch size is configurable) and inserts to DAL thus taking out the >>> possibility of running out of memory. >>> >>> Thanks, >>> Sachith >>> -- >>> Sachith Withana >>> Software Engineer; WSO2 Inc.; http://wso2.com >>> E-mail: sachith AT wso2.com >>> M: +94715518127 >>> Linked-In: <http://goog_416592669> >>> https://lk.linkedin.com/in/sachithwithana >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Malith Dhanushka >> Senior Software Engineer - Data Technologies >> *WSO2, Inc. : wso2.com <http://wso2.com/>* >> *Mobile* : +94 716 506 693 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Thanks & Regards > > Iranga Muthuthanthri > (M) -0777-255773 > Team Product Management > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Malith Dhanushka Senior Software Engineer - Data Technologies *WSO2, Inc. : wso2.com <http://wso2.com/>* *Mobile* : +94 716 506 693
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
