Hi Sachith, +1 for using Hector and this will be a handy utility tool for DAS as well. Because anyone who has Cassandra raw data can use this tool to persist them in any file store facilitated by DAL. Given the fact, is it worth the effort implementing this as a generic tool? WDUT?
Thanks, Malith On Wed, Sep 16, 2015 at 10:54 AM, Sachith Withana <[email protected]> wrote: > Hi all, > > DAS migration tool is used to migrate the data from BAM deployments to > DAS. > > Basically how it works is, it reads the records from Cassandra column > families (of BAM) and inserts them to DAS analytics tables at the Data > Access Layer (DAL) level. > > BAM uses Cassandra 1.x versions and the previous iteration of the tool was > using CQL to get all the records from a given column family and insert them > to DAL. > > But for large amounts of records read from BAM caused CQL to throw an > OutOfMemory exception from the GC, since CQL is trying to load all the > records to memory ( using select * from *tableName* ). > > Therefore we had to introduce pagination support by rewriting the > migration tool using the Hector Driver. > > Now the hector based implementation reads in records batch-wise ( the > batch size is configurable) and inserts to DAL thus taking out the > possibility of running out of memory. > > Thanks, > Sachith > -- > Sachith Withana > Software Engineer; WSO2 Inc.; http://wso2.com > E-mail: sachith AT wso2.com > M: +94715518127 > Linked-In: <http://goog_416592669> > https://lk.linkedin.com/in/sachithwithana > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Malith Dhanushka Senior Software Engineer - Data Technologies *WSO2, Inc. : wso2.com <http://wso2.com/>* *Mobile* : +94 716 506 693
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
