so, http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/ResultSet.html#fetchMoreResults() doesn't work for you?
On Wed, Sep 16, 2015 at 12:22 PM, Malith Dhanushka <[email protected]> wrote: > Hi Sachith, > > +1 for using Hector and this will be a handy utility tool for DAS as well. > Because anyone who has Cassandra raw data can use this tool to persist them > in any file store facilitated by DAL. Given the fact, is it worth the > effort implementing this as a generic tool? WDUT? > > Thanks, > Malith > > On Wed, Sep 16, 2015 at 10:54 AM, Sachith Withana <[email protected]> > wrote: > >> Hi all, >> >> DAS migration tool is used to migrate the data from BAM deployments to >> DAS. >> >> Basically how it works is, it reads the records from Cassandra column >> families (of BAM) and inserts them to DAS analytics tables at the Data >> Access Layer (DAL) level. >> >> BAM uses Cassandra 1.x versions and the previous iteration of the tool >> was using CQL to get all the records from a given column family and insert >> them to DAL. >> >> But for large amounts of records read from BAM caused CQL to throw an >> OutOfMemory exception from the GC, since CQL is trying to load all the >> records to memory ( using select * from *tableName* ). >> >> Therefore we had to introduce pagination support by rewriting the >> migration tool using the Hector Driver. >> >> Now the hector based implementation reads in records batch-wise ( the >> batch size is configurable) and inserts to DAL thus taking out the >> possibility of running out of memory. >> >> Thanks, >> Sachith >> -- >> Sachith Withana >> Software Engineer; WSO2 Inc.; http://wso2.com >> E-mail: sachith AT wso2.com >> M: +94715518127 >> Linked-In: <http://goog_416592669> >> https://lk.linkedin.com/in/sachithwithana >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Malith Dhanushka > Senior Software Engineer - Data Technologies > *WSO2, Inc. : wso2.com <http://wso2.com/>* > *Mobile* : +94 716 506 693 > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Ramith Jayasinghe Technical Lead WSO2 Inc., http://wso2.com lean.enterprise.middleware E: [email protected] P: +94 777542851
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
