Hi all, DAS migration tool is used to migrate the data from BAM deployments to DAS.
Basically how it works is, it reads the records from Cassandra column families (of BAM) and inserts them to DAS analytics tables at the Data Access Layer (DAL) level. BAM uses Cassandra 1.x versions and the previous iteration of the tool was using CQL to get all the records from a given column family and insert them to DAL. But for large amounts of records read from BAM caused CQL to throw an OutOfMemory exception from the GC, since CQL is trying to load all the records to memory ( using select * from *tableName* ). Therefore we had to introduce pagination support by rewriting the migration tool using the Hector Driver. Now the hector based implementation reads in records batch-wise ( the batch size is configurable) and inserts to DAL thus taking out the possibility of running out of memory. Thanks, Sachith -- Sachith Withana Software Engineer; WSO2 Inc.; http://wso2.com E-mail: sachith AT wso2.com M: +94715518127 Linked-In: <http://goog_416592669>https://lk.linkedin.com/in/sachithwithana
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
