I mean are we shipping this with DAS as a default tool? On Wed, Sep 16, 2015 at 1:43 PM, Sachith Withana <[email protected]> wrote:
> @Malith > What do you mean as a generic tool? > > @Ramith > I think that only works with Cassandra 2.0 onwards (If I'm not mistaken) > and BAM uses Cassandra 1.x versions. That was the initial issue. > > On Wed, Sep 16, 2015 at 1:18 PM, Ramith Jayasinghe <[email protected]> > wrote: > >> so, >> >> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/ResultSet.html#fetchMoreResults() >> doesn't work for you? >> >> On Wed, Sep 16, 2015 at 12:22 PM, Malith Dhanushka <[email protected]> >> wrote: >> >>> Hi Sachith, >>> >>> +1 for using Hector and this will be a handy utility tool for DAS as >>> well. Because anyone who has Cassandra raw data can use this tool to >>> persist them in any file store facilitated by DAL. Given the fact, is it >>> worth the effort implementing this as a generic tool? WDUT? >>> >>> Thanks, >>> Malith >>> >>> On Wed, Sep 16, 2015 at 10:54 AM, Sachith Withana <[email protected]> >>> wrote: >>> >>>> Hi all, >>>> >>>> DAS migration tool is used to migrate the data from BAM deployments to >>>> DAS. >>>> >>>> Basically how it works is, it reads the records from Cassandra column >>>> families (of BAM) and inserts them to DAS analytics tables at the Data >>>> Access Layer (DAL) level. >>>> >>>> BAM uses Cassandra 1.x versions and the previous iteration of the tool >>>> was using CQL to get all the records from a given column family and insert >>>> them to DAL. >>>> >>>> But for large amounts of records read from BAM caused CQL to throw an >>>> OutOfMemory exception from the GC, since CQL is trying to load all the >>>> records to memory ( using select * from *tableName* ). >>>> >>>> Therefore we had to introduce pagination support by rewriting the >>>> migration tool using the Hector Driver. >>>> >>>> Now the hector based implementation reads in records batch-wise ( the >>>> batch size is configurable) and inserts to DAL thus taking out the >>>> possibility of running out of memory. >>>> >>>> Thanks, >>>> Sachith >>>> -- >>>> Sachith Withana >>>> Software Engineer; WSO2 Inc.; http://wso2.com >>>> E-mail: sachith AT wso2.com >>>> M: +94715518127 >>>> Linked-In: <http://goog_416592669> >>>> https://lk.linkedin.com/in/sachithwithana >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Malith Dhanushka >>> Senior Software Engineer - Data Technologies >>> *WSO2, Inc. : wso2.com <http://wso2.com/>* >>> *Mobile* : +94 716 506 693 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Ramith Jayasinghe >> Technical Lead >> WSO2 Inc., http://wso2.com >> lean.enterprise.middleware >> >> E: [email protected] >> P: +94 777542851 >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Sachith Withana > Software Engineer; WSO2 Inc.; http://wso2.com > E-mail: sachith AT wso2.com > M: +94715518127 > Linked-In: <http://goog_416592669> > https://lk.linkedin.com/in/sachithwithana > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Malith Dhanushka Senior Software Engineer - Data Technologies *WSO2, Inc. : wso2.com <http://wso2.com/>* *Mobile* : +94 716 506 693
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
