Hi Sachith, On Wed, Sep 16, 2015 at 4:13 AM, Sachith Withana <[email protected]> wrote:
> @Malith > What do you mean as a generic tool? > > @Ramith > I think that only works with Cassandra 2.0 onwards (If I'm not mistaken) > and BAM uses Cassandra 1.x versions. That was the initial issue. > I don't know about 1.x, but it does support Cassandra 1.2. On the other hand I'm not sure how far Hector supports for Cassandra 2.0. In case you're planning to use this tool with Cassandra 2.0, make sure you thoroughly test it. Unless you have already written this, IMO better to go with Datastax driver. Thanks, Bhathiya > > On Wed, Sep 16, 2015 at 1:18 PM, Ramith Jayasinghe <[email protected]> > wrote: > >> so, >> >> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/ResultSet.html#fetchMoreResults() >> doesn't work for you? >> >> On Wed, Sep 16, 2015 at 12:22 PM, Malith Dhanushka <[email protected]> >> wrote: >> >>> Hi Sachith, >>> >>> +1 for using Hector and this will be a handy utility tool for DAS as >>> well. Because anyone who has Cassandra raw data can use this tool to >>> persist them in any file store facilitated by DAL. Given the fact, is it >>> worth the effort implementing this as a generic tool? WDUT? >>> >>> Thanks, >>> Malith >>> >>> On Wed, Sep 16, 2015 at 10:54 AM, Sachith Withana <[email protected]> >>> wrote: >>> >>>> Hi all, >>>> >>>> DAS migration tool is used to migrate the data from BAM deployments to >>>> DAS. >>>> >>>> Basically how it works is, it reads the records from Cassandra column >>>> families (of BAM) and inserts them to DAS analytics tables at the Data >>>> Access Layer (DAL) level. >>>> >>>> BAM uses Cassandra 1.x versions and the previous iteration of the tool >>>> was using CQL to get all the records from a given column family and insert >>>> them to DAL. >>>> >>>> But for large amounts of records read from BAM caused CQL to throw an >>>> OutOfMemory exception from the GC, since CQL is trying to load all the >>>> records to memory ( using select * from *tableName* ). >>>> >>>> Therefore we had to introduce pagination support by rewriting the >>>> migration tool using the Hector Driver. >>>> >>>> Now the hector based implementation reads in records batch-wise ( the >>>> batch size is configurable) and inserts to DAL thus taking out the >>>> possibility of running out of memory. >>>> >>>> Thanks, >>>> Sachith >>>> -- >>>> Sachith Withana >>>> Software Engineer; WSO2 Inc.; http://wso2.com >>>> E-mail: sachith AT wso2.com >>>> M: +94715518127 >>>> Linked-In: <http://goog_416592669> >>>> https://lk.linkedin.com/in/sachithwithana >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Malith Dhanushka >>> Senior Software Engineer - Data Technologies >>> *WSO2, Inc. : wso2.com <http://wso2.com/>* >>> *Mobile* : +94 716 506 693 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Ramith Jayasinghe >> Technical Lead >> WSO2 Inc., http://wso2.com >> lean.enterprise.middleware >> >> E: [email protected] >> P: +94 777542851 >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Sachith Withana > Software Engineer; WSO2 Inc.; http://wso2.com > E-mail: sachith AT wso2.com > M: +94715518127 > Linked-In: <http://goog_416592669> > https://lk.linkedin.com/in/sachithwithana > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Bhathiya Jayasekara* *Senior Software Engineer,* *WSO2 inc., http://wso2.com <http://wso2.com>* *Phone: +94715478185* *LinkedIn: http://www.linkedin.com/in/bhathiyaj <http://www.linkedin.com/in/bhathiyaj>* *Twitter: https://twitter.com/bhathiyax <https://twitter.com/bhathiyax>* *Blog: http://movingaheadblog.blogspot.com <http://movingaheadblog.blogspot.com/>*
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
