>From what I know, for any new development DataStax recommends using CQL over Hector. (unsure if that's relevant here)
On Sun, Sep 20, 2015 at 8:49 AM, Bhathiya Jayasekara <[email protected]> wrote: > Hi Sachith, > > On Wed, Sep 16, 2015 at 4:13 AM, Sachith Withana <[email protected]> wrote: > >> @Malith >> What do you mean as a generic tool? >> >> @Ramith >> I think that only works with Cassandra 2.0 onwards (If I'm not mistaken) >> and BAM uses Cassandra 1.x versions. That was the initial issue. >> > > I don't know about 1.x, but it does support Cassandra 1.2. On the other > hand I'm not sure how far Hector supports for Cassandra 2.0. In case you're > planning to use this tool with Cassandra 2.0, make sure you thoroughly test > it. Unless you have already written this, IMO better to go with Datastax > driver. > > Thanks, > Bhathiya > > >> >> On Wed, Sep 16, 2015 at 1:18 PM, Ramith Jayasinghe <[email protected]> >> wrote: >> >>> so, >>> >>> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/ResultSet.html#fetchMoreResults() >>> doesn't work for you? >>> >>> On Wed, Sep 16, 2015 at 12:22 PM, Malith Dhanushka <[email protected]> >>> wrote: >>> >>>> Hi Sachith, >>>> >>>> +1 for using Hector and this will be a handy utility tool for DAS as >>>> well. Because anyone who has Cassandra raw data can use this tool to >>>> persist them in any file store facilitated by DAL. Given the fact, is it >>>> worth the effort implementing this as a generic tool? WDUT? >>>> >>>> Thanks, >>>> Malith >>>> >>>> On Wed, Sep 16, 2015 at 10:54 AM, Sachith Withana <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> DAS migration tool is used to migrate the data from BAM deployments to >>>>> DAS. >>>>> >>>>> Basically how it works is, it reads the records from Cassandra column >>>>> families (of BAM) and inserts them to DAS analytics tables at the Data >>>>> Access Layer (DAL) level. >>>>> >>>>> BAM uses Cassandra 1.x versions and the previous iteration of the tool >>>>> was using CQL to get all the records from a given column family and insert >>>>> them to DAL. >>>>> >>>>> But for large amounts of records read from BAM caused CQL to throw an >>>>> OutOfMemory exception from the GC, since CQL is trying to load all the >>>>> records to memory ( using select * from *tableName* ). >>>>> >>>>> Therefore we had to introduce pagination support by rewriting the >>>>> migration tool using the Hector Driver. >>>>> >>>>> Now the hector based implementation reads in records batch-wise ( the >>>>> batch size is configurable) and inserts to DAL thus taking out the >>>>> possibility of running out of memory. >>>>> >>>>> Thanks, >>>>> Sachith >>>>> -- >>>>> Sachith Withana >>>>> Software Engineer; WSO2 Inc.; http://wso2.com >>>>> E-mail: sachith AT wso2.com >>>>> M: +94715518127 >>>>> Linked-In: <http://goog_416592669> >>>>> https://lk.linkedin.com/in/sachithwithana >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Malith Dhanushka >>>> Senior Software Engineer - Data Technologies >>>> *WSO2, Inc. : wso2.com <http://wso2.com/>* >>>> *Mobile* : +94 716 506 693 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Ramith Jayasinghe >>> Technical Lead >>> WSO2 Inc., http://wso2.com >>> lean.enterprise.middleware >>> >>> E: [email protected] >>> P: +94 777542851 >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Sachith Withana >> Software Engineer; WSO2 Inc.; http://wso2.com >> E-mail: sachith AT wso2.com >> M: +94715518127 >> Linked-In: <http://goog_416592669> >> https://lk.linkedin.com/in/sachithwithana >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Bhathiya Jayasekara* > *Senior Software Engineer,* > *WSO2 inc., http://wso2.com <http://wso2.com>* > > *Phone: +94715478185 <%2B94715478185>* > *LinkedIn: http://www.linkedin.com/in/bhathiyaj > <http://www.linkedin.com/in/bhathiyaj>* > *Twitter: https://twitter.com/bhathiyax <https://twitter.com/bhathiyax>* > *Blog: http://movingaheadblog.blogspot.com > <http://movingaheadblog.blogspot.com/>* > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Ramith Jayasinghe Technical Lead WSO2 Inc., http://wso2.com lean.enterprise.middleware E: [email protected] P: +94 777542851
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
