Yes. Hector is deprecated now and DataStax recommends using CQL. But I went ahead and used Hector for the implementation since it is for a Cassandra 1.x and BAM also uses Hector driver in its impementation.
On Mon, Sep 21, 2015 at 11:02 AM, Ramith Jayasinghe <[email protected]> wrote: > From what I know, for any new development DataStax recommends using CQL > over Hector. (unsure if that's relevant here) > > On Sun, Sep 20, 2015 at 8:49 AM, Bhathiya Jayasekara <[email protected]> > wrote: > >> Hi Sachith, >> >> On Wed, Sep 16, 2015 at 4:13 AM, Sachith Withana <[email protected]> >> wrote: >> >>> @Malith >>> What do you mean as a generic tool? >>> >>> @Ramith >>> I think that only works with Cassandra 2.0 onwards (If I'm not mistaken) >>> and BAM uses Cassandra 1.x versions. That was the initial issue. >>> >> >> I don't know about 1.x, but it does support Cassandra 1.2. On the other >> hand I'm not sure how far Hector supports for Cassandra 2.0. In case you're >> planning to use this tool with Cassandra 2.0, make sure you thoroughly test >> it. Unless you have already written this, IMO better to go with Datastax >> driver. >> >> Thanks, >> Bhathiya >> >> >>> >>> On Wed, Sep 16, 2015 at 1:18 PM, Ramith Jayasinghe <[email protected]> >>> wrote: >>> >>>> so, >>>> >>>> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/ResultSet.html#fetchMoreResults() >>>> doesn't work for you? >>>> >>>> On Wed, Sep 16, 2015 at 12:22 PM, Malith Dhanushka <[email protected]> >>>> wrote: >>>> >>>>> Hi Sachith, >>>>> >>>>> +1 for using Hector and this will be a handy utility tool for DAS as >>>>> well. Because anyone who has Cassandra raw data can use this tool to >>>>> persist them in any file store facilitated by DAL. Given the fact, is it >>>>> worth the effort implementing this as a generic tool? WDUT? >>>>> >>>>> Thanks, >>>>> Malith >>>>> >>>>> On Wed, Sep 16, 2015 at 10:54 AM, Sachith Withana <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> DAS migration tool is used to migrate the data from BAM deployments >>>>>> to DAS. >>>>>> >>>>>> Basically how it works is, it reads the records from Cassandra column >>>>>> families (of BAM) and inserts them to DAS analytics tables at the Data >>>>>> Access Layer (DAL) level. >>>>>> >>>>>> BAM uses Cassandra 1.x versions and the previous iteration of the >>>>>> tool was using CQL to get all the records from a given column family and >>>>>> insert them to DAL. >>>>>> >>>>>> But for large amounts of records read from BAM caused CQL to throw an >>>>>> OutOfMemory exception from the GC, since CQL is trying to load all the >>>>>> records to memory ( using select * from *tableName* ). >>>>>> >>>>>> Therefore we had to introduce pagination support by rewriting the >>>>>> migration tool using the Hector Driver. >>>>>> >>>>>> Now the hector based implementation reads in records batch-wise ( the >>>>>> batch size is configurable) and inserts to DAL thus taking out the >>>>>> possibility of running out of memory. >>>>>> >>>>>> Thanks, >>>>>> Sachith >>>>>> -- >>>>>> Sachith Withana >>>>>> Software Engineer; WSO2 Inc.; http://wso2.com >>>>>> E-mail: sachith AT wso2.com >>>>>> M: +94715518127 >>>>>> Linked-In: <http://goog_416592669> >>>>>> https://lk.linkedin.com/in/sachithwithana >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Malith Dhanushka >>>>> Senior Software Engineer - Data Technologies >>>>> *WSO2, Inc. : wso2.com <http://wso2.com/>* >>>>> *Mobile* : +94 716 506 693 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Ramith Jayasinghe >>>> Technical Lead >>>> WSO2 Inc., http://wso2.com >>>> lean.enterprise.middleware >>>> >>>> E: [email protected] >>>> P: +94 777542851 >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Sachith Withana >>> Software Engineer; WSO2 Inc.; http://wso2.com >>> E-mail: sachith AT wso2.com >>> M: +94715518127 >>> Linked-In: <http://goog_416592669> >>> https://lk.linkedin.com/in/sachithwithana >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *Bhathiya Jayasekara* >> *Senior Software Engineer,* >> *WSO2 inc., http://wso2.com <http://wso2.com>* >> >> *Phone: +94715478185 <%2B94715478185>* >> *LinkedIn: http://www.linkedin.com/in/bhathiyaj >> <http://www.linkedin.com/in/bhathiyaj>* >> *Twitter: https://twitter.com/bhathiyax <https://twitter.com/bhathiyax>* >> *Blog: http://movingaheadblog.blogspot.com >> <http://movingaheadblog.blogspot.com/>* >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Ramith Jayasinghe > Technical Lead > WSO2 Inc., http://wso2.com > lean.enterprise.middleware > > E: [email protected] > P: +94 777542851 > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Sachith Withana Software Engineer; WSO2 Inc.; http://wso2.com E-mail: sachith AT wso2.com M: +94715518127 Linked-In: <http://goog_416592669>https://lk.linkedin.com/in/sachithwithana
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
