On Tue, Sep 22, 2015 at 10:45 AM, Sachith Withana <[email protected]> wrote:
> Yes. Hector is deprecated now and DataStax recommends using CQL. > But I went ahead and used Hector for the implementation since it is for a > Cassandra 1.x > and BAM also uses Hector driver in its impementation. > +1 'Hector' would needed since the tool will be required for migration > > On Mon, Sep 21, 2015 at 11:02 AM, Ramith Jayasinghe <[email protected]> > wrote: > >> From what I know, for any new development DataStax recommends using CQL >> over Hector. (unsure if that's relevant here) >> >> On Sun, Sep 20, 2015 at 8:49 AM, Bhathiya Jayasekara <[email protected]> >> wrote: >> >>> Hi Sachith, >>> >>> On Wed, Sep 16, 2015 at 4:13 AM, Sachith Withana <[email protected]> >>> wrote: >>> >>>> @Malith >>>> What do you mean as a generic tool? >>>> >>>> @Ramith >>>> I think that only works with Cassandra 2.0 onwards (If I'm not >>>> mistaken) and BAM uses Cassandra 1.x versions. That was the initial issue. >>>> >>> >>> I don't know about 1.x, but it does support Cassandra 1.2. On the other >>> hand I'm not sure how far Hector supports for Cassandra 2.0. In case you're >>> planning to use this tool with Cassandra 2.0, make sure you thoroughly test >>> it. Unless you have already written this, IMO better to go with Datastax >>> driver. >>> >>> Thanks, >>> Bhathiya >>> >>> >>>> >>>> On Wed, Sep 16, 2015 at 1:18 PM, Ramith Jayasinghe <[email protected]> >>>> wrote: >>>> >>>>> so, >>>>> >>>>> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/ResultSet.html#fetchMoreResults() >>>>> doesn't work for you? >>>>> >>>>> On Wed, Sep 16, 2015 at 12:22 PM, Malith Dhanushka <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Sachith, >>>>>> >>>>>> +1 for using Hector and this will be a handy utility tool for DAS as >>>>>> well. Because anyone who has Cassandra raw data can use this tool to >>>>>> persist them in any file store facilitated by DAL. Given the fact, is it >>>>>> worth the effort implementing this as a generic tool? WDUT? >>>>>> >>>>>> Thanks, >>>>>> Malith >>>>>> >>>>>> On Wed, Sep 16, 2015 at 10:54 AM, Sachith Withana <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> DAS migration tool is used to migrate the data from BAM deployments >>>>>>> to DAS. >>>>>>> >>>>>>> Basically how it works is, it reads the records from Cassandra >>>>>>> column families (of BAM) and inserts them to DAS analytics tables at the >>>>>>> Data Access Layer (DAL) level. >>>>>>> >>>>>>> BAM uses Cassandra 1.x versions and the previous iteration of the >>>>>>> tool was using CQL to get all the records from a given column family and >>>>>>> insert them to DAL. >>>>>>> >>>>>>> But for large amounts of records read from BAM caused CQL to throw >>>>>>> an OutOfMemory exception from the GC, since CQL is trying to load all >>>>>>> the >>>>>>> records to memory ( using select * from *tableName* ). >>>>>>> >>>>>>> Therefore we had to introduce pagination support by rewriting the >>>>>>> migration tool using the Hector Driver. >>>>>>> >>>>>>> Now the hector based implementation reads in records batch-wise ( >>>>>>> the batch size is configurable) and inserts to DAL thus taking out the >>>>>>> possibility of running out of memory. >>>>>>> >>>>>>> Thanks, >>>>>>> Sachith >>>>>>> -- >>>>>>> Sachith Withana >>>>>>> Software Engineer; WSO2 Inc.; http://wso2.com >>>>>>> E-mail: sachith AT wso2.com >>>>>>> M: +94715518127 >>>>>>> Linked-In: <http://goog_416592669> >>>>>>> https://lk.linkedin.com/in/sachithwithana >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Malith Dhanushka >>>>>> Senior Software Engineer - Data Technologies >>>>>> *WSO2, Inc. : wso2.com <http://wso2.com/>* >>>>>> *Mobile* : +94 716 506 693 >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Ramith Jayasinghe >>>>> Technical Lead >>>>> WSO2 Inc., http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>>> E: [email protected] >>>>> P: +94 777542851 >>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Sachith Withana >>>> Software Engineer; WSO2 Inc.; http://wso2.com >>>> E-mail: sachith AT wso2.com >>>> M: +94715518127 >>>> Linked-In: <http://goog_416592669> >>>> https://lk.linkedin.com/in/sachithwithana >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> *Bhathiya Jayasekara* >>> *Senior Software Engineer,* >>> *WSO2 inc., http://wso2.com <http://wso2.com>* >>> >>> *Phone: +94715478185 <%2B94715478185>* >>> *LinkedIn: http://www.linkedin.com/in/bhathiyaj >>> <http://www.linkedin.com/in/bhathiyaj>* >>> *Twitter: https://twitter.com/bhathiyax <https://twitter.com/bhathiyax>* >>> *Blog: http://movingaheadblog.blogspot.com >>> <http://movingaheadblog.blogspot.com/>* >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Ramith Jayasinghe >> Technical Lead >> WSO2 Inc., http://wso2.com >> lean.enterprise.middleware >> >> E: [email protected] >> P: +94 777542851 >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Sachith Withana > Software Engineer; WSO2 Inc.; http://wso2.com > E-mail: sachith AT wso2.com > M: +94715518127 > Linked-In: <http://goog_416592669> > https://lk.linkedin.com/in/sachithwithana > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Thanks & Regards Iranga Muthuthanthri (M) -0777-255773 Team Product Management
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
