I initially did a test migration using DataStax, as discussed in the post [1], for my personal testing purposes. However, as suggested by Deep and Tharindu, I started using Sqoop instead. It is pretty simple to migrate such, with Sqoop/Hadoop. I had a few issues which I verified from #sqoop irc. Will share the steps with the list, as we will go ahead with sqoop for the migration.
On a related note, Sanjeewa also reminded that for the effective use of cassandra, we might need to change the data representation and normalizations, than a mere migration from the current database from mysql as it is. We will investigate on that too. [1] http://kkpradeeban.blogspot.com/2012/06/moving-data-from-mysql-to-cassandra.html Thank you. Regards, Pradeeban. On Fri, Jun 15, 2012 at 12:45 PM, Kathiravelu Pradeeban <[email protected]>wrote: > Thanks Sanjeewa and AmilaM. > > Regards, > Pradeeban. > > > On Fri, Jun 15, 2012 at 12:38 PM, Sanjeewa Malalgoda <[email protected]>wrote: > >> +1 for doing this. >> So you have to migrate all bandwidth stat tables. >> BAM_BANDWIDTH_STAT_HOUR_FACT >> BAM_BANDWIDTH_STAT_DAY_FACT >> BAM_BANDWIDTH_STAT_MONTH_FACT >> BAM_BANDWIDTH_STAT_QTR_FACT >> BAM_BANDWIDTH_STAT_YEAR_FACT >> >> Other than that you have to migrate all DIM tables which are used to >> store hour,day,month.qtr and year ID's. >> Also >> BAM_SERVER >> BAM_SERVER_CATEGORY >> BAM_SERVER_USER_DATA >> BAM_SERVICE >> BAM_SERVICE_DATA >> >> All the BAM_SERVICE_STAT tables (hour,day,month,qtr,year) >> All the BAM_ENDPOINT_STAT tables (hour,day,month,qtr,year) >> All the BAM_OPERATION_STAT tables (hour,day,month,qtr,year) >> All the BAM_SERVER_STAT tables (hour,day,month,qtr,year) >> >> for the moment try above tables i will investigate more and give you a >> complete answer. >> >> Thanks, >> >> >> On Fri, Jun 15, 2012 at 12:20 PM, Kathiravelu Pradeeban < >> [email protected]> wrote: >> >>> Hi Sanjeewa, >>> I have been testing the data migration from relational databases (mysql) >>> to cassandra [1]. Now for our data migration for usage/billing, can you >>> list which are the tables to be ported? As of a recent offline chat, we >>> noticed we don't have to port everything, since we are moving to BAM2. We >>> can discuss our findings here for the record. >>> >>> [1] >>> http://kkpradeeban.blogspot.com/2012/06/moving-data-from-mysql-to-cassandra.html >>> >>> Thank you. >>> Regards, >>> Pradeeban. >>> >>> >>> On Tue, Jun 5, 2012 at 3:40 PM, Sanjeewa Malalgoda <[email protected]>wrote: >>> >>>> What we have done is use extended AbstractSummaryGenerator and >>>> implement its >>>> methods. Inside them we have used data services to read and write data >>>> from/to database. >>>> >>>> Thanks. >>>> >>>> >>>> On Tue, Jun 5, 2012 at 3:30 PM, Kasun Weranga <[email protected]> wrote: >>>> >>>>> >>>>> >>>>> On Tue, Jun 5, 2012 at 2:06 PM, Sanjeewa Malalgoda >>>>> <[email protected]>wrote: >>>>> >>>>>> >>>>>> >>>>>> On Tue, Jun 5, 2012 at 1:54 PM, Kasun Weranga <[email protected]>wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Jun 5, 2012 at 1:06 PM, Sanjeewa Malalgoda < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> +1 for doing this change. Actually we have to change few places. >>>>>>>> >>>>>>>> 01. Change *Usage agent's* publisherUtils publish method with new >>>>>>>> publish method pointing to bam2. >>>>>>>> 02. Modify *Summery generation* code for summarize hourly, daily, >>>>>>>> monthly >>>>>>>> >>>>>>> >>>>>>> Are you going to modify the existing summary generation code? I >>>>>>> think better way is to use analyzer framework provided by BAM2 for doing >>>>>>> the summarization. >>>>>>> >>>>>> +for use analyzer frame work. In earlier case also we used extended >>>>>> bam core summery generator code. >>>>>> >>>>>>> 03. Change* Usage service* data retrieving code to get usage data >>>>>>>> from cassandra. >>>>>>>> >>>>>>> >>>>>>> since the earlier implantation read data from RDBMS. You might use >>>>>>> the same implementation(with minimal change) If we use hive queries to >>>>>>> read >>>>>>> data from cassandra, then do the summarization and put the summarized >>>>>>> data >>>>>>> into RDBMS as earlier. >>>>>>> >>>>>>> We access them by calling Data service (meteringquery.dbs). So is it >>>>>> possible to use data services with BAM2. >>>>>> I guess we cant. In that case we might have to write usage service >>>>>> code. >>>>>> Also please note that except initial publishing process, we use data >>>>>> services for almost all the data base operations in usage. >>>>>> >>>>> >>>>> Have you written your own implementation to summarize the usage data ? >>>>> If so changing code to retrieve data from Cassandra will be enough. But it >>>>> doesn't scale. >>>>> >>>>> Thanks, >>>>> KasunW >>>>> >>>>> Thanks, >>>>>>> KasunW. >>>>>>> >>>>>>> (Usage ui and throttling manager will use this service) >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> On Tue, Jun 5, 2012 at 12:09 PM, Kathiravelu Pradeeban < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> Currently Stratos usage and summarizer components are using BAM >>>>>>>>> components. This is to be ported to use the new BAM2 in trunk. >>>>>>>>> We have started working on this, with PublisherUtils of the usage >>>>>>>>> bundle, to begin with publishing. >>>>>>>>> >>>>>>>>> Thank you. >>>>>>>>> Regards, >>>>>>>>> Pradeeban. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Kathiravelu Pradeeban. >>>>>>>>> Cloud Technologies Team. >>>>>>>>> WSO2 Inc. >>>>>>>>> >>>>>>>>> Blog: [Llovizna] http://kkpradeeban.blogspot.com/ >>>>>>>>> M: +94 776 477 976 >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Dev mailing list >>>>>>>>> [email protected] >>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Sanjeewa Malalgoda* >>>>>>>> mobile : +94 713068779 >>>>>>>> <http://sanjeewamalalgoda.blogspot.com/>blog >>>>>>>> :http://sanjeewamalalgoda.blogspot.com/<http://sanjeewamalalgoda.blogspot.com/> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Sanjeewa Malalgoda* >>>>>> mobile : +94 713068779 >>>>>> <http://sanjeewamalalgoda.blogspot.com/>blog >>>>>> :http://sanjeewamalalgoda.blogspot.com/<http://sanjeewamalalgoda.blogspot.com/> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Sanjeewa Malalgoda* >>>> mobile : +94 713068779 >>>> <http://sanjeewamalalgoda.blogspot.com/>blog >>>> :http://sanjeewamalalgoda.blogspot.com/<http://sanjeewamalalgoda.blogspot.com/> >>>> >>> >>> >>> >>> -- >>> Kathiravelu Pradeeban. >>> Cloud Technologies Team. >>> WSO2 Inc. >>> >>> Blog: [Llovizna] http://kkpradeeban.blogspot.com/ >>> M: +94 776 477 976 >>> >>> >> >> >> -- >> *Sanjeewa Malalgoda* >> mobile : +94 713068779 >> <http://sanjeewamalalgoda.blogspot.com/>blog >> :http://sanjeewamalalgoda.blogspot.com/<http://sanjeewamalalgoda.blogspot.com/> >> > > > > -- > Kathiravelu Pradeeban. > Cloud Technologies Team. > WSO2 Inc. > > Blog: [Llovizna] http://kkpradeeban.blogspot.com/ > M: +94 776 477 976 > > -- Kathiravelu Pradeeban. Cloud Technologies Team. WSO2 Inc. Blog: [Llovizna] http://kkpradeeban.blogspot.com/ M: +94 776 477 976
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
