Thanks for reply I have a use case where where I am consuming the Events from RabbitMQ using spark streaming.This event has some fields on which I want to query the PostgreSQL and bring the data and then do the join between event data and PostgreSQl data and put the aggregated data into HDFS, so that I run run analytics query over this data using SparkSQL.
my question is PostgreSQL data in production data so i don't want to hit so many times. at any given 1 seconds time I may have 3000 events,that means I need to fire 3000 parallel query to my PostGreSQl and this data keeps on growing, so my database will go down. I can't migrate this PostgreSQL data since lots of system using it,but I can take this data to some NOSQL like base and query the Hbase, but here issue is How can I make sure that Hbase has upto date data? Any anyone suggest me best approach/ method to handle this case? On 27 July 2015 at 23:16, Esteban Gutierrez <[email protected]> wrote: > Hi Jeetendra, > > Unless you are considering to use Phoenix (https://phoenix.apache.org) or > Trafodion (https://wiki.trafodion.org/wiki/index.php/Main_Page) for data > access to HBase there is no single path to migrate a production application > from PostgreSQL to HBase, also you will have to validate all SQL statements > you use or generate in your application since there might be differences > across SQL implementations (the same you would have to do if migration to > another RDBMS). Replicating the data from PostgreSQL to HBase shouldn't be > that hard if your client implements hooks and you could write on both > platforms at the time, if thats not possible you might need to plug into > the PostgreSQL WAL infrastructure to replicate the edits in real time to > HBase. (I think FB or others have done something similar from MySQL to > HBase). > > cheers, > esteban. > > -- > Cloudera, Inc. > > > On Mon, Jul 27, 2015 at 8:57 AM, Jeetendra Gangele <[email protected]> > wrote: > > > Yes. > > > > On 27 July 2015 at 21:02, Ted Yu <[email protected]> wrote: > > > > > By PSQL did you mean PostgreSQL ? > > > > > > Cheers > > > > > > On Mon, Jul 27, 2015 at 12:39 AM, Jeetendra Gangele < > > [email protected]> > > > wrote: > > > > > > > I have a production data in PSQL and i want ti migrant the data to > > Hbase. > > > > Also if there are any changes in my PSQL data , I wanted to update > the > > > > Hbase. > > > > > > > > Since I am migrating from production, i don't wanted to hit too many > > > > request to my server, also this Hbase data should always be sync with > > > PSQL > > > > data. > > > > > > > > Scoop won't work here any other alternatives? > > > > > > > > > >
