Re: Data Load

Anil, When you say huge volumes, do you mean a large number of tables, or large tables, or both? For a large number of tables, you will likely want to upgrade to the upcoming NiFi release so you can use ListDatabaseTables -> GenerateTableFetch -> ExecuteSQL for the source part, although in

Re: Data Load

Hi Matt, I quickly developed this and this is how i could do this DataLake<-ExecuteSQL->ConvertAveroToJson->SplitJson->EvaluateJsonPath->ReplaceText->PutSQL->Postgres(onCloud) The problem is, this will not scale for huge volumes. Any thoughts? Regards Anil On Tue, May 2, 2017 at 12:07 PM,

Re: Data Load

Yes that sounds like your best bet, assuming you have the "Maximum Value Column" present in the table you want to migrate. Then a flow might look like: QueryDatabaseTable -> ConvertAvroToJSON -> ConvertJSONToSQL -> PutSQL In this flow the target tables would need to be created beforehand. You

Re: Data Load

Thanks Matt for the quick reply. We are using nifi 1.0 release as of now. It's a postgres DB on both sides (on prem and on cloud) and yes incremental load is what i am looking for. so with that, you recommend # 2 option? On Tue, May 2, 2017 at 11:00 AM, Matt Burgess

Data Load

I have a simple use case. DB (On Premise) and DB (On Cloud). I want to use nifi to extract data from on prem DB (huge volumes) and insert into the same table structure that is hosted on cloud. I could use ExecuteSQL on both sides of the fence (to extract from on prem and insert onto cloud).