Thanks Peter. The generate flow file option was a good fit for our case. We are also able to trigger it using a shell script by using curl.
On Tue, Aug 27, 2019, 1:45 AM Peter Turcsanyi <[email protected]> wrote: > Hi Bimal, > > With Variable Registry, you can implement it in the following way: > Put your flow into a Process Group. Use variable references in your > processors (eg. ${db.table}) and define the variables at the process group > level. Then copy the process group (by simply copying it or creating a > template from it first) and set the variables to the proper values in each > process group. You can also configure separate scheduling in each process > group. > The drawback that you need to multiply your flow. > > Another approach: > Defined your flow only once and use FlowFile attributes instead of > variables in variable registry. > Use GenerateFlowFile and add the FlowFile attributes via the dynamic > properties of this processor. Configure a separate GenerateFlowFile for > each of your source tables and connect them to the same "SQL" processor > (which was the entry point earlier). Configure the scheduling on these > GenerateFlowFile-s. > The problem is that not all "SQL" processors support flowfile input. You > can use ExecuteSQL(Record) or GenerateTableFetch in this way, but not > QueryDatabaseTable. > > Regards, > Peter > > On Mon, Aug 26, 2019 at 6:30 PM Bimal Mehta <[email protected]> wrote: > >> Hi, >> >> We have a data flow which extracts data from source database table and >> loads into target hive table. This flow needs to run several times in a >> day to get delta records from source table and also for multiple tables . >> Now we need to replicate this same process for all the different source >> tables. So rather than creating multiple data flows for each separate >> table, can I use the existing flow and pass parameters like source table >> name to that flow and the flow starts. Basically looking for an interface >> where the user can pass the table names that we want to load at a given >> point in time and the flow is triggered for that table. Variable Registry >> comes to mind, but I am not sure how to make it work for this use case. We >> are using NiFi 1.9.0 as part of CDF bundle. >> >> Thanks >> Bimal Mehta >> >
