Benoit, Every NiFi node can run many disparate flows, and you can separate them into distinct process groups for logical divisions.
You can trigger these flows in a few ways. The first, you could set the first processor in the flow to be triggered by event receipt rather than schedule driven so that it doesn't run until a unit of data enters the flow. The other thing you could do is use the REST API to enable/disable individual processors/flows and trigger that API invocation via cron or some other scheduling system external to NiFi. Finally, you could use an ExecuteScript processor to perform more complex custom logic to determine which flows should be running at any given time (and then use the REST API to enable/disable them). If you really find that these changes cannot be synchronized across the single flow residing on a NiFi cluster, you could segment and subdivide your cluster into microclusters where each shares a partial flow (grouped by performance considerations or logical concepts). Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Nov 25, 2016, at 04:13, De Mezzo Benoit <[email protected]> wrote: > > Hi all, > > Here is my problem: I need to run many small etl flow (say a thousand) > at flow-dedicated times. As theses flows may be added, removed, changed > all the time, they can not be easily merged within a big-meta flow to be > run in the Nifi cluster. > > I would like to know if there a way to run multiple flows within a nifi > cluster? > > Or may be there is a way to generate, per flow, a small standalone > binary (jar?) to run only once a flow against an embedded Nifi engine > (each jar will be managed by a task scheduling cluster) ? > > Thanks ! > > Benoit. >
