Benoit, “Microclusters” is not a NiFi feature, it was just me saying you could logically separate flows which do not rely on each other into disparate clusters. So, for example, rather than running 1 12-node cluster, you could run 4 3-node clusters.
Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Dec 5, 2016, at 12:55 AM, De Mezzo Benoit <[email protected]> wrote: > > Thanks! Sorry for the delay! > > What do you mean by creating microclusters? Is this a Nifi functionnality > (ie. hierarchical cluster of clusters) ? Or is it a cluster design that must > be created thoroughly? > > Benoit. > > > > Le 26/11/2016 à 02:13, Andy LoPresto a écrit : >> Benoit, >> >> Every NiFi node can run many disparate flows, and you can separate them into >> distinct process groups for logical divisions. >> >> You can trigger these flows in a few ways. The first, you could set the >> first processor in the flow to be triggered by event receipt rather than >> schedule driven so that it doesn't run until a unit of data enters the flow. >> The other thing you could do is use the REST API to enable/disable >> individual processors/flows and trigger that API invocation via cron or some >> other scheduling system external to NiFi. Finally, you could use an >> ExecuteScript processor to perform more complex custom logic to determine >> which flows should be running at any given time (and then use the REST API >> to enable/disable them). >> >> If you really find that these changes cannot be synchronized across the >> single flow residing on a NiFi cluster, you could segment and subdivide your >> cluster into microclusters where each shares a partial flow (grouped by >> performance considerations or logical concepts). >> >> Andy LoPresto >> [email protected] <mailto:[email protected]> >> [email protected] <mailto:[email protected]> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> On Nov 25, 2016, at 04:13, De Mezzo Benoit <[email protected] >> <mailto:[email protected]>> wrote: >> >>> Hi all, >>> >>> Here is my problem: I need to run many small etl flow (say a thousand) >>> at flow-dedicated times. As theses flows may be added, removed, changed >>> all the time, they can not be easily merged within a big-meta flow to be >>> run in the Nifi cluster. >>> >>> I would like to know if there a way to run multiple flows within a nifi >>> cluster? >>> >>> Or may be there is a way to generate, per flow, a small standalone >>> binary (jar?) to run only once a flow against an embedded Nifi engine >>> (each jar will be managed by a task scheduling cluster) ? >>> >>> Thanks ! >>> >>> Benoit. >>> >
signature.asc
Description: Message signed with OpenPGP using GPGMail
