The trigger should probably be a new Controller Service, because that makes more logical sense, but it’s not available today.
Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Nov 30, 2016, at 5:19 PM, Andy LoPresto <[email protected]> wrote: > > Hi Russ, > > Could you use the cluster state manager [1] to hold this boolean trigger > value which each instance of your customer processor checks before execution, > and then use a simple ExecuteScript processor which simply toggles/explicitly > writes that value? In this way, the ExecuteScript processor is like the light > switch. You can manually start/stop that processor to trigger or stop all the > others, or use the REST API to do the same, or even make the ExecuteScript > processor read from a system/environment variable or the > absence/presence/value of a file on disk to get the desired state value. > > The ExecuteScript processor might have to abuse the StandardStateManager by > first enumerating all instances of the desired “controllable” components > (this could be achieved by dynamically querying a containing process group > for processors by type or manually populating a list of component IDs in a > static list, which the ExecuteScript processor could then store in its own > StateManager) and then manually instantiating a StateManager containing the > local/cluster StateProvider [2] for each component ID and setting the state. > > Not sure if I explained that well, but Mark Payne would be your guy for a > better explanation and possibly a cleaner solution. > > [1] > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#state_management > > <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#state_management> > [2] > https://github.com/apache/nifi/blob/master/nifi-framework-api/src/main/java/org/apache/nifi/components/state/StateProvider.java#L66 > > <https://github.com/apache/nifi/blob/master/nifi-framework-api/src/main/java/org/apache/nifi/components/state/StateProvider.java#L66> > > > Andy LoPresto > [email protected] <mailto:[email protected]> > [email protected] <mailto:[email protected]> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Nov 30, 2016, at 2:00 PM, Russell Bateman <[email protected] >> <mailto:[email protected]>> wrote: >> >> Our usage, for ETL, is controlled very up-close and personal right now. Our >> ETL of medical documents is pretty involved, changes radically from customer >> to customer, and must be baby-sat closely. Anything we're able to do for our >> implementation folk to enable them to pin down waste of resources including >> and especially time to ingest horrendous quantities of information is going >> to serve us for a long time to come. User access to an easy processor like >> the one I've written, though what it does is pretty trivial, makes their >> life so much easier and they can talk back to us about where time (in >> particular) is being spent, in which processor (we have lots of custom >> processors that do very out-of-the-ordinary things), across which subflow, >> etc. >> >> Except that we anticipate moving to a clustered implementation soon, I >> thought about merely looking for a system environment variable or even the >> presence of a file, then setting static state inside the processor to halt >> doing anything. Conversely, a change to that state might start the processor >> back up again (time-stamping, histogramming, etc.). I think this naïve >> control strategy falls apart as soon we go to a cluster. >> >> It's taking me a while to get into the NiFi culture, I think. However, I >> also think that NiFi folk use NiFi in wildly different ways so maybe how I'm >> looking to do something isn't always so un-NiFi, but that others just >> haven't tackled it yet. >> >> Yeah, if NiFi gave us some kind of modifiable, global state, especially if >> less static than /conf/nifi.properties/, but even if requiring a bounce to >> engage it (so, /conf/flow.properties/ or /conf/flow.conf/), that would solve >> our problem pretty elegantly. However, I haven't thought about what problems >> it also creates for you or others. >> >> Russ >> >> >> On 11/30/2016 02:42 PM, Joe Witt wrote: >>> Russ >>> >>> I don't think we provide anything particularly helpful here to do this >>> conveniently. You could of course script this external to NiFi to >>> make HTTP calls to shut off such items. Spitballing ideas here but >>> what about giving you the ability to tag components with some label >>> and then be able to do global execution of some task >>> (stop/start/disable/delete/etc..) against components that you're >>> authorized to and which have those labels. >>> >>> Do you think this would be a typical use case or do you feel this is >>> useful because you're testing right now? Does the above idea make >>> sense or do you have other suggestions? >>> >>> Thanks >>> Joe >>> >>> On Wed, Nov 30, 2016 at 3:14 PM, Russell Bateman <[email protected] >>> <mailto:[email protected]>> wrote: >>>> I've written a custom processor for some trivial profiling, time-stamping, >>>> time-since, histogram-generating, etc., but would like the ability to turn >>>> all instances completely off without having to visit each instance in the >>>> UI. If it works out, I might consider even leaving some instances in >>>> production- or at least staging-environment flows. >>>> >>>> 1. I know that the NiFi Expression Language has access to various system- >>>> or >>>> NiFi properties or settings, but what would someone suggest as best >>>> practice >>>> for this? (Don't invade /conf/nifi.properties/, etc.) >>>> >>>> 2. I guess I'd add a property to configure in my processor and check >>>> whether >>>> it evaluates true/false/etc. based on the source data (whatever that will >>>> be--see previous paragraph)? >>>> >>>> 3. Last, if this processor is thereby reduced merely to >>>> >>>> session.transfer( flowfile, SUCCESS ); >>>> >>>> there isn't any handling even more minimal or faster than that in the sense >>>> of turning a processor off, right? >>>> >>>> Thanks for any suggestions, >>>> >>>> Russ >> >
signature.asc
Description: Message signed with OpenPGP using GPGMail
