I searched JIRA for new ControllerService suggestion that works as a MapCacheClient and can be used locally, so I created one, Add LocalMapCacheService https://issues.apache.org/jira/browse/NIFI-3488
On Thu, Feb 16, 2017 at 1:46 PM, Joe Witt <[email protected]> wrote: > definitely Koji's route makes sense to me. If the wait/notify procs > need to leverage a different/more abstract CS that could be workable. > > On Wed, Feb 15, 2017 at 11:43 PM, Peter Wicks (pwicks) > <[email protected]> wrote: >> Thanks Joe, I'll read through it; and give Koji's method a shot too. >> >> -----Original Message----- >> From: Joe Witt [mailto:[email protected]] >> Sent: Wednesday, February 15, 2017 9:31 PM >> To: [email protected] >> Subject: Re: Discuss: Process Group as a Process / "Single Threaded" Process >> Group >> >> Peter >> >> We have a feature proposal on the nifi wiki for process groups as functions. >> I think we could update that to cover your case as well >> https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups >> with the notion of a single input to the 'function/group' at a time. >> I guess it has a little different intent than you're needing though in that >> you're not saying you need it to be referenceable like a function but rather >> act more like a gate. I wonder if the Wait/Notify processors now on master >> would help your case. >> >> The request as asked is a bit outside the spirit of flow based programming >> fundamentals that NiFi is built on so I'm hopeful we can find another path >> that gets you the result you need. >> >> Thanks >> Joe >> >> On Wed, Feb 15, 2017 at 11:18 PM, Peter Wicks (pwicks) <[email protected]> >> wrote: >>> I've been throwing around some ideas for how to migrate processes that >>> would work great in NiFi if you only processed one FlowFile at a time, but >>> if there are multiple Flow Files in the Flow processing through various >>> Processes it would cause issues. >>> >>> One of the concepts I've been playing with is a Process Group that acts >>> more like a 1 threaded processor. One (and only one) FLowFile goes in to >>> the Process Group, 0+ FlowFiles come out (Output port would be optional, >>> kind of like deciding whether or not to auto end a relationship or continue >>> it, but more manual); but no more than 1 FlowFile is ever pulled in at a >>> time. The processor would have an input port, and so long as the Process >>> Group has 1+ Flow Files in it, no new files will flow into the Process >>> Group through the input port. I've been digging around the code, and it >>> doesn't look to difficult to pull off. >>> >>> >>> - New property on ProcessGroup, accompanying UI/validation >>> >>> o Validation would probably need to verify there was 1 Input Port if >>> enabled. >>> >>> - Updated Logic in LocalPort's onTrigger code to check the parent >>> container and see if it has this setting enabled, and if it does how many >>> total FlowFile's are currently in queue. Only transfer in a new FlowFile if >>> 0 flow files in queue. >>> >>> >>> Thoughts? >>> >>> One specific scenario I've come across is a database table that will be >>> truncated at the start of a flow, populated with a FlowFile's contents, >>> updated by a second step to remove duplicates, and then merged into the >>> final table using a third step. Of course if I could use a separate table >>> for each flow file this wouldn't be an issue, but I've run into constraints >>> where I'm limited to a single table. >>>
