definitely Koji's route makes sense to me. If the wait/notify procs need to leverage a different/more abstract CS that could be workable.
On Wed, Feb 15, 2017 at 11:43 PM, Peter Wicks (pwicks) <[email protected]> wrote: > Thanks Joe, I'll read through it; and give Koji's method a shot too. > > -----Original Message----- > From: Joe Witt [mailto:[email protected]] > Sent: Wednesday, February 15, 2017 9:31 PM > To: [email protected] > Subject: Re: Discuss: Process Group as a Process / "Single Threaded" Process > Group > > Peter > > We have a feature proposal on the nifi wiki for process groups as functions. > I think we could update that to cover your case as well > https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups > with the notion of a single input to the 'function/group' at a time. > I guess it has a little different intent than you're needing though in that > you're not saying you need it to be referenceable like a function but rather > act more like a gate. I wonder if the Wait/Notify processors now on master > would help your case. > > The request as asked is a bit outside the spirit of flow based programming > fundamentals that NiFi is built on so I'm hopeful we can find another path > that gets you the result you need. > > Thanks > Joe > > On Wed, Feb 15, 2017 at 11:18 PM, Peter Wicks (pwicks) <[email protected]> > wrote: >> I've been throwing around some ideas for how to migrate processes that would >> work great in NiFi if you only processed one FlowFile at a time, but if >> there are multiple Flow Files in the Flow processing through various >> Processes it would cause issues. >> >> One of the concepts I've been playing with is a Process Group that acts more >> like a 1 threaded processor. One (and only one) FLowFile goes in to the >> Process Group, 0+ FlowFiles come out (Output port would be optional, kind of >> like deciding whether or not to auto end a relationship or continue it, but >> more manual); but no more than 1 FlowFile is ever pulled in at a time. The >> processor would have an input port, and so long as the Process Group has 1+ >> Flow Files in it, no new files will flow into the Process Group through the >> input port. I've been digging around the code, and it doesn't look to >> difficult to pull off. >> >> >> - New property on ProcessGroup, accompanying UI/validation >> >> o Validation would probably need to verify there was 1 Input Port if >> enabled. >> >> - Updated Logic in LocalPort's onTrigger code to check the parent >> container and see if it has this setting enabled, and if it does how many >> total FlowFile's are currently in queue. Only transfer in a new FlowFile if >> 0 flow files in queue. >> >> >> Thoughts? >> >> One specific scenario I've come across is a database table that will be >> truncated at the start of a flow, populated with a FlowFile's contents, >> updated by a second step to remove duplicates, and then merged into the >> final table using a third step. Of course if I could use a separate table >> for each flow file this wouldn't be an issue, but I've run into constraints >> where I'm limited to a single table. >>
