Thanks Joe, I'll read through it; and give Koji's method a shot too. -----Original Message----- From: Joe Witt [mailto:[email protected]] Sent: Wednesday, February 15, 2017 9:31 PM To: [email protected] Subject: Re: Discuss: Process Group as a Process / "Single Threaded" Process Group
Peter We have a feature proposal on the nifi wiki for process groups as functions. I think we could update that to cover your case as well https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups with the notion of a single input to the 'function/group' at a time. I guess it has a little different intent than you're needing though in that you're not saying you need it to be referenceable like a function but rather act more like a gate. I wonder if the Wait/Notify processors now on master would help your case. The request as asked is a bit outside the spirit of flow based programming fundamentals that NiFi is built on so I'm hopeful we can find another path that gets you the result you need. Thanks Joe On Wed, Feb 15, 2017 at 11:18 PM, Peter Wicks (pwicks) <[email protected]> wrote: > I've been throwing around some ideas for how to migrate processes that would > work great in NiFi if you only processed one FlowFile at a time, but if there > are multiple Flow Files in the Flow processing through various Processes it > would cause issues. > > One of the concepts I've been playing with is a Process Group that acts more > like a 1 threaded processor. One (and only one) FLowFile goes in to the > Process Group, 0+ FlowFiles come out (Output port would be optional, kind of > like deciding whether or not to auto end a relationship or continue it, but > more manual); but no more than 1 FlowFile is ever pulled in at a time. The > processor would have an input port, and so long as the Process Group has 1+ > Flow Files in it, no new files will flow into the Process Group through the > input port. I've been digging around the code, and it doesn't look to > difficult to pull off. > > > - New property on ProcessGroup, accompanying UI/validation > > o Validation would probably need to verify there was 1 Input Port if > enabled. > > - Updated Logic in LocalPort's onTrigger code to check the parent > container and see if it has this setting enabled, and if it does how many > total FlowFile's are currently in queue. Only transfer in a new FlowFile if 0 > flow files in queue. > > > Thoughts? > > One specific scenario I've come across is a database table that will be > truncated at the start of a flow, populated with a FlowFile's contents, > updated by a second step to remove duplicates, and then merged into the final > table using a third step. Of course if I could use a separate table for each > flow file this wouldn't be an issue, but I've run into constraints where I'm > limited to a single table. >
