Hi Peter, Joe Recently, I've worked on NIFI-3452 "Add Wait processor Wait Mode property". Using Wait/Notify combined with Back-pressure FlowFile count 1, it's possible to limit only one FlowFile to be processed in a part of flow. It's already merged into the master. Example is available here: https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67 https://issues.apache.org/jira/browse/NIFI-3452
It may not be an ideal solution, but it will work for your use case. Thanks, Koji On Thu, Feb 16, 2017 at 1:31 PM, Joe Witt <[email protected]> wrote: > Peter > > We have a feature proposal on the nifi wiki for process groups as > functions. I think we could update that to cover your case as well > https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups > with the notion of a single input to the 'function/group' at a time. > I guess it has a little different intent than you're needing though in > that you're not saying you need it to be referenceable like a function > but rather act more like a gate. I wonder if the Wait/Notify > processors now on master would help your case. > > The request as asked is a bit outside the spirit of flow based > programming fundamentals that NiFi is built on so I'm hopeful we can > find another path that gets you the result you need. > > Thanks > Joe > > On Wed, Feb 15, 2017 at 11:18 PM, Peter Wicks (pwicks) > <[email protected]> wrote: >> I've been throwing around some ideas for how to migrate processes that would >> work great in NiFi if you only processed one FlowFile at a time, but if >> there are multiple Flow Files in the Flow processing through various >> Processes it would cause issues. >> >> One of the concepts I've been playing with is a Process Group that acts more >> like a 1 threaded processor. One (and only one) FLowFile goes in to the >> Process Group, 0+ FlowFiles come out (Output port would be optional, kind of >> like deciding whether or not to auto end a relationship or continue it, but >> more manual); but no more than 1 FlowFile is ever pulled in at a time. The >> processor would have an input port, and so long as the Process Group has 1+ >> Flow Files in it, no new files will flow into the Process Group through the >> input port. I've been digging around the code, and it doesn't look to >> difficult to pull off. >> >> >> - New property on ProcessGroup, accompanying UI/validation >> >> o Validation would probably need to verify there was 1 Input Port if >> enabled. >> >> - Updated Logic in LocalPort's onTrigger code to check the parent >> container and see if it has this setting enabled, and if it does how many >> total FlowFile's are currently in queue. Only transfer in a new FlowFile if >> 0 flow files in queue. >> >> >> Thoughts? >> >> One specific scenario I've come across is a database table that will be >> truncated at the start of a flow, populated with a FlowFile's contents, >> updated by a second step to remove duplicates, and then merged into the >> final table using a third step. Of course if I could use a separate table >> for each flow file this wouldn't be an issue, but I've run into constraints >> where I'm limited to a single table. >>
