This is good stuff Koji, there is so much new stuff going into NiFi it's hard to keep up.
I'm a little disappointed to see that you have to use a Distributed Map Cache. This makes sense for many scenarios, but I wonder what other options there are (of course with extra coding), specifically for non-clustered environments. Any ideas? -----Original Message----- From: Koji Kawamura [mailto:[email protected]] Sent: Wednesday, February 15, 2017 9:35 PM To: [email protected] Subject: Re: Discuss: Process Group as a Process / "Single Threaded" Process Group Hi Peter, Joe Recently, I've worked on NIFI-3452 "Add Wait processor Wait Mode property". Using Wait/Notify combined with Back-pressure FlowFile count 1, it's possible to limit only one FlowFile to be processed in a part of flow. It's already merged into the master. Example is available here: https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67 https://issues.apache.org/jira/browse/NIFI-3452 It may not be an ideal solution, but it will work for your use case. Thanks, Koji On Thu, Feb 16, 2017 at 1:31 PM, Joe Witt <[email protected]> wrote: > Peter > > We have a feature proposal on the nifi wiki for process groups as > functions. I think we could update that to cover your case as well > https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Proces > s+Groups with the notion of a single input to the 'function/group' at > a time. > I guess it has a little different intent than you're needing though in > that you're not saying you need it to be referenceable like a function > but rather act more like a gate. I wonder if the Wait/Notify > processors now on master would help your case. > > The request as asked is a bit outside the spirit of flow based > programming fundamentals that NiFi is built on so I'm hopeful we can > find another path that gets you the result you need. > > Thanks > Joe > > On Wed, Feb 15, 2017 at 11:18 PM, Peter Wicks (pwicks) > <[email protected]> wrote: >> I've been throwing around some ideas for how to migrate processes that would >> work great in NiFi if you only processed one FlowFile at a time, but if >> there are multiple Flow Files in the Flow processing through various >> Processes it would cause issues. >> >> One of the concepts I've been playing with is a Process Group that acts more >> like a 1 threaded processor. One (and only one) FLowFile goes in to the >> Process Group, 0+ FlowFiles come out (Output port would be optional, kind of >> like deciding whether or not to auto end a relationship or continue it, but >> more manual); but no more than 1 FlowFile is ever pulled in at a time. The >> processor would have an input port, and so long as the Process Group has 1+ >> Flow Files in it, no new files will flow into the Process Group through the >> input port. I've been digging around the code, and it doesn't look to >> difficult to pull off. >> >> >> - New property on ProcessGroup, accompanying UI/validation >> >> o Validation would probably need to verify there was 1 Input Port if >> enabled. >> >> - Updated Logic in LocalPort's onTrigger code to check the parent >> container and see if it has this setting enabled, and if it does how many >> total FlowFile's are currently in queue. Only transfer in a new FlowFile if >> 0 flow files in queue. >> >> >> Thoughts? >> >> One specific scenario I've come across is a database table that will be >> truncated at the start of a flow, populated with a FlowFile's contents, >> updated by a second step to remove duplicates, and then merged into the >> final table using a third step. Of course if I could use a separate table >> for each flow file this wouldn't be an issue, but I've run into constraints >> where I'm limited to a single table. >>
