I've been throwing around some ideas for how to migrate processes that would work great in NiFi if you only processed one FlowFile at a time, but if there are multiple Flow Files in the Flow processing through various Processes it would cause issues.
One of the concepts I've been playing with is a Process Group that acts more like a 1 threaded processor. One (and only one) FLowFile goes in to the Process Group, 0+ FlowFiles come out (Output port would be optional, kind of like deciding whether or not to auto end a relationship or continue it, but more manual); but no more than 1 FlowFile is ever pulled in at a time. The processor would have an input port, and so long as the Process Group has 1+ Flow Files in it, no new files will flow into the Process Group through the input port. I've been digging around the code, and it doesn't look to difficult to pull off. - New property on ProcessGroup, accompanying UI/validation o Validation would probably need to verify there was 1 Input Port if enabled. - Updated Logic in LocalPort's onTrigger code to check the parent container and see if it has this setting enabled, and if it does how many total FlowFile's are currently in queue. Only transfer in a new FlowFile if 0 flow files in queue. Thoughts? One specific scenario I've come across is a database table that will be truncated at the start of a flow, populated with a FlowFile's contents, updated by a second step to remove duplicates, and then merged into the final table using a third step. Of course if I could use a separate table for each flow file this wouldn't be an issue, but I've run into constraints where I'm limited to a single table.
