Sorry for the long email, I couldn't figure out a shorter way to explain the 
context.

I have several processor groups in our production system. While they all differ 
as to where they get the data, towards the end of each processor group they 
look very similar; that is, they all store the data they've retrieved in a 
similar manner. However, at the very end of each processor group I need to do 
some cleanup, which differs by processor group.

In order to "DRY" up my processor groups, I created an external, "Store File" 
processor group. I modified the existing processor group, "A", to use it. When 
"Store File" finishes, it routes to an Output Port called "Storage Done". Back 
in "A" I have an input port that grabs that and does the cleanup.  Using "A" 
hooked into "Storage File" and then back to "A" for cleanup worked great.

Then I tried to hook in processor group "B" to "Store File".  It also worked, 
until "Store File" sends the data to the "Storage Done" output port.  Because 
this output port for "Store File" is now hooked to the cleanup input ports in 
"A" and "B", flow files are sent to both A & B. This causes errors because the 
cleanup is different for both groups.

I guess I ran into this problem because, in my mental model, I was thinking of 
Input/Output ports like calling a function - when the function was done, it 
would return to the calling function (the calling processor group). Obviously, 
I was wrong about that.

My current planned solution is to add an attribute to both "A" and "B" (and C, 
D, etc) that holds the Processor Group name, and then inside of "Store File" 
use the RouteOnAttribute to route to the "correct" output port.  However, this 
isn't ideal - "Store File" is now somewhat "aware" of the other processor 
groups and thus more strongly coupled than I'd like.

Is there a better way to DRY up code in NiFi than what I'm doing?

John


Reply via email to