[
https://issues.apache.org/jira/browse/NIFI-190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254548#comment-14254548
]
Joseph Witt commented on NIFI-190:
----------------------------------
Joseph G:
First off all thanks very much for contributing. Second, thanks for providing
such a responsible PR, patch, etc.. It is a fully implemented processor, with
full in-line and attached documented, with tests, etc. Nice!
So onto the concept of this patch... In the formal description of 'Flow Based
Programming' there is the concept of 'named input ports'. We've not
implemented them in NiFi because the user experience and added complexity has
just never been worth it. Your use case, and processors of the general
merge/correlation scenario do though tend to make it somewhat desirable. Today
we've avoided it by doing exactly what you've done in this processor which is
to pull from the single queue and use an attribute to indicate the logic notion
of a given 'input port'. In your case you distinguish by referring to 'held
items' or 'signal items' as flow file attributes. ...anyway...i said that to
say that we need to kick off a dialogue about this pattern and what we want to
do going forward.
The other thing these use cases bring about is the idea of how the framework
swaps out flow files in the event of large backlogs. For processors like this
one which could want to iterate through large backlogs of objects this behavior
could be problematic. So this seemingy opens up another topic which is should
we have the framework support checking attributes in swapped out flow files in
the event that queue pulls are done by attribute pattern matching.
Anyway - thanks again. Turnaround time on this might not be super fast but
that is not a reflection of your doing the right thing. It is more a
reflection of us needing time overall to consider how best to handle these
sorts of use cases.
Joe W
> HoldFile processor
> ------------------
>
> Key: NIFI-190
> URL: https://issues.apache.org/jira/browse/NIFI-190
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework, Extensions
> Reporter: Joseph Gresock
> Priority: Minor
> Attachments: HoldFile_example.xml
>
>
> Our team has developed a processor for the following use case:
> * Format A needs to be sent to Endpoint A
> * Format B needs to be sent to Endpoint B, but should not proceed until A has
> reached Endpoint A. We most commonly have this restriction when Endpoint B
> requires some output of Endpoint A.
> The proposed HoldFile processor takes 2 types of flow files as input:
> * Files to be held
> * Signal files that can release corresponding held files, based on the value
> of a configurable "release" attribute
> Signal files are distinguished from held files by the presence of the
> "flow.file.release.value" attribute. The processor is configured with a
> "Release Signal Attribute". Held files with this attribute whose value
> matches a received signal value will be released.
> An example:
> HoldFile is configured with Release Signal Attribute = "myId". Its 'Hold'
> relationship routes back onto itself.
> 1. flowFile 1 { myId : "123" } enters HoldFile. It is routed to the 'Hold'
> relationship
> 2. flowFile 2 { flow.file.release.value : "123" } enters HoldFile. flowfile
> 1 is then routed to 'Release', and flow file 2 is removed from the session.
> Signal flow files will also copy their attributes to matching held files,
> unless otherwise indicated. This is what allows the output of Endpoint A to
> pass to Endpoint B, above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
