[
https://issues.apache.org/jira/browse/NIFI-9691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17496393#comment-17496393
]
ASF subversion and git services commented on NIFI-9691:
-------------------------------------------------------
Commit ce0122bd2530e37eb57ace919a42bb2eb3cd5c02 in nifi's branch
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=ce0122b ]
NIFI-9691: Added ForkEnrichment, JoinEnrichment processors
Signed-off-by: Joe Gresock <[email protected]>
This closes #5772.
> Create processors for joining record-oriented data with enrichment data
> -----------------------------------------------------------------------
>
> Key: NIFI-9691
> URL: https://issues.apache.org/jira/browse/NIFI-9691
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Fix For: 1.16.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> A powerful capability of NiFi is its ability to perform enrichment on data as
> it flows through the system. We have processors such as LookupRecord and
> GeoEnrichRecord. However, there are cases where these don't really provide
> the necessary capabilities for performing enrichment.
> A particularly powerful use case is when we have data that we want to enrich
> by calling out to some web service. In this case, we don't want to send our
> payload to the web service. Instead, we want to transform our payload into a
> request that is reasonable to send to a web service. Then, we want to take
> the result from that web service call and use it to enrich our original
> payload. NiFi does not currently offer a convenient mechanism for doing this.
> We should add two additional processors: ForkEnrichment and JoinEnrichment.
> ForkEnrichment would be used create a clone of the incoming FlowFile,
> assigning relevant attributes to the original and the clone and then sending
> each to a different relationship (original and enrichment).
> Data sent to the 'enrichment' connection can then be transformed into
> whatever is necessary to send as a request to the web service. The result
> would then be fed to the JoinEnrichment processor.
> JoinEnrichment should then take input from the "original" connection and the
> "enrichment" connection and join the records together.
> I can foresee three ways to join together the Records:
> * Correlating the records by their index in the FlowFile with a wrapper
> (i.e., there's a "wrapper" element that encapsulates the first record from
> the "original" FlowFile and the first record from the "enrichment" FlowFile.
> * Correlating the records by their index in the FlowFile and insert the
> Enrichment record into the original payload.
> * Use SQL with a JOIN clause to join the records based on some field within
> the data.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
