Hi All, I've got a case where I need to join two or more separate flow files, based on one or more 's ID within the flow (Multi Record flow file, each record with a unique ID). The content of the second flow file could also contain one or more records each again with a matching ID to the records in the first flow file. The second flow file comes from a different data source to the first and could be made to one record per flow file with the ID as an attribute if this makes it simpler.
It can take some time for there to be a match, so it will need to be able to queue and replay for a configurable period. If after a set period, there is no match release it downstream. When a match is found, the matching record in the secondary flow file needs to be injected into the content of the first flow file record. I see there is the Lookup Record Processor, but there isn't a lookup service to query flow files in a queue or provenance? I want to avoid writing the secondary flow file to a database, or other storage option and would like to do the merge in flight through NiFi. Below is a rough sketch of what I am trying to do. Any suggestions would be much appreciated. Kind Regards, Nathan [cid:[email protected]]
