NIFI-6047 [1] is possibly what you're after, but that won't help you just now because it appears to remain unfinished.
[1] https://issues.apache.org/jira/browse/NIFI-6047 Cheers, Chris Sampson On Mon, 15 Feb 2021, 06:27 Jorge Machado, <[email protected]> wrote: > Hey Jeremy, > > Something linke this > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.DetectDuplicate/index.html > ? > > > On 15. Feb 2021, at 04:45, Jeremy Pemberton-Pigott <[email protected]> > wrote: > > Hi everyone, I'm wondering if there is a Detect Duplicate processor that > can read records from a flow file and as output gives just the > non-duplicates (can be single records or a group of non-duplicates would be > better). I want to use a record reader to avoid splitting the json content > into 10000s of flow files to detect the duplicates. Immediately after this > flow is a record reader/writer going to HBase. > > Jeremy > > >
