Thanks for the replies guys. Yes, NIFI-6047 is pretty much exactly what I'm looking for.
Jeremy On Mon, Feb 15, 2021 at 2:37 PM Chris Sampson <[email protected]> wrote: > NIFI-6047 [1] is possibly what you're after, but that won't help you just > now because it appears to remain unfinished. > > > [1] https://issues.apache.org/jira/browse/NIFI-6047 > > Cheers, > > Chris Sampson > > On Mon, 15 Feb 2021, 06:27 Jorge Machado, <[email protected]> wrote: > >> Hey Jeremy, >> >> Something linke this >> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.DetectDuplicate/index.html >> ? >> >> >> On 15. Feb 2021, at 04:45, Jeremy Pemberton-Pigott <[email protected]> >> wrote: >> >> Hi everyone, I'm wondering if there is a Detect Duplicate processor that >> can read records from a flow file and as output gives just the >> non-duplicates (can be single records or a group of non-duplicates would be >> better). I want to use a record reader to avoid splitting the json content >> into 10000s of flow files to detect the duplicates. Immediately after this >> flow is a record reader/writer going to HBase. >> >> Jeremy >> >> >>
