This looks interesting as well. https://stackoverflow.com/questions/52674532/remove-duplicates-in-nifi
On Sat, Aug 15, 2020 at 10:23 AM Josh Friberg-Wyckoff <j...@thefribergs.com> wrote: > In theory I would think you could use the ExecuteStreamCommand to use the > builtin Operating System sort commands to grab unique records. The Windows > Sort command has an undocumented unique option. The sort command on Linux > distros also has a unique option as well. > > On Sat, Aug 15, 2020 at 5:53 AM Robert R. Bruno <rbru...@gmail.com> wrote: > >> I wanted to see if anyone knew is there a clever way to detect duplicate >> records much like you can with entire flow files with DetectDuplicate? I'd >> really rather not have to split my records into individual flow files since >> this flow is such high volume. >> >> Thanks so much in advance. >> >
