Hi
I've been using NIFI for a time now, but only for simple flows. But now I
got a new use case which is working outside nifi as a cron job, but I would
like the flow to be inside nifi.
I have to split data files into separate files based on zipcodes, or just
zipcodes of interest
I'm using a command line tool which can filter out data based on input
value, or just split all data out in separate files bast on selected field.
I have a csv file with zipcodes of interest and a script running in a
cronjob doing something like this:
read csv records into an array
for file in datafolder:
for rcd in records:
splitApp -in file -filterby rcd -out outfolder/rcd/file
Today NIFI is getting datafile and save them into datafolder. cronjob is
running each 5 minutes and nifi is looking into outfolder to pickup files.
But would it be possible to have NIFI do it all?
How do I lookup records?
Based on the amount of records, make multiple copies of the datafile
On each copy run a ExecuteProcess with record[copy.index] as input value
or would another approach be just to run the splitApp -splitby zipcodes and
load all files into nifi and later on route files based on if the zipcodes
is in records?
kind regards
Jens
