Hi Andy, I think using Wait processor to hold 5 files for a batch id is reasonable. A situation that may be problematic is that if more than one file for the same filetype are put into S3 mistakenly.
To catch those illegal situations, Wait/Notify and RouteOnAttribute combination may be helpful. I've created a simple flow template to illustrate how those can be used together: https://gist.github.com/ijokarumawak/375915c45071c7cbfddd34d5032c8e90 Hope this helps. Thanks, Koji On Wed, Aug 16, 2017 at 11:01 PM, Andrew Loughran <[email protected]> wrote: > Hey everyone, > > This is my first post. > > I'm building out a pipeline with Nifi, but am stuck on an architectural > decision around some fairly basic design. I think I'm stuck as I'm > operating on the wrong paradigm, but the application receiving my flow is > the limitation in this context. > > I'm using ListS3 to poll for csv files. There need to be 5 different types > of file uploaded with a unique batch identifier for them to be released. > I'm using UpdateAttribute to rip the type and batch from the filename, then > using wait to hold the batch. > > At the moment though, I'm holding until a batch has 5 files, rather than 5 > files with each attribute type matching the expected types. > > Is this the wrong way to be thinking about this problem, or does this sound > like a good use case for Nifi - but using a better combination of > processors. If anyone could give me guidance or point me toward an example > template for batch process I'd be grateful. > > Look forward to helping out in the community where I can. > > Thanks, > > Andy >
