Yes Mark, that helps a lot. Thanks. Tom
On Fri, 31 May 2019 at 14:49, Mark Payne <[email protected]> wrote: > Tom, > > You have the Minimum Number of Entries set to 2 and Minimum Group Size set > to 1 MB. That means that > as soon as you have 2 files queued up that total at least 1 MB, it will > create an output FlowFile. That output > will contain as much data as it can, based on what is queued up at that > moment. So if you pull in 2 files, then > a few seconds later pull in 2 more, then a few seconds later 2 more, then > 2 more, you're going to end up with > 8 output FlowFiles, each containing 2 files. > > As I understand it, the idea is to periodically pull in everything in the > directory, merge them together, and then > move on. Later, you'll do another listing, merge those together, and move > on, correct? > > The problem here is that, as you said, you don't know how many files are > in the directory, so MergeContent doesn't > know how many files to wait for, before merging. So one possibility would > be to just set the Minimum Number of Entries > and Minimum Group Size to something much larger. Then set "Max Bin Age" to > say 30 seconds or 60 seconds. That > way, as soon as MergeContent sees a single file, it will wait 30 seconds > or 60 seconds or whatever you have set, and > then merge together all of the files that it has queued up. > > Does that help? > > Thanks > -Mark > > > On May 31, 2019, at 8:27 AM, Tomislav Novosel <[email protected]> > wrote: > > I forgot to mention that I put Correlation Attribute Name as attribute > name which holds directory name from which that 8 files coming from. > And there is not always 8 files, this is just for example. The number of > files changes always as the files come in the folders. > > On Fri, 31 May 2019 at 14:24, Tomislav Novosel <[email protected]> > wrote: > >> Hi all, >> >> I need to create one flowfile from multiple flowfiles which are of files >> in one directory. So if I have e.g. 8 files in directory, I want to merge >> them and the output I want is 1 flowfile for further processing ( I want to >> extract folder name and path of that 8 files). >> >> I tried with MergeContent processor and this is my setup. >> >> <image.png> >> >> But every time i run the flow I get multiple flowfiles in the output >> queue. Sometimes 2, sometimes 3 or even 4. What am I doing wrong? >> Or is there some other way to do this in Nifi? >> >> Thanks in advance, >> BR. >> Tom >> > >
