Unfortunately, the library that we use exposes very few options to us. The full listing will be done each time, as we don't really have the ability to filter what was returned. I would recommend going ahead and increasing the heap size and see if that gives you what you need.
Thanks -Mark > On Jun 1, 2016, at 9:28 AM, Joe Witt <[email protected]> wrote: > > Huagen, > > You can possibly avoid it by giving NiFi a large heap size. The > downside is that there is a good library I am aware of which will let > us handle large remote listing as nicely as we'd like. It does tend > to be a 'send small request', 'wait for a large reply', 'then process > large reply' pattern. We used to have this too with ListFile or its > equivalent but then Java exposed a nicer API for us to use. I do not > *think* we have that available at this time with SFTP. > > So yes for now you might be happy with updating the heap size in > conf/boostrap.conf. > > The processor will be smart about not constantly going back for a > listing while it knows it has a lot of work to do but the first > listing can be prohibitively expensive. We definitely should find a > way to sort that out better if my memory on this is correct. > > Mark Payne; Can you confirm or correct? > > Thanks > Joe > > On Wed, Jun 1, 2016 at 9:24 AM, Huagen peng <[email protected]> wrote: >> Hi, >> >> I tried to use the ListSFTP processor on a server with tens of thousands of >> files and the processor tried for a longtime and emit an OutOfMemoryError. >> Can I fix this error by modifying the JVM settings in the >> conf/bootstrap.conf file? >> >> Thanks, >> >> Huagen
