Huagen, You can possibly avoid it by giving NiFi a large heap size. The downside is that there is a good library I am aware of which will let us handle large remote listing as nicely as we'd like. It does tend to be a 'send small request', 'wait for a large reply', 'then process large reply' pattern. We used to have this too with ListFile or its equivalent but then Java exposed a nicer API for us to use. I do not *think* we have that available at this time with SFTP.
So yes for now you might be happy with updating the heap size in conf/boostrap.conf. The processor will be smart about not constantly going back for a listing while it knows it has a lot of work to do but the first listing can be prohibitively expensive. We definitely should find a way to sort that out better if my memory on this is correct. Mark Payne; Can you confirm or correct? Thanks Joe On Wed, Jun 1, 2016 at 9:24 AM, Huagen peng <[email protected]> wrote: > Hi, > > I tried to use the ListSFTP processor on a server with tens of thousands of > files and the processor tried for a longtime and emit an OutOfMemoryError. > Can I fix this error by modifying the JVM settings in the conf/bootstrap.conf > file? > > Thanks, > > Huagen
