Joe, The JIRA was NIFI-1008 [1]. I looked into this a couple of times but ended up going down very deep rabbit holes and not getting the desired impact. I think we could probably use some refactoring of StandardProcessSession and FileSystemSwapManager to clean up some of the existing code in such a way that this would become easier to tackle.
Thanks -Mark [1] https://issues.apache.org/jira/browse/NIFI-1008 <https://issues.apache.org/jira/browse/NIFI-1008> > On Jun 7, 2016, at 8:37 AM, Joe Witt <[email protected]> wrote: > > Leslie > > You could update the flowfile queue swap threshold higher in such a case. > That said at that size of a heap you may see undesirable GC behavior. > > Simply put what you saw when making tons of tiny flowfiles after a split and > it causing slowness is a bug and should be fine on far smaller heaps. We can > and should push those things out of the heap for large sessions like that. > You should be able to use it like you did and see no perf implications. That > we must fix. > > Mark Payne...do you recall the jira number to tackle this? > > Thanks > Joe > > On Jun 7, 2016 6:23 AM, "Leslie Hartman" <[email protected] > <mailto:[email protected]>> wrote: > Hello: > > If you have a server with say 64 Gigs of memory and wanted to give > 32 Gigs of memory to nifi, what options would you change? > > When I was splitting a large file up into a lot of smaller pieces it ran > out of memory. > > Thank You. > > Leslie Hartman
