ListFile performs a listing using Java's File.listFiles(). This will provide a list of all files in the directory. I do not believe this to be related, though. Googling indicates that when this error occurs it is related to the ability to create a native process in order to interact with the file system. I don't think the issue is related to Java heap but rather available RAM on the box. How much RAM is actually available on the box? You mentioned IOPS - are you running in a virtual cloud environment? Using remote storage such as Amazon EBS?
> On May 4, 2016, at 8:56 AM, Joe Witt <joe.w...@gmail.com> wrote: > > Dale, > > Where there is a fetch file there is usually a list file. And while > the symptom of memory issues is showing up in fetch file i am curious > if the issue might actually be caused in ListFile. How many files are > in the directory being listed? > > Mark, > > Are we using a stream friendly API to list files and do we know if > that API on all platforms really doing things in a stream friendly > way? > > THanks > Joe > > On Wed, May 4, 2016 at 7:37 AM, dale.chang13 <dale.chan...@outlook.com> wrote: >> So I still haven't decrypted this problem, and I am assuming that this is an >> IOPS problem instead of a RAM issue. >> >> I have monitored the memory of the nodes in my cluster during the flow, >> before and after the "cannot allocate memory" exception occurs. However, >> there is no memory leak because the memory used by the JVM remains steady >> between 50 and 100 MB used using jconsole. As a note, I have allocated 1 GB >> as a minimum and 4 GB as a maximum for the heap size for each node. >> >> There are also no changes to the number of active threads (35) in jconsole >> while the NiFi gui shows up to 20 active threads. Additionally the number of >> classes loaded and CPU usage remains the same throughout the whole NiFi >> operation. >> >> The only difference I have seen is disk activity on the drive that is >> configured to read to/write from NiFi. >> >> My question is: does it make sense that this is an IO issue, or a RAM/memory >> issue? >> >> >> >> -- >> View this message in context: >> http://apache-nifi-developer-list.39713.n7.nabble.com/FetchFile-Cannot-Allocate-Enough-Memory-tp9720p9901.html >> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
