I have been trying to run my data flow and I have been running into a problem with being unable to read FetchFiles. I will detail my process below and I would like some confirmation of my suspicions.
First I am ingesting an initial file that is fairly large, which contains the path/filename of a ton of text files within another directory. The goal is to read in the content of that large file, then read in the contents of the thousands of text files, and then store the text file content into Solr. The problem I am having is that the second FetchFile, the one that reads in the smaller text files, frequently reports an error: /FileNotFoundException xxx.txt (Cannot allocate memory); routing to failure/. This FetchFile runs for about 20000 files and then continuously reports the above error for the rest of the files. My suspicion is of two concerns: not enough heap space vs. not enough content_repo/flowfile_repo space. Any ideas or questions? -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/FetchFile-Cannot-Allocate-Enough-Memory-tp9720.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
