Matthew Clarke created NIFI-3423:
------------------------------------
Summary: List based processors don't support source directories
with high file count.
Key: NIFI-3423
URL: https://issues.apache.org/jira/browse/NIFI-3423
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 1.1.1
Reporter: Matthew Clarke
NiFi FlowFile attributes/metadata lives in heap. The List based processors
return a complete listing from the target and then creates a FlowFile for each
File in that returned listing. The FlowFiles being created are not committed to
the list processor's success relationship until all have been created. So you
end up running out of NiFi JVM heap memory before that can happen when the
returned listing is very large.
It would be nice if the list based processors could commit batches (10,000) of
FlowFiles at a time from the returned listing instead of trying to commit them
all at once to help avoid heap exhaustion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
