[
https://issues.apache.org/jira/browse/NIFI-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16834152#comment-16834152
]
Joseph Witt commented on NIFI-6202:
-----------------------------------
This is almost always a problem of custom processor code not written take
advantage of the streaming capabilities nifi offers. Usually the processor in
question reads the full content into an in-memory structure, converts that into
some larger and far more memory intensive structure such as a DOM, makes some
simple change, then serializes that back out. This is completely defeating the
power that NiFi offers. If you simply cannot get around the algorithm being
used needing the full object in memory try splitting the object if possible.
Otherwise, add memory to the JVM so it can handle these objects. But the best
way is indeed to read from the input stream chunks at a time, operate on them,
write them out. This is how NiFi can easily decompress a 1GB object to a 5GB
object without ever having more than a few 100KB of memory in use for that
operation.
If a custom processor is not picking up a flowfile from the queue then it is
possible it already grabbed it but never committed/completed its session.
Generating thread dumps is usually very informative here.
If the processor(s) in question are not custom please let the community know
which ones are causing problematic behavior.
> NiFI flow states one file struck in process and not getting cleared
> --------------------------------------------------------------------
>
> Key: NIFI-6202
> URL: https://issues.apache.org/jira/browse/NIFI-6202
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
> Affects Versions: 1.8.0
> Reporter: sayee
> Priority: Blocker
> Attachments: defect image.PNG
>
>
> Processing large files through NiFi:
> 1. Running a large number of data files through NiFi processor causes 1 file
> struck in Q and not getting cleared.
> 2. only way to clean is delete flow files from repository
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
