[
https://issues.apache.org/jira/browse/NIFI-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988633#comment-14988633
]
Joseph Witt commented on NIFI-752:
----------------------------------
this is like *x1000* better now. Nicely done mr payne.
I am mostly seriously about this next comment - since we swap out in batches of
roughly 10000 flow files at a time and they create files in a flattened
structure i think we can handle a backlog of somewhere south of 1Bil flowfiles
before performance issues will occur due to number of files in a directory.
This is ... pretty good but I wonder if there are some pathological cases that
could be more problematic.
> Swapping and out of FlowFiles should be driven by FlowFileQueue, rather than
> in background thread
> -------------------------------------------------------------------------------------------------
>
> Key: NIFI-752
> URL: https://issues.apache.org/jira/browse/NIFI-752
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Mark Payne
> Fix For: 0.4.0
>
>
> As-is, if a huge number of FlowFiles are generated quickly enough, the JVM
> heap can fill before FlowFiles are swapped in. Conversely, if FlowFiles are
> swapped out, are NiFi is able to work those FlowFiles off, it ends up
> pausing, waiting for more FlowFiles to be swapped in. If the queue itself has
> control of this, these issues will both be alleviated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
