[
https://issues.apache.org/jira/browse/NIFI-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361231#comment-16361231
]
Joseph Percivall edited comment on NIFI-4871 at 2/12/18 6:31 PM:
-----------------------------------------------------------------
This should probably be handled in the same way as funnels, which has a while
loop[1] that transfers all the files in a single onTrigger but in 100 FlowFile
batches.
[1]
[https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/StandardFunnel.java#L365]
was (Author: jpercivall):
This should probably be handled in the same way was funnels, which has a while
loop[1] that transfers all the files in a single onTrigger but in 100 FlowFile
batches.
[1]
https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/StandardFunnel.java#L365
> Data Queuing at Process Group Input Port
> ----------------------------------------
>
> Key: NIFI-4871
> URL: https://issues.apache.org/jira/browse/NIFI-4871
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Brandon DeVries
> Priority: Major
>
> When a flow is under heavy load, we've seen data queue on the way in to a
> process group. There is nothing queued *inside* the process group, and no
> back pressure... it simply isn't moving files fast enough. As a workaround,
> multiple input ports were added... but that's ugly.
> The first issue may be the limit of 100 files transferred on any call to
> onTrigger()[1]. 100 seems low, ideally it would be nice to fill to the back
> pressure point (or some reasonable limit if set to "0"). This may also be
> compounded by some scheduling issues we've seen, but not reliably enough to
> actually diagnose. But the combination of not transferring enough files on
> each run and not running as often as it "should" is definitely causing
> issues. The scheduling issue may be complex, but transferring more files
> should be pretty straightforward...
>
> [1]
> [https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/connectable/LocalPort.java#L76]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
