[
https://issues.apache.org/jira/browse/NIFI-6157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818002#comment-16818002
]
ASF subversion and git services commented on NIFI-6157:
-------------------------------------------------------
Commit cbc07eb663cd4cb67fc13fad93813b60204518c5 in nifi's branch
refs/heads/master from Brandon Rhys DeVries
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=cbc07eb ]
NIFI-6157: Allowing configuration of concurrent tasks and transferred FlowFile
count to Funnels and LocalPorts
This closes #3412
Signed-off-by: Brandon <[email protected]>
> StandardFunnel transferring too slowly
> --------------------------------------
>
> Key: NIFI-6157
> URL: https://issues.apache.org/jira/browse/NIFI-6157
> Project: Apache NiFi
> Issue Type: Improvement
> Affects Versions: 1.9.0, 1.9.1
> Reporter: Brandon DeVries
> Assignee: Brandon DeVries
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> NIFI-6068 made modifications such that a funnel wouldn't hold on to a
> TimerDriven thread excessively. However, now it isn't holding on to the
> thread long enough...
> Since Funnels and Local Ports are scheduled with the timer driven thread
> pool, they're competing for threads with all of the other processors on the
> graph. In a large flow with a large number of processors, potentially with
> multiple assigned concurrent tasks, funnels and ports get to run less and
> less frequently, since they are hard coded to 1 concurrent task.
> I'm open to implementation options, but a couple of possibilities are:
> * Increase the transferred FlowFilecap from 10K to 100K. The thread will
> still be released if less than the requested 1000 FlowFiles are moved in a
> loop, so it won't hold on inappropriately, but it will still have the
> opportunity to move the files that need to be moved. Furthermore, if the
> back pressure on the outgoing relationship is engaged, it will cause the
> thread to be released. Effectively, the amount transferred would be limited
> by the max of 100K and outgoing queue capacity.
> * Like above, but add a property to specify the max number of FlowFiles
> transferred per run. Removing hard coded magic numbers is good... but
> cluttering nifi.properties is bad, so its a trade off.
> * Increase the number of concurrent threads for funnels / ports. This
> probably would want to be a configurable property, as the value should really
> likely be proportional to the "size" of your flow, whatever that means for
> the system in question.
> * Increase the "run duration"... but i don't think i like that.
> * If session.getQueueSize exceeds some threshold, spin off a new thread to
> transfer those files... but that could be dangerous.
> * Create a new thread pool for ports / funnels, so they aren't starved by
> processors. Similar to above, but reuses resources. Still would need to
> determine the correct size of the pool. This could be the best answer in
> theory, but would also require the most code work.
> [~markap14], thoughts?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
