[
https://issues.apache.org/jira/browse/NIFI-6157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16813487#comment-16813487
]
Brandon DeVries edited comment on NIFI-6157 at 4/9/19 2:35 PM:
---------------------------------------------------------------
[~markbean], PR updated
was (Author: devriesb):
PR updated
> StandardFunnel transferring too slowly
> --------------------------------------
>
> Key: NIFI-6157
> URL: https://issues.apache.org/jira/browse/NIFI-6157
> Project: Apache NiFi
> Issue Type: Improvement
> Affects Versions: 1.9.0, 1.9.1
> Reporter: Brandon DeVries
> Assignee: Brandon DeVries
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> NIFI-6068 made modifications such that a funnel wouldn't hold on to a
> TimerDriven thread excessively. However, now it isn't holding on to the
> thread long enough...
> Since Funnels and Local Ports are scheduled with the timer driven thread
> pool, they're competing for threads with all of the other processors on the
> graph. In a large flow with a large number of processors, potentially with
> multiple assigned concurrent tasks, funnels and ports get to run less and
> less frequently, since they are hard coded to 1 concurrent task.
> I'm open to implementation options, but a couple of possibilities are:
> * Increase the transferred FlowFilecap from 10K to 100K. The thread will
> still be released if less than the requested 1000 FlowFiles are moved in a
> loop, so it won't hold on inappropriately, but it will still have the
> opportunity to move the files that need to be moved. Furthermore, if the
> back pressure on the outgoing relationship is engaged, it will cause the
> thread to be released. Effectively, the amount transferred would be limited
> by the max of 100K and outgoing queue capacity.
> * Like above, but add a property to specify the max number of FlowFiles
> transferred per run. Removing hard coded magic numbers is good... but
> cluttering nifi.properties is bad, so its a trade off.
> * Increase the number of concurrent threads for funnels / ports. This
> probably would want to be a configurable property, as the value should really
> likely be proportional to the "size" of your flow, whatever that means for
> the system in question.
> * Increase the "run duration"... but i don't think i like that.
> * If session.getQueueSize exceeds some threshold, spin off a new thread to
> transfer those files... but that could be dangerous.
> * Create a new thread pool for ports / funnels, so they aren't starved by
> processors. Similar to above, but reuses resources. Still would need to
> determine the correct size of the pool. This could be the best answer in
> theory, but would also require the most code work.
> [~markap14], thoughts?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
