[
https://issues.apache.org/jira/browse/MINIFICPP-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17297967#comment-17297967
]
Ferenc Gerlits edited comment on MINIFICPP-1487 at 3/9/21, 9:32 AM:
--------------------------------------------------------------------
This solution has the problem that if the processing of a flow file repeatedly
throws an exception, it can block the incoming queue.
If the flow file throws an exception, we will penalize an re-push the flow
file, and the processor will yield. By default, both the penalty period and
the yield duration are 30 seconds.
Since we always pop from the priority queue in preference to the normal queue
if the priority queue contains a flow file with an expired penalty, we will
always pick up the faulty flow file, and not process any other flow files from
the normal queue.
A possible solution is to introduce exponential backoff, ie. double the penalty
duration each time when it is a repeated penalty (with some reasonable maximum
penalty duration).
was (Author: fgerlits):
This solution has the problem that if the processing of a flow file repeatedly
throws an exception, it can block the incoming queue.
If the flow file throws and exception, we will penalize an re-push the flow
file, and the processor will yield. By default, both the penalty period and
the yield duration are 30 seconds.
Since we always pop from the priority queue in preference to the normal queue
if the priority queue contains a flow file with an expired penalty, we will
always pick up the faulty flow file, and not process any other flow files from
the normal queue.
A possible solution is to introduce exponential backoff, ie. double the penalty
duration each time when it is a repeated penalty (with some reasonable maximum
penalty duration).
> Better handling of a penalized-only input queue
> -----------------------------------------------
>
> Key: MINIFICPP-1487
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1487
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
> Reporter: Ferenc Gerlits
> Assignee: Ferenc Gerlits
> Priority: Minor
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> If the input queue of an Event Driven processor (is not empty but) contains
> penalized flow files only, then the processor will be woken up repeatedly in
> a tight loop, but it will have nothing to do, as it is not able to pick up
> any flow files. This results in 100% CPU usage (of one core).
> One way to avoid this would be if all processors yielded when they have
> nothing to do. Some processors do this, but not all. It would be better to
> solve this problem on the framework level rather than hope that processors
> will work around it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
