[
https://issues.apache.org/jira/browse/NIFI-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-9689:
-------------------------------
Description:
When all FlowFiles in a given queue are penalized, the destination processor is
still scheduled to run. As a result, the Processor is triggered, no data is
available, and it generally returns. However, this means that the UI shows
millions of tasks completed over a short period of time, when no work was
actually done.
Further, it means that we constantly schedule the processor to run as fast as
it can, which wastes compute resources. Instead, we should detect that the head
of the queue is penalized (and therefore all FlowFiles in the queue are
penalized), and treat it the same as we would if the queue were empty, which
allows the "bored yield duration" to take effect, significantly reducing CPU
cycles.
was:
When all FlowFiles in a given queue are penalized, the destination processor is
still scheduled to run. As a result, the Processor is triggered, no data is
available, and it generally returns. However, this means that the UI shows
millions of tasks completed over a short period of time, when no where was
actually done.
Further, it means that we constantly schedule the processor to run as fast as
it can, which wastes compute resources. Instead, we should detect that the head
of the queue is penalized (and therefore all FlowFiles in the queue are
penalized), and treat it the same as we would if the queue were empty, which
allows the "bored yield duration" to take effect, significantly reducing CPU
cycles.
> Do not schedule processors when all FlowFiles are penalized
> -----------------------------------------------------------
>
> Key: NIFI-9689
> URL: https://issues.apache.org/jira/browse/NIFI-9689
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When all FlowFiles in a given queue are penalized, the destination processor
> is still scheduled to run. As a result, the Processor is triggered, no data
> is available, and it generally returns. However, this means that the UI shows
> millions of tasks completed over a short period of time, when no work was
> actually done.
> Further, it means that we constantly schedule the processor to run as fast as
> it can, which wastes compute resources. Instead, we should detect that the
> head of the queue is penalized (and therefore all FlowFiles in the queue are
> penalized), and treat it the same as we would if the queue were empty, which
> allows the "bored yield duration" to take effect, significantly reducing CPU
> cycles.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
