[
https://issues.apache.org/jira/browse/NIFI-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Gresock updated NIFI-2848:
---------------------------------
Description:
Consider the scenario where multiple queues lead to a single component and all
of them are full due to back pressure. With the attached template, it is
easily observable that once a single queue starts to drain due to relieved back
pressure, it will continue to drain as long as it has incoming flow files.
This means that if there's a constant flow of incoming flow files to this
queue, the other queues will never be drained (at least, that's my theory based
on several hours of observation).
To reproduce this:
# Load the template into NiFi 1.0.0
# Play all three GenerateFlowFile processors, but not the UpdateAttribute
processor (this simulates backpressure). Wait until each queue has 1,000 flow
files (max backpressure)
# Stop the GenerateFlowFile processors, and play the UpdateAttribute processor
(this relieves the backpressure)
# Observe which queue has started to drain, and start its GenerateFlowFile
processor
# Observe that the other two queues remain full indefinitely, while the
draining queue continues to replenish and be drained indefinitely
was:
Consider the scenario where multiple queues lead to a single component and all
of them are full due to back pressure. With the attached template, it is
easily observable that once a single queue starts to drain due to relieved back
pressure, it will continue to drain as long as it has incoming flow files.
This means that if there's a constant flow of incoming flow files to this
queue, the other queues will never be drained (at least, that's my theory based
on an hour of observation).
To reproduce this:
# Load the template into NiFi 1.0.0
# Play all three GenerateFlowFile processors, but not the UpdateAttribute
processor (this simulates backpressure). Wait until each queue has 1,000 flow
files (max backpressure)
# Stop the GenerateFlowFile processors, and play the UpdateAttribute processor
(this relieves the backpressure)
# Observe which queue has started to drain, and start its GenerateFlowFile
processor
# Observe that the other two queues remain full indefinitely, while the
draining queue continues to replenish and be drained indefinitely
> Queues aren't fairly drained when leading to a single component
> ---------------------------------------------------------------
>
> Key: NIFI-2848
> URL: https://issues.apache.org/jira/browse/NIFI-2848
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 1.0.0, 0.7.0
> Reporter: Joseph Gresock
> Attachments: Backpressure_prioritization_test.xml
>
>
> Consider the scenario where multiple queues lead to a single component and
> all of them are full due to back pressure. With the attached template, it is
> easily observable that once a single queue starts to drain due to relieved
> back pressure, it will continue to drain as long as it has incoming flow
> files. This means that if there's a constant flow of incoming flow files to
> this queue, the other queues will never be drained (at least, that's my
> theory based on several hours of observation).
> To reproduce this:
> # Load the template into NiFi 1.0.0
> # Play all three GenerateFlowFile processors, but not the UpdateAttribute
> processor (this simulates backpressure). Wait until each queue has 1,000
> flow files (max backpressure)
> # Stop the GenerateFlowFile processors, and play the UpdateAttribute
> processor (this relieves the backpressure)
> # Observe which queue has started to drain, and start its GenerateFlowFile
> processor
> # Observe that the other two queues remain full indefinitely, while the
> draining queue continues to replenish and be drained indefinitely
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
