[ 
https://issues.apache.org/jira/browse/NIFI-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Gresock updated NIFI-2848:
---------------------------------
    Description: 
Consider the scenario where multiple queues lead to a single component and all 
of them are full due to back pressure.  With the attached template, it is 
easily observable that once a single queue starts to drain due to relieved back 
pressure, it will continue to drain as long as it has incoming flow files.  
This means that if there's a constant flow of incoming flow files to this 
queue, the other queues will never be drained (at least, that's my theory based 
on an hour of observation).

To reproduce this: 
# Load the template into NiFi 1.0.0
# Play all three GenerateFlowFile processors, but not the UpdateAttribute 
processor (this simulates backpressure).  Wait until each queue has 1,000 flow 
files (max backpressure)
# Stop the GenerateFlowFile processors, and play the UpdateAttribute processor 
(this relieves the backpressure)
# Observe which queue has started to drain, and start its GenerateFlowFile 
processor
# Observe that the other two queues remain full indefinitely, while the 
draining queue continues to replenish and be drained indefinitely

  was:
Consider the scenario where multiple queues lead to a single component and all 
of them are full due to back pressure.  With the attached template, it is 
easily observable that once a single queue starts to drain due to relieved back 
pressure, it will continue to drain as long as it has incoming flow files.  
This means that if there's a constant flow of incoming flow files to this 
queue, the other two queues will never be drained (at least, that's my theory 
based on an hour of observation).

To reproduce this: 
# Load the template into NiFi 1.0.0
# Play all three GenerateFlowFile processors, but not the UpdateAttribute 
processor (this simulates backpressure).  Wait until each queue has 1,000 flow 
files (max backpressure)
# Stop the GenerateFlowFile processors, and play the UpdateAttribute processor 
(this relieves the backpressure)
# Observe which queue has started to drain, and start its GenerateFlowFile 
processor
# Observe that the other two queues remain full indefinitely, while the 
draining queue continues to replenish and be drained indefinitely


> Queues aren't fairly drained when leading to a single component
> ---------------------------------------------------------------
>
>                 Key: NIFI-2848
>                 URL: https://issues.apache.org/jira/browse/NIFI-2848
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 1.0.0, 0.7.0
>            Reporter: Joseph Gresock
>         Attachments: Backpressure_prioritization_test.xml
>
>
> Consider the scenario where multiple queues lead to a single component and 
> all of them are full due to back pressure.  With the attached template, it is 
> easily observable that once a single queue starts to drain due to relieved 
> back pressure, it will continue to drain as long as it has incoming flow 
> files.  This means that if there's a constant flow of incoming flow files to 
> this queue, the other queues will never be drained (at least, that's my 
> theory based on an hour of observation).
> To reproduce this: 
> # Load the template into NiFi 1.0.0
> # Play all three GenerateFlowFile processors, but not the UpdateAttribute 
> processor (this simulates backpressure).  Wait until each queue has 1,000 
> flow files (max backpressure)
> # Stop the GenerateFlowFile processors, and play the UpdateAttribute 
> processor (this relieves the backpressure)
> # Observe which queue has started to drain, and start its GenerateFlowFile 
> processor
> # Observe that the other two queues remain full indefinitely, while the 
> draining queue continues to replenish and be drained indefinitely



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to