zhijiang created FLINK-11082:
--------------------------------
Summary: Increase backlog only if it is available for consumption
Key: FLINK-11082
URL: https://issues.apache.org/jira/browse/FLINK-11082
Project: Flink
Issue Type: Sub-task
Components: Network
Affects Versions: 1.8.0
Reporter: zhijiang
Assignee: zhijiang
The backlog should indicate how many buffers are available in subpartition for
downstream's consumption. The availability is considered from two factors. One
is {{BufferConsumer}} finished, and the other is flush triggered.
In current implementation, when the {{BufferConsumer}} is added into the
subpartition, then the backlog is increased as a result, but this
{{BufferConsumer}} is not yet available for network transport.
Furthermore, the backlog would affect requesting floating buffers on downstream
side. That means some floating buffers are fetched in advance but not be used
for long time, so the floating buffers are not made use of efficiently.
We found this scenario extremely for rebalance selector on upstream side, so we
want to change when to increase backlog by finishing {{BufferConsumer}} or
flush triggered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
