[
https://issues.apache.org/jira/browse/FLINK-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739380#comment-16739380
]
Piotr Nowojski commented on FLINK-11082:
----------------------------------------
[~zjwang] why should we report backlog of -1? Shouldn't your last point work
like that:
- Peek the last buffer from queue, the flushRequested is set false. Regardless
f this last buffer is finished or not, the backlog should be decreased to 0. If
the buffer is finished, pop it from the queue. If the buffer is not finished,
keep it with backlog of 0.
- Backlog increases to 1 if we flush or if we enqueue second buffer.
> Increase backlog only if it is available for consumption
> --------------------------------------------------------
>
> Key: FLINK-11082
> URL: https://issues.apache.org/jira/browse/FLINK-11082
> Project: Flink
> Issue Type: Sub-task
> Components: Network
> Affects Versions: 1.5.6, 1.6.3, 1.7.1, 1.8.0
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Major
>
> The backlog should indicate how many buffers are available in subpartition
> for downstream's consumption. The availability is considered from two
> factors. One is {{BufferConsumer}} finished, and the other is flush triggered.
> In current implementation, when the {{BufferConsumer}} is added into the
> subpartition, then the backlog is increased as a result, but this
> {{BufferConsumer}} is not yet available for network transport.
> Furthermore, the backlog would affect requesting floating buffers on
> downstream side. That means some floating buffers are fetched in advance but
> not be used for long time, so the floating buffers are not made use of
> efficiently.
> We found this scenario extremely for rebalance selector on upstream side, so
> we want to change when to increase backlog by finishing {{BufferConsumer}} or
> flush triggered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
