[
https://issues.apache.org/jira/browse/FLINK-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16732053#comment-16732053
]
Piotr Nowojski commented on FLINK-11082:
----------------------------------------
> Furthermore, the backlog would affect requesting floating buffers on
> downstream side. That means some floating buffers are fetched in advance but
> not be used for long time, so the floating buffers are not made use of
> efficiently.
I think that wouldn't be an issue, if floating vs exclusive were abstract
concept. Like if we assign a floating credit to a channel (for example bumping
number of assigned buffers from 2 up to 3), once we release ANY buffer from
this channel and the number goes back down to "2", it could be interpreted/be
an equivalent of saying that the channel released floating buffer and currently
holds only to two exclusive buffers. As far as I remember, currently that's not
the case, right? If we have assigned 2 exclusive buffers, then we assign one
floating buffer, floating buffer is released only we process all three of those
buffers, right?
Maybe that's the root problem that we should fix? I think I vaguely remember
this same behaviour causing some other problems.
> Increase backlog only if it is available for consumption
> --------------------------------------------------------
>
> Key: FLINK-11082
> URL: https://issues.apache.org/jira/browse/FLINK-11082
> Project: Flink
> Issue Type: Sub-task
> Components: Network
> Affects Versions: 1.8.0
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Minor
>
> The backlog should indicate how many buffers are available in subpartition
> for downstream's consumption. The availability is considered from two
> factors. One is {{BufferConsumer}} finished, and the other is flush triggered.
> In current implementation, when the {{BufferConsumer}} is added into the
> subpartition, then the backlog is increased as a result, but this
> {{BufferConsumer}} is not yet available for network transport.
> Furthermore, the backlog would affect requesting floating buffers on
> downstream side. That means some floating buffers are fetched in advance but
> not be used for long time, so the floating buffers are not made use of
> efficiently.
> We found this scenario extremely for rebalance selector on upstream side, so
> we want to change when to increase backlog by finishing {{BufferConsumer}} or
> flush triggered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
