[
https://issues.apache.org/jira/browse/FLINK-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880427#comment-16880427
]
zhijiang commented on FLINK-11082:
----------------------------------
[~pnowojski], if the requested floating buffers are not used in idle channels,
then they would be returned back to local buffer pool eventually with all the
exclusive buffers recycled. E.g.
* If the current backlog is 2 and only 1 exclusive buffer is available in
remote channel, then it would request (initial credit + backlog -
numAvailableBuffers) 3 floating buffers.
* If the backlog is 1 and total 3 buffers available (all floating buffers) in
remote channel.
* If the backlog is 0, we would return 1 floating buffer back to local buffer
pool, keep 2 available buffers(both floating) in remote channel.
* If the backlog is 0, and one exclusive buffer is processed to recycle to
remote channel, then it triggers returning one floating buffer back to local
pool, still two buffers available in remote channel(one floating and one
exclusive).
* If the backlog is 0, and another exclusive buffer is processed to recycle to
remote channel, then it also triggers returning one floating buffer back to
local pool, still two buffers available in remote channel(both are exclusive).
So the floating buffers would not be occupied by remote channel long time if
there are already (backlog + initial credit) available buffers in remote
channel.
> Fix the calculation of backlog in PipelinedSubpartition
> -------------------------------------------------------
>
> Key: FLINK-11082
> URL: https://issues.apache.org/jira/browse/FLINK-11082
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Affects Versions: 1.5.6, 1.7.1
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The backlog of subpartition should indicate how many buffers are consumable,
> then the consumer could feedback the corresponding credits for transporting
> these buffers. But in current PipelinedSubpartitionimplementation, the
> backlog is increased by 1 when a BufferConsumer is added into
> PipelinedSubpartition, and decreased by 1 when a BufferConsumer is removed
> from PipelinedSubpartition. So the backlog only reflects how many buffers are
> retained in PipelinedSubpartition, which is not always equivalent to the
> number of consumable buffers.
> The backlog inconsistency might result in floating buffers misdistribution on
> consumer side, because the consumer would request floating buffers based on
> backlog value, then one floating buffer might not be used in
> RemoteInputChannel long time after requesting.
> Considering the solution, the last buffer in PipelinedSubpartition could only
> be consumable in the case of flush triggered or partition finished. So we
> could calculate the backlog precisely based on partition flushed/finished
> conditions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
