[
https://issues.apache.org/jira/browse/FLINK-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880476#comment-16880476
]
Nico Kruber commented on FLINK-11082:
-------------------------------------
Yes, [~zjwang] - nonetheless, the floating buffers sit idle until an exclusive
buffer comes back (even if the backlog is 0) which I was worried about at first.
However, [~pnowojski] convinced me that this may not be a big issue after all:
We are kind of making the assumption, that {{#exclusive_buffers +
#floating_buffers}} is enough to achieve full throughput but even if some
floating buffers sit around being idle, we will in that case have filled
exclusive buffers to work on: in total at least {{#exclusive_buffers +
#floating_buffers}} which is fine.
We just have to keep it in mind when reasoning about the use of the floating
buffers, e.g. a high-load channel should have the floating buffers available
but as soon as any other channel has 1 buffer at least, it will also fight for
a floating buffer even if it does not use it at the moment. Because of the
reasoning above, it should still be ok.
> Fix the calculation of backlog in PipelinedSubpartition
> -------------------------------------------------------
>
> Key: FLINK-11082
> URL: https://issues.apache.org/jira/browse/FLINK-11082
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Affects Versions: 1.5.6, 1.7.1
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The backlog of subpartition should indicate how many buffers are consumable,
> then the consumer could feedback the corresponding credits for transporting
> these buffers. But in current PipelinedSubpartitionimplementation, the
> backlog is increased by 1 when a BufferConsumer is added into
> PipelinedSubpartition, and decreased by 1 when a BufferConsumer is removed
> from PipelinedSubpartition. So the backlog only reflects how many buffers are
> retained in PipelinedSubpartition, which is not always equivalent to the
> number of consumable buffers.
> The backlog inconsistency might result in floating buffers misdistribution on
> consumer side, because the consumer would request floating buffers based on
> backlog value, then one floating buffer might not be used in
> RemoteInputChannel long time after requesting.
> Considering the solution, the last buffer in PipelinedSubpartition could only
> be consumable in the case of flush triggered or partition finished. So we
> could calculate the backlog precisely based on partition flushed/finished
> conditions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
