[
https://issues.apache.org/jira/browse/FLINK-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739246#comment-16739246
]
zhijiang edited comment on FLINK-11082 at 1/10/19 9:42 AM:
-----------------------------------------------------------
Considering implementation, it seems a bit complicated for both finished
{{BufferBuilder}} and flush factors.
Increase backlog could be triggered by two conditions which are adding from
second {{BufferConsumer}} and flush triggered.
Decrease backlog could also be triggered by two conditions which are popping
finished {{BufferConsumer}} from queue and {{flushRequested}} set as false for
the last element in queue. So it might cause the backlog to be {{-1}}
sometimes. For example:
* Two BufferConsumers in queue, the second buffer is unfinished, then the
current backlog is 1.
* Add one more {{BufferConsumer}} in queue, three buffers in queue, then
backlog is 2.
* Flush triggered, still three BufferConsumers in queue, then backlog is 3.
* Pop one {{BufferConsumer}} from queue, two buffers left in queue, backlog is
2.
* Pop one {{BufferConsumer}} from queue again, one buffer left in queue,
backlog is 1.
* Peek the last buffer from queue, the {{flushRequested}} is set false. If
this last buffer is not finished, the backlog should be decreased to 0 and it
seems no problem. But if the last buffer is already finished, then it would be
popped from queue and decreased again to be -1. When the new {{BufferConsumer}}
is added into queue, the backlog will increase to 0.
So the backlog might encounter decrease twice for the last buffer in queue.
Although the backlog is actually correct, it might seem strange for the value
of -1. What do you think?
was (Author: zjwang):
Considering implementation, it seems a bit complicated for both finished
{{BufferBuilder}} and flush factors.
Increase backlog could be triggered by two conditions which are adding from
second {{BufferConsumer}} and flush triggered.
Decrease backlog could also be triggered by two conditions which are popping
finished {{BufferConsumer}} from queue and {{flushRequested}} set as false for
the last element in queue. So it might cause the backlog to be {{-1}}
sometimes. For example:
* Two {{BufferConsumer}}s in queue, the second buffer is unfinished, then the
current backlog is 1.
* Add one more {{BufferConsumer}} in queue, three buffers in queue, then
backlog is 2.
* Flush triggered, still three {{BufferConsumer}}s in queue, then backlog is 3.
* Pop one {{BufferConsumer}} from queue, two buffers left in queue, backlog is
2.
* Pop one {{BufferConsumer}} from queue again, one buffer left in queue,
backlog is 1.
* Peek the last buffer from queue, the {{flushRequested}} is set false. If
this last buffer is not finished, the backlog should be decreased to 0 and it
seems no problem. But if the last buffer is already finished, then it would be
popped from queue and decreased again to be -1. When the new {{BufferConsumer}}
is added into queue, the backlog will increase to 0.
So the backlog might encounter decrease twice for the last buffer in queue.
Although the backlog is actually correct, it might seem strange for the value
of -1. What do you think?
> Increase backlog only if it is available for consumption
> --------------------------------------------------------
>
> Key: FLINK-11082
> URL: https://issues.apache.org/jira/browse/FLINK-11082
> Project: Flink
> Issue Type: Sub-task
> Components: Network
> Affects Versions: 1.5.6, 1.6.3, 1.7.1, 1.8.0
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Major
>
> The backlog should indicate how many buffers are available in subpartition
> for downstream's consumption. The availability is considered from two
> factors. One is {{BufferConsumer}} finished, and the other is flush triggered.
> In current implementation, when the {{BufferConsumer}} is added into the
> subpartition, then the backlog is increased as a result, but this
> {{BufferConsumer}} is not yet available for network transport.
> Furthermore, the backlog would affect requesting floating buffers on
> downstream side. That means some floating buffers are fetched in advance but
> not be used for long time, so the floating buffers are not made use of
> efficiently.
> We found this scenario extremely for rebalance selector on upstream side, so
> we want to change when to increase backlog by finishing {{BufferConsumer}} or
> flush triggered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
