[
https://issues.apache.org/jira/browse/FLINK-11037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713032#comment-16713032
]
Till Rohrmann commented on FLINK-11037:
---------------------------------------
Thanks for opening this issue [~zjwang]. From a high level perspective your
proposal makes sense. I guess the next step would be to dig a bit more into the
details:
- How should the greedy algorithm exactly look like? Take all what you can get
wrt to the buffer queue on the sender's side?
- How will the interface of the {{LocalBufferPool}} change? Will it be
something like {{Collection<Buffer> tryRequestBuffers(int numberBuffers)}}
where {{numberBuffers}} does not need to be fulfilled by the {{LocalBufferPool}}
- How will the user choose the algorithm?
> Introduce another greedy mechanism for distributing floating buffers
> --------------------------------------------------------------------
>
> Key: FLINK-11037
> URL: https://issues.apache.org/jira/browse/FLINK-11037
> Project: Flink
> Issue Type: Sub-task
> Components: Network
> Affects Versions: 1.8.0
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Minor
>
> The current mechanism for distributing floating buffers is fair for all the
> listeners. In detail, each input channel can only request one floating buffer
> each time although this channel may actually need more floating buffers. Then
> this channel has to loop to request floating buffer until all are satisfied
> or pool is exhausted.
> In generally speaking, this way seems fair for all the concurrent channels
> invoked by netty nio thread. But every request from LocalBufferPool needs to
> syn lock and it is hard to say how to distribute all the available floating
> buffers behaves better in real scenarios.
> Therefore we propose another greedy mechanism to request more floating
> buffers each time. In extreme case, we can even request all the required
> buffers at a time or partial ones via configured parameters. On the other
> side, LocalBufferPool can also decide how many floating buffers should been
> assigned based on some factors, such as how many total channels and how many
> total floating buffers.
> The motivation is making better use of floating buffer resources and it may
> need extra metrics for adjusting the mechanism dynamically.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
