zhijiang created FLINK-15187:
--------------------------------
Summary: Reuse LocalBufferPool for FileBufferReader in blocking
partition
Key: FLINK-15187
URL: https://issues.apache.org/jira/browse/FLINK-15187
Project: Flink
Issue Type: Task
Components: Runtime / Network
Reporter: zhijiang
If we take the file type via
`taskmanager.network.bounded-blocking-subpartition-type` for batch job, while
creating the respective view for reading the subpartition persistent data, it
would create two unpolled memory segments for every subpartition. This portion
of temporary memory is not managed and calculated by framework, so it might
cause OOM error concern.
We can also reuse the ResultPartition's `LocalBufferPool` to read subpartition
data to avoid this memory overhead. But there are additional two problems for
reuse directly.
* The current core size of `LocalBufferPool` is `numberOfSubpartitions + 1`,
but every subpartition needs two segments for pre-reading atm. We can remove
the pre-reading to make the current core pool size suitable for the reading
requirements, because the pre-reading function seems has no obvious benefits in
practice which is only effecting for the last data.
* When task finishes, it would destroy the `LocalBufferPool` even though the
respective `ResultPartition still alive, so the following subpartition view can
not reuse the pool directly. We should adjust the respective logics to either
delay destroy the pool or create a new pool for subpartition view.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
