[
https://issues.apache.org/jira/browse/FLINK-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijiang closed FLINK-15187.
----------------------------
Resolution: Fixed
Resolved in FLINK-15981
> Reuse LocalBufferPool for FileBufferReader in blocking partition
> ----------------------------------------------------------------
>
> Key: FLINK-15187
> URL: https://issues.apache.org/jira/browse/FLINK-15187
> Project: Flink
> Issue Type: Task
> Components: Runtime / Network
> Reporter: Zhijiang
> Priority: Minor
>
> If we take the file type via
> `taskmanager.network.bounded-blocking-subpartition-type` for batch job, while
> creating the respective view for reading the subpartition persistent data, it
> would create two unpolled memory segments for every subpartition. This
> portion of temporary memory is not managed and calculated by framework, so it
> might cause OOM error concern.
> We can also reuse the ResultPartition's `LocalBufferPool` to read
> subpartition data to avoid this memory overhead. But there are additional two
> problems for reuse directly.
> * The current core size of `LocalBufferPool` is `numberOfSubpartitions + 1`,
> but every subpartition needs two segments for pre-reading atm. We can remove
> the pre-reading to make the current core pool size suitable for the reading
> requirements, because the pre-reading function seems has no obvious benefits
> in practice which is only effecting for the last data.
> * When task finishes, it would destroy the `LocalBufferPool` even though the
> respective `ResultPartition still alive, so the following subpartition view
> can not reuse the pool directly. We should adjust the respective logics to
> either delay destroy the pool or create a new pool for subpartition view.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
