[ https://issues.apache.org/jira/browse/FLINK-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhijiang closed FLINK-15187. ---------------------------- Resolution: Fixed Resolved in FLINK-15981 > Reuse LocalBufferPool for FileBufferReader in blocking partition > ---------------------------------------------------------------- > > Key: FLINK-15187 > URL: https://issues.apache.org/jira/browse/FLINK-15187 > Project: Flink > Issue Type: Task > Components: Runtime / Network > Reporter: Zhijiang > Priority: Minor > > If we take the file type via > `taskmanager.network.bounded-blocking-subpartition-type` for batch job, while > creating the respective view for reading the subpartition persistent data, it > would create two unpolled memory segments for every subpartition. This > portion of temporary memory is not managed and calculated by framework, so it > might cause OOM error concern. > We can also reuse the ResultPartition's `LocalBufferPool` to read > subpartition data to avoid this memory overhead. But there are additional two > problems for reuse directly. > * The current core size of `LocalBufferPool` is `numberOfSubpartitions + 1`, > but every subpartition needs two segments for pre-reading atm. We can remove > the pre-reading to make the current core pool size suitable for the reading > requirements, because the pre-reading function seems has no obvious benefits > in practice which is only effecting for the last data. > * When task finishes, it would destroy the `LocalBufferPool` even though the > respective `ResultPartition still alive, so the following subpartition view > can not reuse the pool directly. We should adjust the respective logics to > either delay destroy the pool or create a new pool for subpartition view. -- This message was sent by Atlassian Jira (v8.3.4#803005)