[
https://issues.apache.org/jira/browse/FLINK-28623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yingjie Cao resolved FLINK-28623.
---------------------------------
Fix Version/s: 1.16.0
Resolution: Fixed
Merged into master via 87d4f70e49255b551d0106117978b1aa0747358c
> Optimize the use of off heap memory by blocking and hybrid shuffle reader
> -------------------------------------------------------------------------
>
> Key: FLINK-28623
> URL: https://issues.apache.org/jira/browse/FLINK-28623
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Reporter: Weijie Guo
> Assignee: Weijie Guo
> Priority: Minor
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> Currently, each FileReader(PartitionFileReader or
> HsSubpartitionFileReaderImpl) will internally allocate a headerbuffer with
> the size of 8B. Beside, PartitionFileReader also has a 12B indexEntryBuf.
> Because FileReader is of subpartition granularity, If the parallelism becomes
> very big, and there are many slots on each TM, the memory occupation will
> even reach the MB level. In fact, all FileReader of the same ResultPartition
> read data in a single thread, so we only need to allocate a headerbuffer to a
> ResultPartition to optimize this phenomenon.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
