[
https://issues.apache.org/jira/browse/FLINK-28623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yingjie Cao reassigned FLINK-28623:
-----------------------------------
Assignee: Weijie Guo
> Optimize the use of off heap memory by blocking and hybrid shuffle reader
> -------------------------------------------------------------------------
>
> Key: FLINK-28623
> URL: https://issues.apache.org/jira/browse/FLINK-28623
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Reporter: Weijie Guo
> Assignee: Weijie Guo
> Priority: Minor
> Labels: pull-request-available
>
> Currently, each FileReader(PartitionFileReader or
> HsSubpartitionFileReaderImpl) will internally allocate a headerbuffer with
> the size of 8B. Beside, PartitionFileReader also has a 12B indexEntryBuf.
> Because FileReader is of subpartition granularity, If the parallelism becomes
> very big, and there are many slots on each TM, the memory occupation will
> even reach the MB level. In fact, all FileReader of the same ResultPartition
> read data in a single thread, so we only need to allocate a headerbuffer to a
> ResultPartition to optimize this phenomenon.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
