[
https://issues.apache.org/jira/browse/HUDI-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17495346#comment-17495346
]
Bo Cui commented on HUDI-3460:
------------------------------
hi [~danny0405] WDYT?
> Flink TM Memory Optimization
> ----------------------------
>
> Key: HUDI-3460
> URL: https://issues.apache.org/jira/browse/HUDI-3460
> Project: Apache Hudi
> Issue Type: Improvement
> Components: flink
> Reporter: Bo Cui
> Priority: Major
>
> HoodieTableSource uses the MapReduce Memory Config,
> [https://github.com/apache/hudi/blob/359fbfde798b50edc06ee1d0520efcd971a289bc/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java#L81]
> but the config value may be the `MaximumHeapSizeMB * 0.8`, excessive memory
> usage affects flink TM stability
> (https://github.com/apache/hadoop/blob/007c2011efe552d54956dce35e8a3ee095cdef2c/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java#L1966)
> so HoodieTableSource should uses FlinkOptions's own config and 64MB(Flink
> rocksdb default is 64MB).
>
> and write.merge.max_memory,compaction.max_memory should also be changed to
> 64MB
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
