Bo Cui created HUDI-3460:
----------------------------
Summary: Flink TM Memory Optimization
Key: HUDI-3460
URL: https://issues.apache.org/jira/browse/HUDI-3460
Project: Apache Hudi
Issue Type: Improvement
Components: flink
Reporter: Bo Cui
HoodieTableSource uses the MapReduce Memory Config,
[https://github.com/apache/hudi/blob/359fbfde798b50edc06ee1d0520efcd971a289bc/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java#L81]
but the config value may be the `MaximumHeapSizeMB * 0.8`, excessive memory
usage affects flink TM stability
(https://github.com/apache/hadoop/blob/007c2011efe552d54956dce35e8a3ee095cdef2c/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java#L1966)
so HoodieTableSource should uses FlinkOptions's own config and 64MB(Flink
rocksdb default is 64MB).
and write.merge.max_memory,compaction.max_memory should also be changed to 64MB
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
