sivabalan narayanan created HUDI-7979:
-----------------------------------------
Summary: Fix out of the box defaults with spillable memory configs
Key: HUDI-7979
URL: https://issues.apache.org/jira/browse/HUDI-7979
Project: Apache Hudi
Issue Type: Improvement
Components: reader-core, writer-core
Reporter: sivabalan narayanan
Looks like we are very conservative wrt memory configs used for spillable map
based FSV.
For eg, we are only allocating 15Mb out of the box to file groups when using
spillable map based FSV.
public long getMaxMemoryForFileGroupMap() \{
long totalMemory = getLong(SPILLABLE_MEMORY);
return totalMemory - getMaxMemoryForPendingCompaction() -
getMaxMemoryForBootstrapBaseFile();
}
SPILLABLE_MEMORY = default is 100Mb.
getMaxMemoryForPendingCompaction = 80% of 100MB.
getMaxMemoryForBootstrapBaseFile = 5% of 100Mb.
so, overall, out of the box we are allocating only 15Mb for
getMaxMemoryForFileGroupMap.
ref:
[https://github.com/apache/hudi/blob/bb0621edee97507cf2460e8cb57b5307510b917e/hudi-[…]/apache/hudi/common/table/view/FileSystemViewStorageConfig.java|https://github.com/apache/hudi/blob/bb0621edee97507cf2460e8cb57b5307510b917e/hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewStorageConfig.java#L224]
Wondering do we even need 80% for pending compaction tracker in our FSV. I am
thinking to make it 15%. so that we can give more memory to actual file groups.
We may not have lot of pending compactions for a given table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
