yihua commented on code in PR #12647:
URL: https://github.com/apache/hudi/pull/12647#discussion_r1960716957
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieCommonConfig.java:
##########
@@ -136,6 +136,19 @@ public class HoodieCommonConfig extends HoodieConfig {
.markAdvanced()
.withDocumentation("Property to control the max memory in bytes for dfs
input stream buffer size");
+ public static final ConfigProperty<Boolean>
HOODIE_FILE_INDEX_USE_SPILLABLE_MAP = ConfigProperty
+ .key("hoodie.file.index.cache.use.spillable.map")
+ .defaultValue(false)
+ .markAdvanced()
+ .withDocumentation("Property to enable spillable map for caching input
file slices in org.apache.hudi.BaseHoodieTableFileIndex");
+
+ public static final ConfigProperty<Long> HOODIE_FILE_INDEX_SPILLABLE_MEMORY
= ConfigProperty
+ .key("hoodie.file.index.cache.spillable.mem")
+ .defaultValue(500 * 1024L * 1024L) // 500 MB
+ .markAdvanced()
+ .sinceVersion("0.16.0")
Review Comment:
```suggestion
.sinceVersion("1.1.0")
```
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieCommonConfig.java:
##########
@@ -136,6 +136,19 @@ public class HoodieCommonConfig extends HoodieConfig {
.markAdvanced()
.withDocumentation("Property to control the max memory in bytes for dfs
input stream buffer size");
+ public static final ConfigProperty<Boolean>
HOODIE_FILE_INDEX_USE_SPILLABLE_MAP = ConfigProperty
+ .key("hoodie.file.index.cache.use.spillable.map")
+ .defaultValue(false)
Review Comment:
Could we turn this on by default as it's validated by production workload?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieCommonConfig.java:
##########
@@ -136,6 +136,19 @@ public class HoodieCommonConfig extends HoodieConfig {
.markAdvanced()
.withDocumentation("Property to control the max memory in bytes for dfs
input stream buffer size");
+ public static final ConfigProperty<Boolean>
HOODIE_FILE_INDEX_USE_SPILLABLE_MAP = ConfigProperty
+ .key("hoodie.file.index.cache.use.spillable.map")
+ .defaultValue(false)
+ .markAdvanced()
+ .withDocumentation("Property to enable spillable map for caching input
file slices in org.apache.hudi.BaseHoodieTableFileIndex");
Review Comment:
```suggestion
.sinceVersion("1.1.0")
.withDocumentation("Property to enable spillable map for caching input
file slices in org.apache.hudi.BaseHoodieTableFileIndex");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
