xushiyan commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r994473005
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -93,9 +94,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String
basePath, List<Stri
instantRange, withOperationField,
forceFullScan, partitionName, internalSchema);
try {
+ String[] localDirs = FileIOUtils.getConfiguredLocalDirs();
+ spillableMapBasePath = localDirs.length > 0 ? localDirs[0] :
spillableMapBasePath;
Review Comment:
this is to silently overwrite user-input config, which we should avoid. We
should always respect user-provided setting, then fallback to infer from
environment, then fallback to a sane default like the existing `/tmp/`. I'd
suggest improve the `SPILLABLE_MAP_BASE_PATH` with inferring function.
```java
public static final ConfigProperty<String> SPILLABLE_MAP_BASE_PATH =
ConfigProperty
.key("hoodie.memory.spillable.map.path")
.defaultValue("/tmp/")
.withInferFunction(() -> {
// if user doesn't configure it, infer the path from env var
// if nothing to infer from, return Option.empty() then it will
fallback to the default /tmp/
})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
