slfan1989 commented on code in PR #9065:
URL: https://github.com/apache/hudi/pull/9065#discussion_r1244702211
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieMemoryConfig.java:
##########
@@ -140,7 +144,9 @@ private HoodieMemoryConfig() {
public static String getDefaultSpillableMapBasePath() {
String[] localDirs = FileIOUtils.getConfiguredLocalDirs();
- return (localDirs != null && localDirs.length > 0) ? localDirs[0] :
"/tmp/";
+ List<String> localDirLists = Arrays.asList(localDirs);
+ Collections.shuffle(localDirLists);
+ return !localDirLists.isEmpty() ? localDirLists.get(0) : "/tmp/";
Review Comment:
Thank you very much for helping with the code review! `LOCAL_DIRS` is an
environment variable set by YARN during Container startup, taking into
consideration disk space availability and cases where a disk may be
inaccessible.
- Hudi#FileIOUtils#getYarnLocalDirs
```
private static String getYarnLocalDirs() {
String localDirs = System.getenv("LOCAL_DIRS");
if (localDirs == null) {
throw new HoodieIOException("Yarn Local dirs can't be empty");
}
return localDirs;
}
```
- YARN
LinuxContainerExecutor#prepareContainer
```
builder.setExecutionAttribute(LOCALIZED_RESOURCES,
ctx.getLocalizedResources())
.setExecutionAttribute(USER, ctx.getUser())
.setExecutionAttribute(CONTAINER_LOCAL_DIRS,
ctx.getContainerLocalDirs())
.setExecutionAttribute(CONTAINER_RUN_CMDS, ctx.getCommands())
.setExecutionAttribute(CONTAINER_ID_STR,
ctx.getContainer().getContainerId().toString());
```
ContainerLaunch#call
```
List<String> containerLocalDirs = getContainerLocalDirs(localDirs);
if (truststore != null) {
addTruststoreVars(environment, containerWorkDir);
}
prepareContainer(localResources, containerLocalDirs);
// Write out the environment
exec.writeLaunchEnv(containerScriptOutStream, environment,
localResources, launchContext.getCommands(), containerLogDir, user,
nmEnvVars);
```
ContainerLaunch#prepareContainer
```
protected List<String> getContainerLocalDirs(List<String> localDirs) {
List<String> containerLocalDirs = new ArrayList<>(localDirs.size());
String user = container.getUser();
String appIdStr = app.getAppId().toString();
String relativeContainerLocalDir = ContainerLocalizer.USERCACHE
+ Path.SEPARATOR + user + Path.SEPARATOR +
ContainerLocalizer.APPCACHE
+ Path.SEPARATOR + appIdStr + Path.SEPARATOR;
for (String localDir : localDirs) {
containerLocalDirs.add(localDir + Path.SEPARATOR
+ relativeContainerLocalDir);
}
return containerLocalDirs;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
