slfan1989 commented on code in PR #9065:
URL: https://github.com/apache/hudi/pull/9065#discussion_r1244716949
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieMemoryConfig.java:
##########
@@ -140,7 +144,9 @@ private HoodieMemoryConfig() {
public static String getDefaultSpillableMapBasePath() {
String[] localDirs = FileIOUtils.getConfiguredLocalDirs();
- return (localDirs != null && localDirs.length > 0) ? localDirs[0] :
"/tmp/";
+ List<String> localDirLists = Arrays.asList(localDirs);
+ Collections.shuffle(localDirLists);
+ return !localDirLists.isEmpty() ? localDirLists.get(0) : "/tmp/";
Review Comment:
In YARN, the NodeManager (NM) monitors the availability and utilization of
local disks. In general, if a machine has 25% or more faulty disks, the NM will
be marked as unavailable. Additionally, if the utilization of a disk exceeds
90%, it will also be marked as unavailable.
- yarn.nodemanager.disk-health-checker.min-healthy-disks
```
The minimum fraction of number of disks to be healthy for the nodemanager to
launch new containers. This applies to nm-local-dirs and nm-log-dirs.
public static final String NM_MIN_HEALTHY_DISKS_FRACTION =
NM_DISK_HEALTH_CHECK_PREFIX + "min-healthy-disks";
public static final float DEFAULT_NM_MIN_HEALTHY_DISKS_FRACTION = 0.25F;
```
-
yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage
```
The maximum percentage of disk space that can be used after which a disk is
marked as offline. Values can range from 0.0 to 100.0. If the value is
greater than or equal to 100, NM will check for full disk. This applies to
nm-local-dirs and nm-log-dirs.
public static final String NM_MAX_PER_DISK_UTILIZATION_PERCENTAGE =
NM_DISK_HEALTH_CHECK_PREFIX +
"max-disk-utilization-per-disk-percentage";
public static final float DEFAULT_NM_MAX_PER_DISK_UTILIZATION_PERCENTAGE =
90.0F;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
