Jason Lowe commented on YARN-3943:

Thanks for the patch, [~zxu]!

I think this patch causes backwards compatibility problems for users who have 
increased the max-disk-utilization-per-disk-percentage from the default.  For 
example, if someone changed it to 100 and only wants disks to be treated full 
if they actually fill up completely then this patch will silently make those 
disks unusable once they fill until they free up at least 10% of their space.  
That's not going to be good if the cluster runs the disks over 90% most of the 
time, as it will effectively take the disk out of rotation for a long, long 
time.  A more compatible change would be to _not_ set a value in yarn-default 
for this property.  If the property is set then we use it, but if we get a null 
for it then we know it isn't specified and we shouldn't just use the same 
threshold for low as for high.

Shouldn't we be capping the max of diskUtilizationPercentageCutoffLow based on 
diskUtilizationPercentageCutoffHigh rather than 100.0?

Nit: I think using Math.max / Math.min could make the range capping more 
readable (certainly less redundant with the identifiers and constants), but 
it's not must-fix.

Nit: "Use" should be "Using" in the following log message.  "Use" implies we 
are giving a directive requiring the user to do it, while "Using" states the 
code is doing it for them.
+      if (lowUsableSpacePercentagePerDisk > highUsableSpacePercentagePerDisk) {
+        LOG.warn("Use " + YarnConfiguration.
+            YarnConfiguration.NM_WM_LOW_PER_DISK_UTILIZATION_PERCENTAGE +
+            ", because " + YarnConfiguration.
+            " is not configured properly.");

> Use separate threshold configurations for disk-full detection and 
> disk-not-full detection.
> ------------------------------------------------------------------------------------------
>                 Key: YARN-3943
>                 URL: https://issues.apache.org/jira/browse/YARN-3943
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>            Priority: Critical
>         Attachments: YARN-3943.000.patch, YARN-3943.001.patch
> Use separate threshold configurations to check when disks become full and 
> when disks become good. Currently the configuration 
> "yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage"
>  and "yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb" are 
> used to check both when disks become full and when disks become good. It will 
> be better to use two configurations: one is used when disks become full from 
> not-full and the other one is used when disks become not-full from full. So 
> we can avoid oscillating frequently.
> For example: we can set the one for disk-full detection higher than the one 
> for disk-not-full detection.

This message was sent by Atlassian JIRA

Reply via email to