[jira] [Updated] (YARN-9914) Use separate configs for free disk space checking for full and not-full disks
[
https://issues.apache.org/jira/browse/YARN-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9914:
Fix Version/s: (was: 2.11.0)
> Use separate configs for free disk space checking for full and not-full disks
> -
>
> Key: YARN-9914
> URL: https://issues.apache.org/jira/browse/YARN-9914
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 2.9.3, 3.2.2, 3.1.4
>
> Attachments: YARN-9914-branch-2.8.001.patch, YARN-9914.001.patch,
> YARN-9914.002.patch
>
>
> [YARN-3943] added separate configurations for the nodemanager health check
> disk utilization full disk check:
> {{max-disk-utilization-per-disk-percentage}} - threshold for marking a good
> disk full
> {{disk-utilization-watermark-low-per-disk-percentage}} - threshold for
> marking a full disk as not full.
> On our clusters, we do not use these configs. We instead use
> {{min-free-space-per-disk-mb}} so we can specify the limit in mb instead of
> percent of utilization. We have observed the same oscillation behavior as
> described in [YARN-3943] with this parameter. I would like to add an optional
> config to specify a separate threshold for marking a full disk as not full:
> {{min-free-space-per-disk-mb}} - threshold at which a good disk is marked full
> {{disk-free-space-per-disk-high-watermark-mb}} - threshold at which a full
> disk is marked good.
> So for example, we could set {{min-free-space-per-disk-mb = 5GB}}, which
> would cause a disk to be marked full when free space goes below 5GB, and
> {{disk-free-space-per-disk-high-watermark-mb = 10GB}} to keep the disk in the
> full state until free space goes above 10GB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9914) Use separate configs for free disk space checking for full and not-full disks
[
https://issues.apache.org/jira/browse/YARN-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated YARN-9914:
--
Fix Version/s: 2.8.6
Thanks for the 2.8 patch, [~Jim_Brennan]! I cleaned up the checkstyle warnings
and committed this to branc-2.8
Final commit list: trunk, branch-3.2, branch-3.1, branch-3.0, branch-2,
branch-2.10, branch-2.9, branch-2.8
> Use separate configs for free disk space checking for full and not-full disks
> -
>
> Key: YARN-9914
> URL: https://issues.apache.org/jira/browse/YARN-9914
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 2.9.3, 3.2.2, 3.1.4, 2.11.0
>
> Attachments: YARN-9914-branch-2.8.001.patch, YARN-9914.001.patch,
> YARN-9914.002.patch
>
>
> [YARN-3943] added separate configurations for the nodemanager health check
> disk utilization full disk check:
> {{max-disk-utilization-per-disk-percentage}} - threshold for marking a good
> disk full
> {{disk-utilization-watermark-low-per-disk-percentage}} - threshold for
> marking a full disk as not full.
> On our clusters, we do not use these configs. We instead use
> {{min-free-space-per-disk-mb}} so we can specify the limit in mb instead of
> percent of utilization. We have observed the same oscillation behavior as
> described in [YARN-3943] with this parameter. I would like to add an optional
> config to specify a separate threshold for marking a full disk as not full:
> {{min-free-space-per-disk-mb}} - threshold at which a good disk is marked full
> {{disk-free-space-per-disk-high-watermark-mb}} - threshold at which a full
> disk is marked good.
> So for example, we could set {{min-free-space-per-disk-mb = 5GB}}, which
> would cause a disk to be marked full when free space goes below 5GB, and
> {{disk-free-space-per-disk-high-watermark-mb = 10GB}} to keep the disk in the
> full state until free space goes above 10GB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9914) Use separate configs for free disk space checking for full and not-full disks
[
https://issues.apache.org/jira/browse/YARN-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan updated YARN-9914:
--
Attachment: YARN-9914-branch-2.8.001.patch
> Use separate configs for free disk space checking for full and not-full disks
> -
>
> Key: YARN-9914
> URL: https://issues.apache.org/jira/browse/YARN-9914
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Fix For: 2.10.0, 3.0.4, 3.3.0, 2.9.3, 3.2.2, 3.1.4, 2.11.0
>
> Attachments: YARN-9914-branch-2.8.001.patch, YARN-9914.001.patch,
> YARN-9914.002.patch
>
>
> [YARN-3943] added separate configurations for the nodemanager health check
> disk utilization full disk check:
> {{max-disk-utilization-per-disk-percentage}} - threshold for marking a good
> disk full
> {{disk-utilization-watermark-low-per-disk-percentage}} - threshold for
> marking a full disk as not full.
> On our clusters, we do not use these configs. We instead use
> {{min-free-space-per-disk-mb}} so we can specify the limit in mb instead of
> percent of utilization. We have observed the same oscillation behavior as
> described in [YARN-3943] with this parameter. I would like to add an optional
> config to specify a separate threshold for marking a full disk as not full:
> {{min-free-space-per-disk-mb}} - threshold at which a good disk is marked full
> {{disk-free-space-per-disk-high-watermark-mb}} - threshold at which a full
> disk is marked good.
> So for example, we could set {{min-free-space-per-disk-mb = 5GB}}, which
> would cause a disk to be marked full when free space goes below 5GB, and
> {{disk-free-space-per-disk-high-watermark-mb = 10GB}} to keep the disk in the
> full state until free space goes above 10GB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9914) Use separate configs for free disk space checking for full and not-full disks
[
https://issues.apache.org/jira/browse/YARN-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated YARN-9914:
--
Fix Version/s: 2.11.0
3.1.4
3.2.2
2.9.3
3.3.0
3.0.4
2.10.0
Thanks for the patch, [~Jim_Brennan]! I cleaned up the small checkstyle issues
and committed this to trunk, branch-3.2, branch-3.1, branch-3.0, branch-2,
branch-2.10, and branch-2.9. There was a small conflict with branch-2.8. If
you'd like it to go back that far, please put up a new patch for that branch.
Otherwise, feel free to close as resolved.
> Use separate configs for free disk space checking for full and not-full disks
> -
>
> Key: YARN-9914
> URL: https://issues.apache.org/jira/browse/YARN-9914
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Fix For: 2.10.0, 3.0.4, 3.3.0, 2.9.3, 3.2.2, 3.1.4, 2.11.0
>
> Attachments: YARN-9914.001.patch, YARN-9914.002.patch
>
>
> [YARN-3943] added separate configurations for the nodemanager health check
> disk utilization full disk check:
> {{max-disk-utilization-per-disk-percentage}} - threshold for marking a good
> disk full
> {{disk-utilization-watermark-low-per-disk-percentage}} - threshold for
> marking a full disk as not full.
> On our clusters, we do not use these configs. We instead use
> {{min-free-space-per-disk-mb}} so we can specify the limit in mb instead of
> percent of utilization. We have observed the same oscillation behavior as
> described in [YARN-3943] with this parameter. I would like to add an optional
> config to specify a separate threshold for marking a full disk as not full:
> {{min-free-space-per-disk-mb}} - threshold at which a good disk is marked full
> {{disk-free-space-per-disk-high-watermark-mb}} - threshold at which a full
> disk is marked good.
> So for example, we could set {{min-free-space-per-disk-mb = 5GB}}, which
> would cause a disk to be marked full when free space goes below 5GB, and
> {{disk-free-space-per-disk-high-watermark-mb = 10GB}} to keep the disk in the
> full state until free space goes above 10GB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9914) Use separate configs for free disk space checking for full and not-full disks
[
https://issues.apache.org/jira/browse/YARN-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan updated YARN-9914:
--
Attachment: YARN-9914.002.patch
> Use separate configs for free disk space checking for full and not-full disks
> -
>
> Key: YARN-9914
> URL: https://issues.apache.org/jira/browse/YARN-9914
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Attachments: YARN-9914.001.patch, YARN-9914.002.patch
>
>
> [YARN-3943] added separate configurations for the nodemanager health check
> disk utilization full disk check:
> {{max-disk-utilization-per-disk-percentage}} - threshold for marking a good
> disk full
> {{disk-utilization-watermark-low-per-disk-percentage}} - threshold for
> marking a full disk as not full.
> On our clusters, we do not use these configs. We instead use
> {{min-free-space-per-disk-mb}} so we can specify the limit in mb instead of
> percent of utilization. We have observed the same oscillation behavior as
> described in [YARN-3943] with this parameter. I would like to add an optional
> config to specify a separate threshold for marking a full disk as not full:
> {{min-free-space-per-disk-mb}} - threshold at which a good disk is marked full
> {{disk-free-space-per-disk-high-watermark-mb}} - threshold at which a full
> disk is marked good.
> So for example, we could set {{min-free-space-per-disk-mb = 5GB}}, which
> would cause a disk to be marked full when free space goes below 5GB, and
> {{disk-free-space-per-disk-high-watermark-mb = 10GB}} to keep the disk in the
> full state until free space goes above 10GB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9914) Use separate configs for free disk space checking for full and not-full disks
[
https://issues.apache.org/jira/browse/YARN-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan updated YARN-9914:
--
Attachment: YARN-9914.001.patch
> Use separate configs for free disk space checking for full and not-full disks
> -
>
> Key: YARN-9914
> URL: https://issues.apache.org/jira/browse/YARN-9914
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Attachments: YARN-9914.001.patch
>
>
> [YARN-3943] added separate configurations for the nodemanager health check
> disk utilization full disk check:
> {{max-disk-utilization-per-disk-percentage}} - threshold for marking a good
> disk full
> {{disk-utilization-watermark-low-per-disk-percentage}} - threshold for
> marking a full disk as not full.
> On our clusters, we do not use these configs. We instead use
> {{min-free-space-per-disk-mb}} so we can specify the limit in mb instead of
> percent of utilization. We have observed the same oscillation behavior as
> described in [YARN-3943] with this parameter. I would like to add an optional
> config to specify a separate threshold for marking a full disk as not full:
> {{min-free-space-per-disk-mb}} - threshold at which a good disk is marked full
> {{disk-free-space-per-disk-high-watermark-mb}} - threshold at which a full
> disk is marked good.
> So for example, we could set {{min-free-space-per-disk-mb = 5GB}}, which
> would cause a disk to be marked full when free space goes below 5GB, and
> {{disk-free-space-per-disk-high-watermark-mb = 10GB}} to keep the disk in the
> full state until free space goes above 10GB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
