[
https://issues.apache.org/jira/browse/TEZ-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4750:
------------------------------
Description:
TLDR: same as TEZ-4366, just in case of tez_run_example.sh
Running dev-support/bin/tez_run_example.sh on a developer laptop might fail
silently: the DAG hangs in ACCEPTED forever with the misleading RM diagnostic.
{code}
Skipping AM assignment as cluster resource is empty. Queue Resource Limit for
AM = <memory:0, vCores:0>
{code}
The NodeManager is actually running and registered — but marked UNHEALTHY
because YARN's default disk-health-checker trips at 90% disk utilization, and
both nm-local-dir (/tmp/hadoop-$USER/nm-local-dir) and logs/userlogs sit on the
user's root volume, which is commonly >90% full on a working laptop.
This is genuinely painful to diagnose:
- The user-facing symptom points at the scheduler / queue configuration, not
the disk.
- yarn node -list alone shows the node exists — you have to know to run yarn
node -status <id> to see the health report.
- The NM logs happily say Registered with ResourceManager … <memory:8192,
vCores:8> seconds before it gets flipped to UNHEALTHY, which reinforces the
wrong hypothesis.
- Everything else (SSH, JAVA_HOME, tarball extraction) can be perfectly correct
and you still get zero throughput.
Looking at the unhealthy nodemanager web interface, I found the root cause:
{code}
1/1 local-dirs usable space is below configured utilization percentage/no more
usable space [ /tmp/hadoop-lbodor/nm-local-dir : used space above threshold of
90.0% ] ; 1/1 log-dirs usable space is below configured utilization
percentage/no more usable space [
/Users/lbodor/apache/tez/hadoop-3.4.1/logs/userlogs : used space above
threshold of 90.0% ]
{code}
was:
Running dev-support/bin/tez_run_example.sh on a developer laptop might fail
silently: the DAG hangs in ACCEPTED forever with the misleading RM diagnostic.
{code}
Skipping AM assignment as cluster resource is empty. Queue Resource Limit for
AM = <memory:0, vCores:0>
{code}
The NodeManager is actually running and registered — but marked UNHEALTHY
because YARN's default disk-health-checker trips at 90% disk utilization, and
both nm-local-dir (/tmp/hadoop-$USER/nm-local-dir) and logs/userlogs sit on the
user's root volume, which is commonly >90% full on a working laptop.
This is genuinely painful to diagnose:
- The user-facing symptom points at the scheduler / queue configuration, not
the disk.
- yarn node -list alone shows the node exists — you have to know to run yarn
node -status <id> to see the health report.
- The NM logs happily say Registered with ResourceManager … <memory:8192,
vCores:8> seconds before it gets flipped to UNHEALTHY, which reinforces the
wrong hypothesis.
- Everything else (SSH, JAVA_HOME, tarball extraction) can be perfectly correct
and you still get zero throughput.
Looking at the unhealthy nodemanager web interface, I found the root cause:
{code}
1/1 local-dirs usable space is below configured utilization percentage/no more
usable space [ /tmp/hadoop-lbodor/nm-local-dir : used space above threshold of
90.0% ] ; 1/1 log-dirs usable space is below configured utilization
percentage/no more usable space [
/Users/lbodor/apache/tez/hadoop-3.4.1/logs/userlogs : used space above
threshold of 90.0% ]
{code}
> Relax yarn nodemanager health checker in tez_run_example.sh
> -----------------------------------------------------------
>
> Key: TEZ-4750
> URL: https://issues.apache.org/jira/browse/TEZ-4750
> Project: Apache Tez
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> TLDR: same as TEZ-4366, just in case of tez_run_example.sh
> Running dev-support/bin/tez_run_example.sh on a developer laptop might fail
> silently: the DAG hangs in ACCEPTED forever with the misleading RM diagnostic.
> {code}
> Skipping AM assignment as cluster resource is empty. Queue Resource Limit for
> AM = <memory:0, vCores:0>
> {code}
> The NodeManager is actually running and registered — but marked UNHEALTHY
> because YARN's default disk-health-checker trips at 90% disk utilization, and
> both nm-local-dir (/tmp/hadoop-$USER/nm-local-dir) and logs/userlogs sit on
> the user's root volume, which is commonly >90% full on a working laptop.
> This is genuinely painful to diagnose:
> - The user-facing symptom points at the scheduler / queue configuration, not
> the disk.
> - yarn node -list alone shows the node exists — you have to know to run yarn
> node -status <id> to see the health report.
> - The NM logs happily say Registered with ResourceManager … <memory:8192,
> vCores:8> seconds before it gets flipped to UNHEALTHY, which reinforces the
> wrong hypothesis.
> - Everything else (SSH, JAVA_HOME, tarball extraction) can be perfectly
> correct and you still get zero throughput.
> Looking at the unhealthy nodemanager web interface, I found the root cause:
> {code}
> 1/1 local-dirs usable space is below configured utilization percentage/no
> more usable space [ /tmp/hadoop-lbodor/nm-local-dir : used space above
> threshold of 90.0% ] ; 1/1 log-dirs usable space is below configured
> utilization percentage/no more usable space [
> /Users/lbodor/apache/tez/hadoop-3.4.1/logs/userlogs : used space above
> threshold of 90.0% ]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
