Wangda Tan created YARN-7739:
--------------------------------
Summary: Revisit scheduler resource normalization behavior for max
allocation
Key: YARN-7739
URL: https://issues.apache.org/jira/browse/YARN-7739
Project: Hadoop YARN
Issue Type: Bug
Reporter: Wangda Tan
Priority: Critical
Currently, YARN Scheduler normalizes requested resource based on the maximum
allocation derived from configured maximum allocation and maximum registered
node resources. Basically, the scheduler will silently cap asked resource by
maximum allocation.
This could cause issues for applications, for example, a Spark job which needs
12 GB memory to run, however in the cluster, registered NMs have at most 8 GB
mem on each node. So scheduler allocates 8GB memory container to the requested
application.
Once app receives containers from RM, if it doesn't double check allocated
resources, it will lead to OOM and hard to debug because scheduler silently
caps maximum allocation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
