[
https://issues.apache.org/jira/browse/YARN-7739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324511#comment-16324511
]
Jason Lowe commented on YARN-7739:
----------------------------------
I'm OK with getting rid of implicitly shrinking containers since I think it is
causing more problems than it is solving, but it is technically an incompatible
change. If there was some app out there that was relying on asking for a
9999GB container and then expecting it to be silently truncated to the
cluster/queue's max allocation then it will break after that change.
>From a compatibility standpoint we could add a flag to the request indicating
>whether the request is OK with silent truncation, but most apps would have to
>be updated to set the flag saying it's _not_ OK to truncate to get the
>behavior most apps would want which is no truncation. Not a fan of that
>either. IMHO truncation is a bug, so maybe it's not so much an incompatible
>change as it is a bugfix. ;-)
I would recommend pinging [~vinodkv] about this since I believe he was involved
in the early days discussions that led to the silent truncation to max behavior.
> Revisit scheduler resource normalization behavior for max allocation
> --------------------------------------------------------------------
>
> Key: YARN-7739
> URL: https://issues.apache.org/jira/browse/YARN-7739
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Priority: Critical
>
> Currently, YARN Scheduler normalizes requested resource based on the maximum
> allocation derived from configured maximum allocation and maximum registered
> node resources. Basically, the scheduler will silently cap asked resource by
> maximum allocation.
> This could cause issues for applications, for example, a Spark job which
> needs 12 GB memory to run, however in the cluster, registered NMs have at
> most 8 GB mem on each node. So scheduler allocates 8GB memory container to
> the requested application.
> Once app receives containers from RM, if it doesn't double check allocated
> resources, it will lead to OOM and hard to debug because scheduler silently
> caps maximum allocation.
> When non-mandatory resources introduced, this becomes worse. For resources
> like GPU, we typically set minimum allocation to 0 since not all nodes have
> GPU devices. So it is possible that application asks 4 GPUs but get 0 GPU, it
> gonna be a big problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
