[
https://issues.apache.org/jira/browse/YARN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816512#comment-15816512
]
Devaraj K commented on YARN-5764:
---------------------------------
Thanks a lot [~leftnoteasy] for review and comments.
bq. What is the benefit to manually specify NUMA node? Since this is
potentially complex for end user to specify, I think it's better to directly
read data from OS.
If the users want to share the NUMA resources in Node Manager machine for
non-Yarn applications, then users can specify what all numa nodes and each node
capabilities can be used by Yarn using this declaration. I understand there are
configurations for specifying numa nodes, each node memory and cpu's. But if we
don't have provision for separating the NUMA resources for Yarn, we could end
up overlapping the resources used by Yarn and Non-Yarn applications.
bq. Does the changes work on platform other than Linux?
This patch works for Linux, if this approach is agreeable then I will update
for windows as well.
bq. I'm not quite sure about if this could happen: with this patch, YARN will
launch process one by one on each NUMA node to bind memory/cpu. Is it possible
that there's another process (outside of YARN) uses memory of NUMA node which
causes processes launched by YARN failed to bind or run?
I do think it could happen for memory, we can avoid this using the NUMA node
topology declaration for specifying the NUMA resources for Yarn applications.
And also it would not be an issue with the soft binding option which you
mentioned in the below comment.
bq. This patch uses hard binding (get allocated resource on specified node or
fail), is it better to specify soft binding (prefer to allocate and can also
accept other node). I think soft binding should be default behavior to support
NUMA.
I think it is a good suggestion, I can update the patch with this by changing
'\--membind=nodes' to '\--preferred=node'.
I will look forward for your further comments.
> NUMA awareness support for launching containers
> -----------------------------------------------
>
> Key: YARN-5764
> URL: https://issues.apache.org/jira/browse/YARN-5764
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, yarn
> Reporter: Olasoji
> Assignee: Devaraj K
> Attachments: NUMA Awareness for YARN Containers.pdf,
> YARN-5764-v0.patch, YARN-5764-v1.patch
>
>
> The purpose of this feature is to improve Hadoop performance by minimizing
> costly remote memory accesses on non SMP systems. Yarn containers, on launch,
> will be pinned to a specific NUMA node and all subsequent memory allocations
> will be served by the same node, reducing remote memory accesses. The current
> default behavior is to spread memory across all NUMA nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
