This is an automated email from the ASF dual-hosted git repository.
prabhujoseph pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 7eefdf86422 YARN-11195. Adding document to enable numa (#4501)
7eefdf86422 is described below
commit 7eefdf86422a15833fc3dbe69da10e7f6b0b4794
Author: Samrat <[email protected]>
AuthorDate: Tue Jun 28 17:46:43 2022 +0530
YARN-11195. Adding document to enable numa (#4501)
Contributed by Samrat Deb.
---
.../src/site/markdown/UsingNuma.md | 206 +++++++++++++++++++++
1 file changed, 206 insertions(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/UsingNuma.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/UsingNuma.md
new file mode 100644
index 00000000000..c93469cae3f
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/UsingNuma.md
@@ -0,0 +1,206 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# NUMA
+
+Non-uniform memory access (NUMA) is a computer memory design used in
multiprocessing,
+where the memory access time depends on the memory location relative to the
processor.
+Under NUMA, a processor can access its own local memory faster than non-local
memory
+(memory local to another processor or memory shared between processors).
+Yarn Containers can make benefit of this NUMA design to get better performance
by binding to a
+specific NUMA node and all subsequent memory allocations will be served by the
same node,
+reducing remote memory accesses. NUMA support for YARN Container has to be
enabled only if worker
+node machines has NUMA support.
+
+# Enabling NUMA
+
+### Prerequisites
+
+- As of now, NUMA awareness works only with `LinuxContainerExecutor` (LCE)
+- To use the feature of NUMA awareness in the cluster,It must be enabled with
+ LinuxContainerExecutor (LCE)
+- Steps to enable SecureContainer (LCE) for cluster is documented
[here](SecureContainer.md)
+
+## Configurations
+
+**1) Enable/Disable the NUMA awareness**
+
+This property enables the NUMA awareness feature in the Node Manager
+for the containers. By default, the value of this property is false which
means it is disabled.
+If this property is `true` then only the below configurations will be
applicable otherwise they
+will be ignored.
+
+In `yarn-site.xml` add
+
+```
+ <property>
+ <name>yarn.nodemanager.numa-awareness.enabled</name>
+ <value>true</value>
+ </property>
+```
+
+**2) NUMA topology**
+
+This property decides whether to read the NUMA topology from the system or
from the
+configurations. If this property value is true then the topology will be read
from the system using
+`numactl --hardware` command in UNIX systems and similar way in windows.
+If this property is false then the topology will be read using the below
configurations.
+Default value of this configuration is false which means NodeManager will read
the NUMA topology
+from the below configurations.
+
+In `yarn-site.xml` add
+
+```
+ <property>
+ <name>yarn.nodemanager.numa-awareness.read-topology</name>
+ <value>false</value>
+ </property>
+```
+
+**3) Numa command**
+
+This property is passed when `yarn.nodemanager.numa-awareness.read-topology`
is set to true.
+It is recommended to verify the installation of `numactl` command in the Linux
OS of every node.
+
+Use `/usr/bin/numactl --hardware` to verify.
+Sample output of `/usr/bin/numactl --hardware`
+
+```
+available: 2 nodes (0-1)
+node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 48
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
+node 0 size: 191297 MB
+node 0 free: 186539 MB
+node 1 cpus: 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
46 47 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
+node 1 size: 191383 MB
+node 1 free: 185914 MB
+node distances:
+node 0 1
+ 0: 10 21
+ 1: 21 10
+```
+
+In `yarn-site.xml` add
+
+```
+ <property>
+ <name>yarn.nodemanager.numa-awareness.numactl.cmd</name>
+ <value>/usr/bin/numactl</value>
+ </property>
+```
+
+**4) NUMA nodes id’s**
+
+This property is used to provide the NUMA node ids as comma separated
values.It will be read only
+when the `yarn.nodemanager.numa-awareness.read-topology` is false.
+
+In ```yarn-site.xml``` add
+
+```
+ <property>
+ <name>yarn.nodemanager.numa-awareness.node-ids</name>
+ <value>0,1</value>
+ </property>
+```
+
+**5) NUMA Node memory**
+
+This property will be used to read the memory(in MB) configured for each NUMA
node specified in
+`yarn.nodemanager.numa-awareness.node-ids` by substituting the node id in the
place of
+`<NODE_ID>`.It will be read only when the
`yarn.nodemanager.numa-awareness.read-topology`
+is false.
+
+In `yarn-site.xml` add
+
+```
+ <property>
+ <name>yarn.nodemanager.numa-awareness.<NODE_ID>.memory</name>
+ <value>191297</value>
+ </property>
+```
+
+The value passed is the per node memory available , from the above sample
output of
+`numactl --hardware` the value passed for the property is the memory available
i.e `191297`
+
+**6) NUMA Node CPUs**
+
+This property will be used to read the number of CPUs configured for each node
specified in
+`yarn.nodemanager.numa-awareness.node-ids` by substituting the node id in the
place of
+`<NODE_ID>`.It will be read only when the
`yarn.nodemanager.numa-awareness.read-topology` is false.
+
+In ```yarn-site.xml``` add
+
+```
+ <property>
+ <name>yarn.nodemanager.numa-awareness.<NODE_ID>.cpus</name>
+ <value>48</value>
+ </property>
+```
+
+referring to the `numactl --hardware` output , number of cpu's in a node is
`48`.
+
+**7) Passing java_opts for map/reduce**
+
+Every container has to be aware of NUMA and the JVM can be notified via
passing NUMA flag.
+Spark, Tez and other YARN Applications also need to set the container JVM Opts
to leverage
+NUMA Support.
+
+In ```mapred-site.xml``` add
+
+```
+ <property>
+ <name>mapreduce.reduce.java.opts</name>
+ <value>-XX:+UseNUMA</value>
+ </property>
+ <property>
+ <name>mapreduce.map.java.opts</name>
+ <value>-XX:+UseNUMA</value>
+ </property>
+```
+
+# Default configuration
+
+| Property | Default value |
+| --- |-----|
+|yarn.nodemanager.numa-awareness.enabled|false|
+|yarn.nodemanager.numa-awareness.read-topology|false|
+
+# Enable numa balancing at OS Level (Optional)
+
+In linux, by default numa balancing is by default off. For more performance
improvement,
+NumaBalancing can be turned on for all the nodes in cluster
+
+```
+echo 1 | sudo tee /proc/sys/kernel/numa_balancing
+```
+
+# Verify
+
+**1) NodeManager log**
+
+In any of the NodeManager, grep log file using below command
+
+`grep "NUMA resources allocation is enabled," *`
+
+Sample log with `LinuxContainerExecutor` enabled message
+
+```
+<nodemanager_ip>.log.2022-06-24-19.gz:2022-06-24 19:16:40,178 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.numa.NumaResourceHandlerImpl
(main): NUMA resources allocation is enabled, initializing NUMA resources
allocator.
+```
+
+**2) Container Log**
+
+Grep the NodeManager log using below grep command to check if a container is
assigned with NUMA node
+resources.
+
+`grep "NUMA node" | grep <container_id>`
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
