Jean-Baptiste Guet created YARN-11572:
-----------------------------------------
Summary: hadoop-yarn cgroup directory is deleted after each
"systemctl daemon-reload" command
Key: YARN-11572
URL: https://issues.apache.org/jira/browse/YARN-11572
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.3.4
Environment:
Reporter: Jean-Baptiste Guet
I have an Hadoop cluster and I need to activate cgroups in order to use GPU in
docker environment. I followed the documentation for the setup.
{*}To summarize{*}: I do manage myself the cgroups creation (cpu, cpuacct and
devices), which results as expected on the creation of 3 directories in
{_}{{/sys/fs/cgroup/}}{_}. However, upon each {_}systemctl daemon-reload{_},
the _/sys/fs/cgroup-hadoop-yarn_ directory is systematically deleted, which
prevents Yarn's nodemanager from working.
{*}In details{*}:
As it's written in the documentation, I kept the parameter
_yarn.nodemanager.linux-container-executor.cgroups.mount_ to _false_ in order
manage the cgroup myself (security reason).
As I'm on CentOS 8, I use cgroup v1. I defined the parameters :
* _yarn.nodemanager.linux-container-executor.cgroups.hierarchy_ to
_/hadoop-yarn_
* _yarn.nodemanager.linux-container-executor.cgroups.mount-path_ to
_/sys/fs/cgroup_
Yarn needs 3 cgroups : cpu, cpuacct and devices.
In order to have the /haddop-yarn persistent, I've install libcgroup rpm then
I've updated /etc/cgconfig.conf with
{code:java}
group hadoop-yarn {
perm {
admin {
uid = yarn;
gid = hadoop;
}
task {
uid = yarn;
gid = hadoop;
}
}
cpu {}
cpuacct {}
devices {}
}
{code}
{{}}
and I've started cgconfig service. The 3 directories are created :
{{}}
{code:java}
{code}
{{$ ll /sys/fs/cgroup/\{cpu,cpuacct,devices}/hadoop-yarn/ -d}}
{{drwxr-xr-x 2 yarn hadoop 0 Sep 8 13:15 /sys/fs/cgroup/cpuacct/hadoop-yarn/}}
{{drwxr-xr-x 2 yarn hadoop 0 Sep 8 13:15 /sys/fs/cgroup/cpu/hadoop-yarn/ }}
{{drwxr-xr-x 2 yarn hadoop 0 Sep 8 13:27 /sys/fs/cgroup/devices/hadoop-yarn/ }}
At this point, I can restart the Yarn NodeManager.
However, each time that someone execute {{{}systemctl daemon-reload{}}}, the
devices directory is deleted :
{{}}
{code:java}
{code}
{{$ ll /sys/fs/cgroup/\{cpu,cpuacct,devices}/hadoop-yarn/ -d}}
{{ls: cannot access '/sys/fs/cgroup/devices/hadoop-yarn/': No such file or
directory }}
{{drwxr-xr-x 2 yarn hadoop 0 Sep 8 13:15 /sys/fs/cgroup/cpuacct/hadoop-yarn/ }}
{{drwxr-xr-x 2 yarn hadoop 0 Sep 8 13:15 /sys/fs/cgroup/cpu/hadoop-yarn/}}
{{ }}
I see nothing in logs, I have no idea why this directory is deleted. And of
course, Yarn NodeManager needs this directory, so the NodeManager doesn't work
anymore and needs to be restarted (once the directory has been re-created of
course).
As an other solution of cgconfig service, I've tested to create my own service
that will create these directories.
{{}}
{code:java}
{code}
{{vim /etc/systemd/system/hadoop-yarn-cgroup.service}}
{{[Unit]}}
{{Description=Custom cgroup for Hadoop YARN}}
{{[Service]}}
{{ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpu/hadoop-yarn}}
{{ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpuacct/hadoop-yarn}}
{{ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/devices/hadoop-yarn}}
{{ExecStartPre=/usr/bin/chown -R yarn:hadoop /sys/fs/cgroup/cpu/hadoop-yarn/}}
{{ExecStartPre=/usr/bin/chown -R yarn:hadoop
/sys/fs/cgroup/cpuacct/hadoop-yarn/}}
{{ExecStartPre=/usr/bin/chown -R yarn:hadoop
/sys/fs/cgroup/devices/hadoop-yarn/}}
{{ExecStart=/bin/true}}
{{Slice=hadoop-yarn.slice}}
{{MemoryAccounting=yes}}
{{MemoryLimit=1G}}
{{[Install]}}
{{WantedBy=multi-user.target }}
The behaviour is the same :
* directories are created
* systemctl daemon-reload
* devices/hadoop-yarn directory is deleted
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
