[
https://issues.apache.org/jira/browse/YARN-11572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773267#comment-17773267
]
Jean-Baptiste Guet commented on YARN-11572:
-------------------------------------------
I have managed to solve the issue, which was in fact caused by another service.
The culprit was the Prometheus exporter (set up as a service) provided at
[https://github.com/NVIDIA/dcgm-exporter]. It was unexpected since its primary
function is to collect metrics, but it was indeed causing the problem. Notably,
starting, stopping, or restarting this service had no effect on the cgroups
directory, so I decided to keep it disabled.
I am thus resolving the issue.
> hadoop-yarn cgroup directory is deleted after each "systemctl daemon-reload"
> command
> ------------------------------------------------------------------------------------
>
> Key: YARN-11572
> URL: https://issues.apache.org/jira/browse/YARN-11572
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 3.3.4
> Environment:
>
> Reporter: Jean-Baptiste Guet
> Priority: Major
>
> I have an Hadoop cluster and I need to activate cgroups in order to use GPU
> in docker environment. I followed the documentation for the setup.
>
> {*}To summarize{*}: I do manage myself the cgroups creation (cpu, cpuacct and
> devices), which results as expected on the creation of 3 directories in
> {_}{{/sys/fs/cgroup/}}{_}. However, upon each {_}systemctl daemon-reload{_},
> the _/sys/fs/cgroup-hadoop-yarn_ directory is systematically deleted, which
> prevents Yarn's nodemanager from working.
>
> {*}In details{*}:
> As it's written in the documentation, I kept the parameter
> _yarn.nodemanager.linux-container-executor.cgroups.mount_ to _false_ in order
> manage the cgroup myself (security reason).
> As I'm on CentOS 8, I use cgroup v1. I defined the parameters :
> * _yarn.nodemanager.linux-container-executor.cgroups.hierarchy_ to
> _/hadoop-yarn_
> * _yarn.nodemanager.linux-container-executor.cgroups.mount-path_ to
> _/sys/fs/cgroup_
> Yarn needs 3 cgroups : cpu, cpuacct and devices.
> In order to have the /haddop-yarn persistent, I've install libcgroup rpm then
> I've updated /etc/cgconfig.conf with
> {code:java}
> group hadoop-yarn {
> perm {
> admin {
> uid = yarn;
> gid = hadoop;
> }
> task {
> uid = yarn;
> gid = hadoop;
> }
> }
> cpu {}
> cpuacct {}
> devices {}
> }
> {code}
> and I've started cgconfig service. The 3 directories are created :
> {code:java}
> $ ll /sys/fs/cgroup/{cpu,cpuacct,devices}/hadoop-yarn/ -d
> drwxr-xr-x 2 yarn hadoop 0 Sep 8 13:15 /sys/fs/cgroup/cpuacct/hadoop-yarn/
> {{drwxr-xr-x 2 yarn hadoop 0 Sep 8 13:15 /sys/fs/cgroup/cpu/hadoop-yarn/ }}
> {{drwxr-xr-x 2 yarn hadoop 0 Sep 8 13:27 /sys/fs/cgroup/devices/hadoop-yarn/
> }}
> {code}
>
> At this point, I can restart the Yarn NodeManager.
> However, each time that someone execute {{{}systemctl daemon-reload{}}}, the
> devices directory is deleted :
> {code:java}
> $ ll /sys/fs/cgroup/{cpu,cpuacct,devices}/hadoop-yarn/ -d
> {{ls: cannot access '/sys/fs/cgroup/devices/hadoop-yarn/': No such file or
> directory }}
> {{drwxr-xr-x 2 yarn hadoop 0 Sep 8 13:15 /sys/fs/cgroup/cpuacct/hadoop-yarn/
> }}
> drwxr-xr-x 2 yarn hadoop 0 Sep 8 13:15 /sys/fs/cgroup/cpu/hadoop-yarn/{code}
>
> I see nothing in logs, I have no idea why this directory is deleted. And of
> course, Yarn NodeManager needs this directory, so the NodeManager doesn't
> work anymore and needs to be restarted (once the directory has been
> re-created of course).
> As an other solution of cgconfig service, I've tested to create my own
> service that will create these directories.
> {code:java}
> vim /etc/systemd/system/hadoop-yarn-cgroup.service
> [Unit]
> Description=Custom cgroup for Hadoop YARN
> [Service]
> ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpu/hadoop-yarn
> ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpuacct/hadoop-yarn
> ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/devices/hadoop-yarn
> ExecStartPre=/usr/bin/chown -R yarn:hadoop /sys/fs/cgroup/cpu/hadoop-yarn/
> ExecStartPre=/usr/bin/chown -R yarn:hadoop /sys/fs/cgroup/cpuacct/hadoop-yarn/
> ExecStartPre=/usr/bin/chown -R yarn:hadoop /sys/fs/cgroup/devices/hadoop-yarn/
> ExecStart=/bin/true
> Slice=hadoop-yarn.slice
> MemoryAccounting=yes
> MemoryLimit=1G
> [Install]
> WantedBy=multi-user.target{code}
>
> The behaviour is the same :
> * directories are created
> * systemctl daemon-reload
> * devices/hadoop-yarn directory is deleted
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
