[jira] [Commented] (MESOS-9462) Devices in a container are inaccessible due to `nodev` on `/var/run`.
[ https://issues.apache.org/jira/browse/MESOS-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715052#comment-16715052 ] Andrei Budnik commented on MESOS-9462: -- [https://reviews.apache.org/r/69540/] > Devices in a container are inaccessible due to `nodev` on `/var/run`. > - > > Key: MESOS-9462 > URL: https://issues.apache.org/jira/browse/MESOS-9462 > Project: Mesos > Issue Type: Bug > Components: containerization >Affects Versions: 1.8.0 >Reporter: Jie Yu >Assignee: Andrei Budnik >Priority: Blocker > Labels: regression > > A recent [patch|https://reviews.apache.org/r/69086/] (commit > ede8155d1d043137e15007c48da36ac5fa0b5124) changes the behavior of how > standard device nodes (e.g., /dev/null, etc.) are setup. It uses bind mount > (from host) now (instead of mknod). > The devices nodes are created under > `/var/run/mesos/containers//devices`, and then bind mounted to > the container root filesystem. This is problematic for those Linux distros > that mount `/var/run` (or `/run`) as `nodev`. For instance, CentOS 7.4: > {noformat} > [jie@core-dev ~]$ cat /proc/self/mountinfo | grep "/run\ " > > > 24 62 0:19 / /run rw,nosuid,nodev shared:23 - tmpfs tmpfs rw,seclabel,mode=755 > [jie@core-dev ~]$ cat /etc/redhat-release > CentOS Linux release 7.4.1708 (Core) > {noformat} > As a result, the `/dev/null` devices in the container will inherit the > `nodev` from `/run` on the host > {noformat} > 629 625 0:121 > /mesos/containers/49f1da14-d741-4030-994c-0d8ed5093b13/devices/null /dev/null > rw,nosuid,nodev - tmpfs tmpfs rw,mode=755 > {noformat} > This will cause "Permission Denied" error when a process in the container > tries to open the device node. > You can try to reproduce this issue using Mesos Mini > {noformat} > docker run --rm --privileged -p 5050:5050 -p 5051:5051 -p 8080:8080 > mesos/mesos-mini:master-2018-12-06 > {noformat} > And the, go to Marathon UI (http://localhost:8080), and launch an app using > the following config > {code} > { > "id": "/test", > "cmd": "dd if=/dev/zero of=file bs=1024 count=1 oflag=dsync", > "cpus": 1, > "mem": 128, > "disk": 128, > "instances": 1, > "container": { > "type": "MESOS", > "docker": { > "image": "ubuntu:18.04" > } > } > } > {code} > You'll see the task failed with "Permission Denied". > The task will run normally if you use `mesos/mesos-mini:master-2018-12-01` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9462) Devices in a container are inaccessible due to `nodev` on `/var/run`.
[ https://issues.apache.org/jira/browse/MESOS-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714274#comment-16714274 ] Jie Yu commented on MESOS-9462: --- commit f050bf01af8f9f92bbada2c0a2025a459290ed98 (HEAD -> master, origin/master, origin/HEAD, remount_runtime_dir) Author: Jie Yu Date: Fri Dec 7 16:51:19 2018 -0800 Made sure containers runtime dir has device file access. Make sure that container's runtime dir has device file access. Some Linux distributions will mount `/run` with `nodev`, restricting accessing to device files under `/run`. However, Mesos prepares device files for containers under container's runtime dir (which is typically under `/run`) and bind mount into container root filesystems. Therefore, we need to make sure those device files can be accessed by the container. We need to do a self bind mount and remount with proper options if necessary. See MESOS-9462 for more details. Review: https://reviews.apache.org/r/69532 commit 35cdfa2c0bcb75f5801ec60671a3f978b4aa645f Author: Jie Yu Date: Sun Dec 9 19:14:09 2018 -0800 Used strings::format in os::shell. Previously, `strings::internal::format` was used. It causes issues when std::string is passed in as parameters. Switched to use `strings::format` instead in `os::shell` implementation. Review: https://reviews.apache.org/r/69537 > Devices in a container are inaccessible due to `nodev` on `/var/run`. > - > > Key: MESOS-9462 > URL: https://issues.apache.org/jira/browse/MESOS-9462 > Project: Mesos > Issue Type: Bug > Components: containerization >Affects Versions: 1.8.0 >Reporter: Jie Yu >Assignee: Jie Yu >Priority: Blocker > Labels: regression > > A recent [patch|https://reviews.apache.org/r/69086/] (commit > ede8155d1d043137e15007c48da36ac5fa0b5124) changes the behavior of how > standard device nodes (e.g., /dev/null, etc.) are setup. It uses bind mount > (from host) now (instead of mknod). > The devices nodes are created under > `/var/run/mesos/containers//devices`, and then bind mounted to > the container root filesystem. This is problematic for those Linux distros > that mount `/var/run` (or `/run`) as `nodev`. For instance, CentOS 7.4: > {noformat} > [jie@core-dev ~]$ cat /proc/self/mountinfo | grep "/run\ " > > > 24 62 0:19 / /run rw,nosuid,nodev shared:23 - tmpfs tmpfs rw,seclabel,mode=755 > [jie@core-dev ~]$ cat /etc/redhat-release > CentOS Linux release 7.4.1708 (Core) > {noformat} > As a result, the `/dev/null` devices in the container will inherit the > `nodev` from `/run` on the host > {noformat} > 629 625 0:121 > /mesos/containers/49f1da14-d741-4030-994c-0d8ed5093b13/devices/null /dev/null > rw,nosuid,nodev - tmpfs tmpfs rw,mode=755 > {noformat} > This will cause "Permission Denied" error when a process in the container > tries to open the device node. > You can try to reproduce this issue using Mesos Mini > {noformat} > docker run --rm --privileged -p 5050:5050 -p 5051:5051 -p 8080:8080 > mesos/mesos-mini:master-2018-12-06 > {noformat} > And the, go to Marathon UI (http://localhost:8080), and launch an app using > the following config > {code} > { > "id": "/test", > "cmd": "dd if=/dev/zero of=file bs=1024 count=1 oflag=dsync", > "cpus": 1, > "mem": 128, > "disk": 128, > "instances": 1, > "container": { > "type": "MESOS", > "docker": { > "image": "ubuntu:18.04" > } > } > } > {code} > You'll see the task failed with "Permission Denied". > The task will run normally if you use `mesos/mesos-mini:master-2018-12-01` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9462) Devices in a container are inaccessible due to `nodev` on `/var/run`.
[ https://issues.apache.org/jira/browse/MESOS-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713286#comment-16713286 ] Jie Yu commented on MESOS-9462: --- The fix is to make sure is a mounted w/o `nodev`. If the distro mount `/var/run` using `nodev`, during agent startup, we can perform a one time self bind mount, and a remount with `-o remount,dev`. This can ensure that device nodes created in `` can be accessed. > Devices in a container are inaccessible due to `nodev` on `/var/run`. > - > > Key: MESOS-9462 > URL: https://issues.apache.org/jira/browse/MESOS-9462 > Project: Mesos > Issue Type: Bug > Components: containerization >Affects Versions: 1.8.0 >Reporter: Jie Yu >Priority: Blocker > Labels: regression > > A recent [patch|https://reviews.apache.org/r/69086/] (commit > ede8155d1d043137e15007c48da36ac5fa0b5124) changes the behavior of how > standard device nodes (e.g., /dev/null, etc.) are setup. It uses bind mount > (from host) now (instead of mknod). > The devices nodes are created under > `/var/run/mesos/containers//devices`, and then bind mounted to > the container root filesystem. This is problematic for those Linux distros > that mount `/var/run` (or `/run`) as `nodev`. For instance, CentOS 7.4: > {noformat} > [jie@core-dev ~]$ cat /proc/self/mountinfo | grep "/run\ " > > > 24 62 0:19 / /run rw,nosuid,nodev shared:23 - tmpfs tmpfs rw,seclabel,mode=755 > [jie@core-dev ~]$ cat /etc/redhat-release > CentOS Linux release 7.4.1708 (Core) > {noformat} > As a result, the `/dev/null` devices in the container will inherit the > `nodev` from `/run` on the host > {noformat} > 629 625 0:121 > /mesos/containers/49f1da14-d741-4030-994c-0d8ed5093b13/devices/null /dev/null > rw,nosuid,nodev - tmpfs tmpfs rw,mode=755 > {noformat} > This will cause "Permission Denied" error when a process in the container > tries to open the device node. > You can try to reproduce this issue using Mesos Mini > {noformat} > docker run --rm --privileged -p 5050:5050 -p 5051:5051 -p 8080:8080 > mesos/mesos-mini:master-2018-12-06 > {noformat} > And the, go to Marathon UI (http://localhost:8080), and launch an app using > the following config > {code} > { > "id": "/test", > "cmd": "dd if=/dev/zero of=file bs=1024 count=1 oflag=dsync", > "cpus": 1, > "mem": 128, > "disk": 128, > "instances": 1, > "container": { > "type": "MESOS", > "docker": { > "image": "ubuntu:18.04" > } > } > } > {code} > You'll see the task failed with "Permission Denied". > The task will run normally if you use `mesos/mesos-mini:master-2018-12-01` -- This message was sent by Atlassian JIRA (v7.6.3#76005)