[jira] [Commented] (MESOS-9462) Devices in a container are inaccessible due to `nodev` on `/var/run`.

2018-12-10 Thread Andrei Budnik (JIRA)


[ 
https://issues.apache.org/jira/browse/MESOS-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715052#comment-16715052
 ] 

Andrei Budnik commented on MESOS-9462:
--

[https://reviews.apache.org/r/69540/]

> Devices in a container are inaccessible due to `nodev` on `/var/run`.
> -
>
> Key: MESOS-9462
> URL: https://issues.apache.org/jira/browse/MESOS-9462
> Project: Mesos
>  Issue Type: Bug
>  Components: containerization
>Affects Versions: 1.8.0
>Reporter: Jie Yu
>Assignee: Andrei Budnik
>Priority: Blocker
>  Labels: regression
>
> A recent [patch|https://reviews.apache.org/r/69086/] (commit 
> ede8155d1d043137e15007c48da36ac5fa0b5124) changes the behavior of how 
> standard device nodes (e.g., /dev/null, etc.) are setup. It uses bind mount 
> (from host) now (instead of mknod).
> The devices nodes are created under 
> `/var/run/mesos/containers//devices`, and then bind mounted to 
> the container root filesystem. This is problematic for those Linux distros 
> that mount `/var/run` (or `/run`) as `nodev`. For instance, CentOS 7.4:
> {noformat}
> [jie@core-dev ~]$ cat /proc/self/mountinfo | grep "/run\ "
>   
>
> 24 62 0:19 / /run rw,nosuid,nodev shared:23 - tmpfs tmpfs rw,seclabel,mode=755
> [jie@core-dev ~]$ cat /etc/redhat-release 
> CentOS Linux release 7.4.1708 (Core) 
> {noformat}
> As a result, the `/dev/null` devices in the container will inherit the 
> `nodev` from `/run` on the host
> {noformat}
> 629 625 0:121 
> /mesos/containers/49f1da14-d741-4030-994c-0d8ed5093b13/devices/null /dev/null 
> rw,nosuid,nodev - tmpfs tmpfs rw,mode=755
> {noformat}
> This will cause "Permission Denied" error when a process in the container 
> tries to open the device node.
> You can try to reproduce this issue using Mesos Mini
> {noformat}
> docker run --rm --privileged -p 5050:5050 -p 5051:5051 -p 8080:8080 
> mesos/mesos-mini:master-2018-12-06
> {noformat}
> And the, go to Marathon UI (http://localhost:8080), and launch an app using 
> the following config
> {code}
> {
>   "id": "/test",
>   "cmd": "dd if=/dev/zero of=file bs=1024 count=1 oflag=dsync",
>   "cpus": 1,
>   "mem": 128,
>   "disk": 128,
>   "instances": 1,
>   "container": {
> "type": "MESOS",
> "docker": {
>   "image": "ubuntu:18.04"
> }
>   }
> }
> {code}
> You'll see the task failed with "Permission Denied".
> The task will run normally if you use `mesos/mesos-mini:master-2018-12-01`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (MESOS-9462) Devices in a container are inaccessible due to `nodev` on `/var/run`.

2018-12-09 Thread Jie Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/MESOS-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714274#comment-16714274
 ] 

Jie Yu commented on MESOS-9462:
---

commit f050bf01af8f9f92bbada2c0a2025a459290ed98 (HEAD -> master, origin/master, 
origin/HEAD, remount_runtime_dir)
Author: Jie Yu 
Date:   Fri Dec 7 16:51:19 2018 -0800

Made sure containers runtime dir has device file access.

Make sure that container's runtime dir has device file access.  Some
Linux distributions will mount `/run` with `nodev`, restricting
accessing to device files under `/run`. However, Mesos prepares device
files for containers under container's runtime dir (which is typically
under `/run`) and bind mount into container root filesystems. Therefore,
we need to make sure those device files can be accessed by the
container. We need to do a self bind mount and remount with proper
options if necessary. See MESOS-9462 for more details.

Review: https://reviews.apache.org/r/69532

commit 35cdfa2c0bcb75f5801ec60671a3f978b4aa645f
Author: Jie Yu 
Date:   Sun Dec 9 19:14:09 2018 -0800

Used strings::format in os::shell.

Previously, `strings::internal::format` was used. It causes issues when
std::string is passed in as parameters. Switched to use
`strings::format` instead in `os::shell` implementation.

Review: https://reviews.apache.org/r/69537

> Devices in a container are inaccessible due to `nodev` on `/var/run`.
> -
>
> Key: MESOS-9462
> URL: https://issues.apache.org/jira/browse/MESOS-9462
> Project: Mesos
>  Issue Type: Bug
>  Components: containerization
>Affects Versions: 1.8.0
>Reporter: Jie Yu
>Assignee: Jie Yu
>Priority: Blocker
>  Labels: regression
>
> A recent [patch|https://reviews.apache.org/r/69086/] (commit 
> ede8155d1d043137e15007c48da36ac5fa0b5124) changes the behavior of how 
> standard device nodes (e.g., /dev/null, etc.) are setup. It uses bind mount 
> (from host) now (instead of mknod).
> The devices nodes are created under 
> `/var/run/mesos/containers//devices`, and then bind mounted to 
> the container root filesystem. This is problematic for those Linux distros 
> that mount `/var/run` (or `/run`) as `nodev`. For instance, CentOS 7.4:
> {noformat}
> [jie@core-dev ~]$ cat /proc/self/mountinfo | grep "/run\ "
>   
>
> 24 62 0:19 / /run rw,nosuid,nodev shared:23 - tmpfs tmpfs rw,seclabel,mode=755
> [jie@core-dev ~]$ cat /etc/redhat-release 
> CentOS Linux release 7.4.1708 (Core) 
> {noformat}
> As a result, the `/dev/null` devices in the container will inherit the 
> `nodev` from `/run` on the host
> {noformat}
> 629 625 0:121 
> /mesos/containers/49f1da14-d741-4030-994c-0d8ed5093b13/devices/null /dev/null 
> rw,nosuid,nodev - tmpfs tmpfs rw,mode=755
> {noformat}
> This will cause "Permission Denied" error when a process in the container 
> tries to open the device node.
> You can try to reproduce this issue using Mesos Mini
> {noformat}
> docker run --rm --privileged -p 5050:5050 -p 5051:5051 -p 8080:8080 
> mesos/mesos-mini:master-2018-12-06
> {noformat}
> And the, go to Marathon UI (http://localhost:8080), and launch an app using 
> the following config
> {code}
> {
>   "id": "/test",
>   "cmd": "dd if=/dev/zero of=file bs=1024 count=1 oflag=dsync",
>   "cpus": 1,
>   "mem": 128,
>   "disk": 128,
>   "instances": 1,
>   "container": {
> "type": "MESOS",
> "docker": {
>   "image": "ubuntu:18.04"
> }
>   }
> }
> {code}
> You'll see the task failed with "Permission Denied".
> The task will run normally if you use `mesos/mesos-mini:master-2018-12-01`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (MESOS-9462) Devices in a container are inaccessible due to `nodev` on `/var/run`.

2018-12-07 Thread Jie Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/MESOS-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713286#comment-16713286
 ] 

Jie Yu commented on MESOS-9462:
---

The fix is to make sure  is a mounted w/o `nodev`. If the distro 
mount `/var/run` using `nodev`, during agent startup, we can perform a one time 
self bind mount, and a remount with `-o remount,dev`. This can ensure that 
device nodes created in `` can be accessed.

> Devices in a container are inaccessible due to `nodev` on `/var/run`.
> -
>
> Key: MESOS-9462
> URL: https://issues.apache.org/jira/browse/MESOS-9462
> Project: Mesos
>  Issue Type: Bug
>  Components: containerization
>Affects Versions: 1.8.0
>Reporter: Jie Yu
>Priority: Blocker
>  Labels: regression
>
> A recent [patch|https://reviews.apache.org/r/69086/] (commit 
> ede8155d1d043137e15007c48da36ac5fa0b5124) changes the behavior of how 
> standard device nodes (e.g., /dev/null, etc.) are setup. It uses bind mount 
> (from host) now (instead of mknod).
> The devices nodes are created under 
> `/var/run/mesos/containers//devices`, and then bind mounted to 
> the container root filesystem. This is problematic for those Linux distros 
> that mount `/var/run` (or `/run`) as `nodev`. For instance, CentOS 7.4:
> {noformat}
> [jie@core-dev ~]$ cat /proc/self/mountinfo | grep "/run\ "
>   
>
> 24 62 0:19 / /run rw,nosuid,nodev shared:23 - tmpfs tmpfs rw,seclabel,mode=755
> [jie@core-dev ~]$ cat /etc/redhat-release 
> CentOS Linux release 7.4.1708 (Core) 
> {noformat}
> As a result, the `/dev/null` devices in the container will inherit the 
> `nodev` from `/run` on the host
> {noformat}
> 629 625 0:121 
> /mesos/containers/49f1da14-d741-4030-994c-0d8ed5093b13/devices/null /dev/null 
> rw,nosuid,nodev - tmpfs tmpfs rw,mode=755
> {noformat}
> This will cause "Permission Denied" error when a process in the container 
> tries to open the device node.
> You can try to reproduce this issue using Mesos Mini
> {noformat}
> docker run --rm --privileged -p 5050:5050 -p 5051:5051 -p 8080:8080 
> mesos/mesos-mini:master-2018-12-06
> {noformat}
> And the, go to Marathon UI (http://localhost:8080), and launch an app using 
> the following config
> {code}
> {
>   "id": "/test",
>   "cmd": "dd if=/dev/zero of=file bs=1024 count=1 oflag=dsync",
>   "cpus": 1,
>   "mem": 128,
>   "disk": 128,
>   "instances": 1,
>   "container": {
> "type": "MESOS",
> "docker": {
>   "image": "ubuntu:18.04"
> }
>   }
> }
> {code}
> You'll see the task failed with "Permission Denied".
> The task will run normally if you use `mesos/mesos-mini:master-2018-12-01`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)