[jira] [Assigned] (MESOS-9462) Devices in a container are inaccessible due to `nodev` on `/var/run`.
[ https://issues.apache.org/jira/browse/MESOS-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu reassigned MESOS-9462: - Assignee: Andrei Budnik (was: Jie Yu) > Devices in a container are inaccessible due to `nodev` on `/var/run`. > - > > Key: MESOS-9462 > URL: https://issues.apache.org/jira/browse/MESOS-9462 > Project: Mesos > Issue Type: Bug > Components: containerization >Affects Versions: 1.8.0 >Reporter: Jie Yu >Assignee: Andrei Budnik >Priority: Blocker > Labels: regression > > A recent [patch|https://reviews.apache.org/r/69086/] (commit > ede8155d1d043137e15007c48da36ac5fa0b5124) changes the behavior of how > standard device nodes (e.g., /dev/null, etc.) are setup. It uses bind mount > (from host) now (instead of mknod). > The devices nodes are created under > `/var/run/mesos/containers//devices`, and then bind mounted to > the container root filesystem. This is problematic for those Linux distros > that mount `/var/run` (or `/run`) as `nodev`. For instance, CentOS 7.4: > {noformat} > [jie@core-dev ~]$ cat /proc/self/mountinfo | grep "/run\ " > > > 24 62 0:19 / /run rw,nosuid,nodev shared:23 - tmpfs tmpfs rw,seclabel,mode=755 > [jie@core-dev ~]$ cat /etc/redhat-release > CentOS Linux release 7.4.1708 (Core) > {noformat} > As a result, the `/dev/null` devices in the container will inherit the > `nodev` from `/run` on the host > {noformat} > 629 625 0:121 > /mesos/containers/49f1da14-d741-4030-994c-0d8ed5093b13/devices/null /dev/null > rw,nosuid,nodev - tmpfs tmpfs rw,mode=755 > {noformat} > This will cause "Permission Denied" error when a process in the container > tries to open the device node. > You can try to reproduce this issue using Mesos Mini > {noformat} > docker run --rm --privileged -p 5050:5050 -p 5051:5051 -p 8080:8080 > mesos/mesos-mini:master-2018-12-06 > {noformat} > And the, go to Marathon UI (http://localhost:8080), and launch an app using > the following config > {code} > { > "id": "/test", > "cmd": "dd if=/dev/zero of=file bs=1024 count=1 oflag=dsync", > "cpus": 1, > "mem": 128, > "disk": 128, > "instances": 1, > "container": { > "type": "MESOS", > "docker": { > "image": "ubuntu:18.04" > } > } > } > {code} > You'll see the task failed with "Permission Denied". > The task will run normally if you use `mesos/mesos-mini:master-2018-12-01` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9462) Devices in a container are inaccessible due to `nodev` on `/var/run`.
[ https://issues.apache.org/jira/browse/MESOS-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714274#comment-16714274 ] Jie Yu commented on MESOS-9462: --- commit f050bf01af8f9f92bbada2c0a2025a459290ed98 (HEAD -> master, origin/master, origin/HEAD, remount_runtime_dir) Author: Jie Yu Date: Fri Dec 7 16:51:19 2018 -0800 Made sure containers runtime dir has device file access. Make sure that container's runtime dir has device file access. Some Linux distributions will mount `/run` with `nodev`, restricting accessing to device files under `/run`. However, Mesos prepares device files for containers under container's runtime dir (which is typically under `/run`) and bind mount into container root filesystems. Therefore, we need to make sure those device files can be accessed by the container. We need to do a self bind mount and remount with proper options if necessary. See MESOS-9462 for more details. Review: https://reviews.apache.org/r/69532 commit 35cdfa2c0bcb75f5801ec60671a3f978b4aa645f Author: Jie Yu Date: Sun Dec 9 19:14:09 2018 -0800 Used strings::format in os::shell. Previously, `strings::internal::format` was used. It causes issues when std::string is passed in as parameters. Switched to use `strings::format` instead in `os::shell` implementation. Review: https://reviews.apache.org/r/69537 > Devices in a container are inaccessible due to `nodev` on `/var/run`. > - > > Key: MESOS-9462 > URL: https://issues.apache.org/jira/browse/MESOS-9462 > Project: Mesos > Issue Type: Bug > Components: containerization >Affects Versions: 1.8.0 >Reporter: Jie Yu >Assignee: Jie Yu >Priority: Blocker > Labels: regression > > A recent [patch|https://reviews.apache.org/r/69086/] (commit > ede8155d1d043137e15007c48da36ac5fa0b5124) changes the behavior of how > standard device nodes (e.g., /dev/null, etc.) are setup. It uses bind mount > (from host) now (instead of mknod). > The devices nodes are created under > `/var/run/mesos/containers//devices`, and then bind mounted to > the container root filesystem. This is problematic for those Linux distros > that mount `/var/run` (or `/run`) as `nodev`. For instance, CentOS 7.4: > {noformat} > [jie@core-dev ~]$ cat /proc/self/mountinfo | grep "/run\ " > > > 24 62 0:19 / /run rw,nosuid,nodev shared:23 - tmpfs tmpfs rw,seclabel,mode=755 > [jie@core-dev ~]$ cat /etc/redhat-release > CentOS Linux release 7.4.1708 (Core) > {noformat} > As a result, the `/dev/null` devices in the container will inherit the > `nodev` from `/run` on the host > {noformat} > 629 625 0:121 > /mesos/containers/49f1da14-d741-4030-994c-0d8ed5093b13/devices/null /dev/null > rw,nosuid,nodev - tmpfs tmpfs rw,mode=755 > {noformat} > This will cause "Permission Denied" error when a process in the container > tries to open the device node. > You can try to reproduce this issue using Mesos Mini > {noformat} > docker run --rm --privileged -p 5050:5050 -p 5051:5051 -p 8080:8080 > mesos/mesos-mini:master-2018-12-06 > {noformat} > And the, go to Marathon UI (http://localhost:8080), and launch an app using > the following config > {code} > { > "id": "/test", > "cmd": "dd if=/dev/zero of=file bs=1024 count=1 oflag=dsync", > "cpus": 1, > "mem": 128, > "disk": 128, > "instances": 1, > "container": { > "type": "MESOS", > "docker": { > "image": "ubuntu:18.04" > } > } > } > {code} > You'll see the task failed with "Permission Denied". > The task will run normally if you use `mesos/mesos-mini:master-2018-12-01` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (MESOS-9463) Parallel test runner gets confused if a GTEST_FILTER expression also matches a sequential filter
[ https://issues.apache.org/jira/browse/MESOS-9463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Bannier reassigned MESOS-9463: --- Assignee: (was: Benjamin Bannier) Labels: parallel-tests test (was: ) > Parallel test runner gets confused if a GTEST_FILTER expression also matches > a sequential filter > > > Key: MESOS-9463 > URL: https://issues.apache.org/jira/browse/MESOS-9463 > Project: Mesos > Issue Type: Bug >Reporter: Benjamin Bannier >Priority: Major > Labels: parallel-tests, test > > Users expect the be able to select tests to run via {{make check}} with a > {{GTEST_FILTER}} environment variable. The parallel test runner on the other > hand programmatically also injects filter expressions to select tests to > execute sequentially. > This causes e.g., all {{*ROOT_*}} tests to be run in the sequential phase for > superusers, even if a {{GTEST_FILTER}} was set. > It seems that need to handle set {{GTEST_FILTER}} environment variables more > carefully. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MESOS-9464) Parallel test runner executes all tests regardless of GTEST_FILTER for superusers
Benjamin Bannier created MESOS-9464: --- Summary: Parallel test runner executes all tests regardless of GTEST_FILTER for superusers Key: MESOS-9464 URL: https://issues.apache.org/jira/browse/MESOS-9464 Project: Mesos Issue Type: Bug Reporter: Benjamin Bannier The parallel test runner currently does not correctly handle superuser sessions. We probably need to add some smarts to how test set of sequential tests is formed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MESOS-9463) Parallel test runner gets confused if a GTEST_FILTER expression also matches a sequential filter
Benjamin Bannier created MESOS-9463: --- Summary: Parallel test runner gets confused if a GTEST_FILTER expression also matches a sequential filter Key: MESOS-9463 URL: https://issues.apache.org/jira/browse/MESOS-9463 Project: Mesos Issue Type: Bug Reporter: Benjamin Bannier Users expect the be able to select tests to run via {{make check}} with a {{GTEST_FILTER}} environment variable. The parallel test runner on the other hand programmatically also injects filter expressions to select tests to execute sequentially. This causes e.g., all {{*ROOT_*}} tests to be run in the sequential phase for superusers, even if a {{GTEST_FILTER}} was set. It seems that need to handle set {{GTEST_FILTER}} environment variables more carefully. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (MESOS-9463) Parallel test runner gets confused if a GTEST_FILTER expression also matches a sequential filter
[ https://issues.apache.org/jira/browse/MESOS-9463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Bannier reassigned MESOS-9463: --- Assignee: Benjamin Bannier > Parallel test runner gets confused if a GTEST_FILTER expression also matches > a sequential filter > > > Key: MESOS-9463 > URL: https://issues.apache.org/jira/browse/MESOS-9463 > Project: Mesos > Issue Type: Bug >Reporter: Benjamin Bannier >Assignee: Benjamin Bannier >Priority: Major > > Users expect the be able to select tests to run via {{make check}} with a > {{GTEST_FILTER}} environment variable. The parallel test runner on the other > hand programmatically also injects filter expressions to select tests to > execute sequentially. > This causes e.g., all {{*ROOT_*}} tests to be run in the sequential phase for > superusers, even if a {{GTEST_FILTER}} was set. > It seems that need to handle set {{GTEST_FILTER}} environment variables more > carefully. -- This message was sent by Atlassian JIRA (v7.6.3#76005)