[jira] [Comment Edited] (MESOS-9536) Nested container launched with non-root user may not be able to write to its sandbox via the environment variable `MESOS_SANDBOX`
[
https://issues.apache.org/jira/browse/MESOS-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846398#comment-16846398
]
Qian Zhang edited comment on MESOS-9536 at 5/23/19 3:10 AM:
The root cause of the secret issue is, in [this
patch|https://reviews.apache.org/r/70514/], for the nested container which has
no its own rootfs we bind mount it sandbox directory to the directory
`/mnt/mesos/sandbox` (i.e., the agent flag `–sandbox_directory`) in the
container's mount namespace, these two directories can see the same file
operations (e.g., create/delete files) but NOT the same mount/unmount
operations. The reason in `launch.cpp` we make the `/` in the container mount
namespace as a slave mount of the `/` in the host mount namespace, and the
above bind mount is also a slave mount of the `/` in the host mount namespace,
so the mount operations we do under host's `/` can be propagated to these two
slave mounts, but the mount propagation will not happen between these two slave
mounts. So the ramfs that the `volume/secret` isolator mounts in container's
sandbox directory will not be propagated to `/mnt/mesos/sandbox` and the
subsequent bind mount from the ramfs to the target secret file in the
container's sandbox directory will not be propagated to `/mnt/mesos/sandbox`
too, that's why we see a correct secret file in container's sandbox directory
but an empty secret file in `/mnt/mesos/sandbox`.
The possible fix would be, before bind mount container's sandbox to
`/mnt/mesos/sandbox`, we should make container's sandbox a shared mount first,
in this way the container's sandbox and `/mnt/mesos/sandbox` will be in the
same peer mount group so the mount propagation will happen between them. And we
may also need to allow shared mount in `launch.cpp`, here is a draft fix based
on [this patch|https://reviews.apache.org/r/70514/]:
{code:java}
diff --git a/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
b/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
index 7b50258ef..70dc82b84 100644
--- a/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
+++ b/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
@@ -765,6 +765,12 @@ Future>
LinuxFilesystemIsolatorProcess::prepare(
// sandbox, if we still set `MESOS_SANDBOX` to `containerConfig.directory()`
// for nested container, it will not have permission to access its sandbox
// via `MESOS_SANDBOX` if its user is different from its parent's user.
+ *launchInfo.add_mounts() = createContainerMount(
+ containerConfig.directory(), containerConfig.directory(), MS_BIND);
+
+ *launchInfo.add_mounts() = createContainerMount(
+ containerConfig.directory(), containerConfig.directory(), MS_SHARED);
+
*launchInfo.add_mounts() = createContainerMount(
containerConfig.directory(), flags.sandbox_directory, MS_BIND | MS_REC);
}
diff --git a/src/slave/containerizer/mesos/launch.cpp
b/src/slave/containerizer/mesos/launch.cpp
index 88b97a572..e3e851b14 100644
--- a/src/slave/containerizer/mesos/launch.cpp
+++ b/src/slave/containerizer/mesos/launch.cpp
@@ -299,7 +299,7 @@ static Try prepareMounts(const
ContainerLaunchInfo& launchInfo)
launchInfo.mounts().begin(),
launchInfo.mounts().end(),
[](const ContainerMountInfo& mount) {
- return (mount.flags() & MS_SHARED) != 0;
+ return !mount.has_source() && ((mount.flags() & MS_SHARED) != 0);
}) != launchInfo.mounts().end();
if (!hasSharedMount) {
@@ -348,7 +348,7 @@ static Try prepareMounts(const
ContainerLaunchInfo& launchInfo)
foreach (const ContainerMountInfo& mount, launchInfo.mounts()) {
// Skip those mounts that are used for setting up propagation.
- if ((mount.flags() & MS_SHARED) != 0) {
+ if (!mount.has_source() && ((mount.flags() & MS_SHARED) != 0)) {
continue;
}
{code}
And there may be another issue, the nested container may not have permissions
to enter `/mnt/mesos/sandbox` if it is launched as a non-root user. The root
cause is in [this patch|https://reviews.apache.org/r/70514/] we create the
directory `/mnt/mesos/sandbox` as root user (since agent is running as root),
we may need to set the permissions of `/mnt` and `/mnt/mesos` to `drwxr-xr-x`
so that the non-root user can enter it.
was (Author: qianzhang):
The root cause of the secret issue is, in [this
patch|https://reviews.apache.org/r/70514/], for the nested container which has
no its own rootfs we bind mount it sandbox directory to the directory
`/mnt/mesos/sandbox` (i.e., the agent flag `–sandbox_directory`), these two
directories can see the same file operations (e.g., create/delete files) but
NOT the same mount/unmount operations. The reason in `launch.cpp` we make the
`/` in the container mount namespace as a slave mount of the `/` in the host
mount namespace, and the above bind mount is also a slave mount of the `/` in
the host mount namespace, so the mount operations we do under host's `/` can be
[jira] [Comment Edited] (MESOS-9536) Nested container launched with non-root user may not be able to write to its sandbox via the environment variable `MESOS_SANDBOX`
[
https://issues.apache.org/jira/browse/MESOS-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846398#comment-16846398
]
Qian Zhang edited comment on MESOS-9536 at 5/23/19 3:10 AM:
The root cause of the secret issue is, in [this
patch|https://reviews.apache.org/r/70514/], for the nested container which has
no its own rootfs we bind mount it sandbox directory to the directory
`/mnt/mesos/sandbox` (i.e., the agent flag `–sandbox_directory`) in the
container's mount namespace, these two directories can see the same file
operations (e.g., create/delete files) but NOT the same mount/unmount
operations. The reason is in `launch.cpp` we make the `/` in the container
mount namespace as a slave mount of the `/` in the host mount namespace, and
the above bind mount is also a slave mount of the `/` in the host mount
namespace, so the mount operations we do under host's `/` can be propagated to
these two slave mounts, but the mount propagation will not happen between these
two slave mounts. So the ramfs that the `volume/secret` isolator mounts in
container's sandbox directory will not be propagated to `/mnt/mesos/sandbox`
and the subsequent bind mount from the ramfs to the target secret file in the
container's sandbox directory will not be propagated to `/mnt/mesos/sandbox`
too, that's why we see a correct secret file in container's sandbox directory
but an empty secret file in `/mnt/mesos/sandbox`.
The possible fix would be, before bind mount container's sandbox to
`/mnt/mesos/sandbox`, we should make container's sandbox a shared mount first,
in this way the container's sandbox and `/mnt/mesos/sandbox` will be in the
same peer mount group so the mount propagation will happen between them. And we
may also need to allow shared mount in `launch.cpp`, here is a draft fix based
on [this patch|https://reviews.apache.org/r/70514/]:
{code:java}
diff --git a/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
b/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
index 7b50258ef..70dc82b84 100644
--- a/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
+++ b/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
@@ -765,6 +765,12 @@ Future>
LinuxFilesystemIsolatorProcess::prepare(
// sandbox, if we still set `MESOS_SANDBOX` to `containerConfig.directory()`
// for nested container, it will not have permission to access its sandbox
// via `MESOS_SANDBOX` if its user is different from its parent's user.
+ *launchInfo.add_mounts() = createContainerMount(
+ containerConfig.directory(), containerConfig.directory(), MS_BIND);
+
+ *launchInfo.add_mounts() = createContainerMount(
+ containerConfig.directory(), containerConfig.directory(), MS_SHARED);
+
*launchInfo.add_mounts() = createContainerMount(
containerConfig.directory(), flags.sandbox_directory, MS_BIND | MS_REC);
}
diff --git a/src/slave/containerizer/mesos/launch.cpp
b/src/slave/containerizer/mesos/launch.cpp
index 88b97a572..e3e851b14 100644
--- a/src/slave/containerizer/mesos/launch.cpp
+++ b/src/slave/containerizer/mesos/launch.cpp
@@ -299,7 +299,7 @@ static Try prepareMounts(const
ContainerLaunchInfo& launchInfo)
launchInfo.mounts().begin(),
launchInfo.mounts().end(),
[](const ContainerMountInfo& mount) {
- return (mount.flags() & MS_SHARED) != 0;
+ return !mount.has_source() && ((mount.flags() & MS_SHARED) != 0);
}) != launchInfo.mounts().end();
if (!hasSharedMount) {
@@ -348,7 +348,7 @@ static Try prepareMounts(const
ContainerLaunchInfo& launchInfo)
foreach (const ContainerMountInfo& mount, launchInfo.mounts()) {
// Skip those mounts that are used for setting up propagation.
- if ((mount.flags() & MS_SHARED) != 0) {
+ if (!mount.has_source() && ((mount.flags() & MS_SHARED) != 0)) {
continue;
}
{code}
And there may be another issue, the nested container may not have permissions
to enter `/mnt/mesos/sandbox` if it is launched as a non-root user. The root
cause is in [this patch|https://reviews.apache.org/r/70514/] we create the
directory `/mnt/mesos/sandbox` as root user (since agent is running as root),
we may need to set the permissions of `/mnt` and `/mnt/mesos` to `drwxr-xr-x`
so that the non-root user can enter it.
was (Author: qianzhang):
The root cause of the secret issue is, in [this
patch|https://reviews.apache.org/r/70514/], for the nested container which has
no its own rootfs we bind mount it sandbox directory to the directory
`/mnt/mesos/sandbox` (i.e., the agent flag `–sandbox_directory`) in the
container's mount namespace, these two directories can see the same file
operations (e.g., create/delete files) but NOT the same mount/unmount
operations. The reason in `launch.cpp` we make the `/` in the container mount
namespace as a slave mount of the `/` in the host mount namespace, and the
above bind mount is also a slave mount of the `/` in the host mount namespace,
so the mount
[jira] [Commented] (MESOS-9536) Nested container launched with non-root user may not be able to write to its sandbox via the environment variable `MESOS_SANDBOX`
[
https://issues.apache.org/jira/browse/MESOS-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846398#comment-16846398
]
Qian Zhang commented on MESOS-9536:
---
The root cause of the secret issue is, in [this
patch|https://reviews.apache.org/r/70514/], for the nested container which has
no its own rootfs we bind mount it sandbox directory to the directory
`/mnt/mesos/sandbox` (i.e., the agent flag `–sandbox_directory`), these two
directories can see the same file operations (e.g., create/delete files) but
NOT the same mount/unmount operations. The reason in `launch.cpp` we make the
`/` in the container mount namespace as a slave mount of the `/` in the host
mount namespace, and the above bind mount is also a slave mount of the `/` in
the host mount namespace, so the mount operations we do under host's `/` can be
propagated to these two slave mounts, but the mount propagation will not happen
between these two slave mounts. So the ramfs that the `volume/secret` isolator
mounts in container's sandbox directory will not be propagated to
`/mnt/mesos/sandbox` and the subsequent bind mount from the ramfs to the target
secret file in the container's sandbox directory will not be propagated to
`/mnt/mesos/sandbox` too, that's why we see a correct secret file in
container's sandbox directory but an empty secret file in `/mnt/mesos/sandbox`.
The possible fix would be, before bind mount container's sandbox to
`/mnt/mesos/sandbox`, we should make container's sandbox a shared mount first,
in this way the container's sandbox and `/mnt/mesos/sandbox` will be in the
same peer mount group so the mount propagation will happen between them. And we
may also need to allow shared mount in `launch.cpp`, here is a draft fix based
on [this patch|https://reviews.apache.org/r/70514/]:
{code:java}
diff --git a/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
b/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
index 7b50258ef..70dc82b84 100644
--- a/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
+++ b/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
@@ -765,6 +765,12 @@ Future>
LinuxFilesystemIsolatorProcess::prepare(
// sandbox, if we still set `MESOS_SANDBOX` to `containerConfig.directory()`
// for nested container, it will not have permission to access its sandbox
// via `MESOS_SANDBOX` if its user is different from its parent's user.
+ *launchInfo.add_mounts() = createContainerMount(
+ containerConfig.directory(), containerConfig.directory(), MS_BIND);
+
+ *launchInfo.add_mounts() = createContainerMount(
+ containerConfig.directory(), containerConfig.directory(), MS_SHARED);
+
*launchInfo.add_mounts() = createContainerMount(
containerConfig.directory(), flags.sandbox_directory, MS_BIND | MS_REC);
}
diff --git a/src/slave/containerizer/mesos/launch.cpp
b/src/slave/containerizer/mesos/launch.cpp
index 88b97a572..e3e851b14 100644
--- a/src/slave/containerizer/mesos/launch.cpp
+++ b/src/slave/containerizer/mesos/launch.cpp
@@ -299,7 +299,7 @@ static Try prepareMounts(const
ContainerLaunchInfo& launchInfo)
launchInfo.mounts().begin(),
launchInfo.mounts().end(),
[](const ContainerMountInfo& mount) {
- return (mount.flags() & MS_SHARED) != 0;
+ return !mount.has_source() && ((mount.flags() & MS_SHARED) != 0);
}) != launchInfo.mounts().end();
if (!hasSharedMount) {
@@ -348,7 +348,7 @@ static Try prepareMounts(const
ContainerLaunchInfo& launchInfo)
foreach (const ContainerMountInfo& mount, launchInfo.mounts()) {
// Skip those mounts that are used for setting up propagation.
- if ((mount.flags() & MS_SHARED) != 0) {
+ if (!mount.has_source() && ((mount.flags() & MS_SHARED) != 0)) {
continue;
}
{code}
And there may be another issue, the nested container may not have permissions
to enter `/mnt/mesos/sandbox` if it is launched as a non-root user. The root
cause is in [this patch|https://reviews.apache.org/r/70514/] we create the
directory `/mnt/mesos/sandbox` as root user (since agent is running as root),
we may need to set the permissions of `/mnt` and `/mnt/mesos` to `drwxr-xr-x`
so that the non-root user can enter it.
> Nested container launched with non-root user may not be able to write to its
> sandbox via the environment variable `MESOS_SANDBOX`
> -
>
> Key: MESOS-9536
> URL: https://issues.apache.org/jira/browse/MESOS-9536
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Affects Versions: 1.6.0, 1.6.1, 1.7.0, 1.8.0
>Reporter: Qian Zhang
>Assignee: Qian Zhang
>Priority: Critical
>
> Launch a nested container to write to its sandbox via the env var
> `MESOS_SANDBOX`. The nested container is launched with a non-root user (e.g.,
> `nobody`)
[jira] [Assigned] (MESOS-9769) Add direct containerized support for filesystem operations.
[ https://issues.apache.org/jira/browse/MESOS-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Peach reassigned MESOS-9769: -- Assignee: James Peach > Add direct containerized support for filesystem operations. > --- > > Key: MESOS-9769 > URL: https://issues.apache.org/jira/browse/MESOS-9769 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: James Peach >Assignee: James Peach >Priority: Major > > When setting up the container filesystems, we use `pre_exec_commands` to make > ABI symlinks and other things. The problem with this is that, depending of > the order of operations, we may not have the full security policy in place > yet, but since we are running in the context of the container's mount > namespaces, the programs we execute are under the control of whoever built > the container image. > [~jieyu] and I previously discussed adding filesystem operations to the > `ContainerLaunchInfo`. Just `ln` would be sufficient for the `cgroups` and > `linux/filesystem` isolators. Secrets and port mapping isolators need more, > so we should discuss and file new tickets if necessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (MESOS-9757) Design doc for container debug endpoint.
[ https://issues.apache.org/jira/browse/MESOS-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gilbert Song reassigned MESOS-9757: --- Assignee: Andrei Budnik > Design doc for container debug endpoint. > > > Key: MESOS-9757 > URL: https://issues.apache.org/jira/browse/MESOS-9757 > Project: Mesos > Issue Type: Task > Components: containerization >Reporter: Gilbert Song >Assignee: Andrei Budnik >Priority: Major > Labels: containerization > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MESOS-9795) Support /dev/shm and configurable IPC namespace.
Gilbert Song created MESOS-9795: --- Summary: Support /dev/shm and configurable IPC namespace. Key: MESOS-9795 URL: https://issues.apache.org/jira/browse/MESOS-9795 Project: Mesos Issue Type: Epic Components: containerization Reporter: Gilbert Song Assignee: Qian Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9775) Design doc for UCR shared memory.
[ https://issues.apache.org/jira/browse/MESOS-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846119#comment-16846119 ] Gilbert Song commented on MESOS-9775: - https://docs.google.com/document/d/10t1jf97vrejUWEVSvxGtqw4vhzfPef41JMzb5jw7l1s/edit > Design doc for UCR shared memory. > - > > Key: MESOS-9775 > URL: https://issues.apache.org/jira/browse/MESOS-9775 > Project: Mesos > Issue Type: Task > Components: containerization >Reporter: Gilbert Song >Assignee: Qian Zhang >Priority: Major > Labels: containerization > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (MESOS-9775) Design doc for UCR shared memory.
[ https://issues.apache.org/jira/browse/MESOS-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gilbert Song reassigned MESOS-9775: --- Assignee: Qian Zhang > Design doc for UCR shared memory. > - > > Key: MESOS-9775 > URL: https://issues.apache.org/jira/browse/MESOS-9775 > Project: Mesos > Issue Type: Task > Components: containerization >Reporter: Gilbert Song >Assignee: Qian Zhang >Priority: Major > Labels: containerization > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MESOS-9794) Design doc for container debug endpoint.
Gilbert Song created MESOS-9794: --- Summary: Design doc for container debug endpoint. Key: MESOS-9794 URL: https://issues.apache.org/jira/browse/MESOS-9794 Project: Mesos Issue Type: Task Components: containerization Reporter: Gilbert Song Design doc for container debug endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (MESOS-9793) Implement UPDATE_FRAMEWORK call in V0 API
[ https://issues.apache.org/jira/browse/MESOS-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Sekretenko reassigned MESOS-9793: Assignee: Andrei Sekretenko > Implement UPDATE_FRAMEWORK call in V0 API > - > > Key: MESOS-9793 > URL: https://issues.apache.org/jira/browse/MESOS-9793 > Project: Mesos > Issue Type: Task >Reporter: Andrei Sekretenko >Assignee: Andrei Sekretenko >Priority: Major > Labels: multitenancy, resource-management > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MESOS-9793) Implement UPDATE_FRAMEWORK call in V0 API
Andrei Sekretenko created MESOS-9793: Summary: Implement UPDATE_FRAMEWORK call in V0 API Key: MESOS-9793 URL: https://issues.apache.org/jira/browse/MESOS-9793 Project: Mesos Issue Type: Bug Reporter: Andrei Sekretenko -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-8808) CSI documentation has a broken link to a non-existent page.
[
https://issues.apache.org/jira/browse/MESOS-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846008#comment-16846008
]
Chun-Hung Hsiao commented on MESOS-8808:
The link is missing because of this pending RR:
https://reviews.apache.org/r/65112/
> CSI documentation has a broken link to a non-existent page.
> ---
>
> Key: MESOS-8808
> URL: https://issues.apache.org/jira/browse/MESOS-8808
> Project: Mesos
> Issue Type: Bug
> Components: documentation, storage
>Affects Versions: 1.5.0
>Reporter: Gastón Kleiman
>Priority: Major
> Labels: csi, documentation, mesosphere
>
> There's a broken link to a non-existent {{resource-provider.md}} document
> here: https://mesos.apache.org/documentation/latest/csi/#resource-providers
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (MESOS-8960) StorageLocalResourceProviderTest.ROOT_Metrics is flaky.
[
https://issues.apache.org/jira/browse/MESOS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chun-Hung Hsiao updated MESOS-8960:
---
Comment: was deleted
(was: Actually this has been fixed. It's a duplicate of MESOS-8548.)
> StorageLocalResourceProviderTest.ROOT_Metrics is flaky.
> ---
>
> Key: MESOS-8960
> URL: https://issues.apache.org/jira/browse/MESOS-8960
> Project: Mesos
> Issue Type: Bug
> Components: test
> Environment: Ubuntu 16.04 with SSL and GRPC
>Reporter: Alexander Rukletsov
>Priority: Major
> Labels: flaky-test, mesosphere, storage
> Attachments: StorageLocalResourceProviderTest.ROOT_Metrics-badrun.txt
>
>
> {noformat}
> ../../src/tests/storage_local_resource_provider_tests.cpp:2859
> Failed to wait 15secs for pluginRestarted
> {noformat}
> Full log attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (MESOS-4260) Volumes not handled correctly for command tasks
[ https://issues.apache.org/jira/browse/MESOS-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Orlando Hohmeier reassigned MESOS-4260: --- Assignee: (was: Timothy Chen) > Volumes not handled correctly for command tasks > --- > > Key: MESOS-4260 > URL: https://issues.apache.org/jira/browse/MESOS-4260 > Project: Mesos > Issue Type: Bug > Components: containerization >Reporter: Timothy Chen >Priority: Major > > Volumes are currently not handled correctly as we strip the volumes in > ExecutorInfo when running a command task -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MESOS-9792) Invalid cpu-shares mapping for Windows containers
Andrei Stryia created MESOS-9792: Summary: Invalid cpu-shares mapping for Windows containers Key: MESOS-9792 URL: https://issues.apache.org/jira/browse/MESOS-9792 Project: Mesos Issue Type: Bug Reporter: Andrei Stryia According to documentation Windows containers have cpu-shares range as 1-1 https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/resource-controls#cpu-shares-without-hyper-v-isolation. While Mesos map it as proportion of 1024. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
