[
https://issues.apache.org/jira/browse/MESOS-9116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16603472#comment-16603472
]
Gilbert Song commented on MESOS-9116:
-------------------------------------
Backporting to 1.4.x is not trivial. Adding new dependencies MESOS-6894 would
change the containerizer layout and users still have to drain all running
containers during upgrade. Let's do not backport to 1.4.x and people who need
this fix should upgrade to 1.5.x +
> Launch nested container session fails due to incorrect detection of `mnt`
> namespace of command executor's task.
> ---------------------------------------------------------------------------------------------------------------
>
> Key: MESOS-9116
> URL: https://issues.apache.org/jira/browse/MESOS-9116
> Project: Mesos
> Issue Type: Bug
> Components: agent, containerization
> Reporter: Andrei Budnik
> Assignee: Andrei Budnik
> Priority: Critical
> Labels: mesosphere
> Fix For: 1.4.3, 1.5.2, 1.6.2, 1.7.0
>
> Attachments: pstree.png
>
>
> Launch nested container call might fail with the following error:
> {code:java}
> Failed to enter mount namespace: Failed to open '/proc/29473/ns/mnt': No such
> file or directory
> {code}
> This happens when the containerizer launcher [tries to
> enter|https://github.com/apache/mesos/blob/077f122d52671412a2ab5d992d535712cc154002/src/slave/containerizer/mesos/launch.cpp#L879-L892]
> `mnt` namespace using the pid of a terminated process. The pid [was
> detected|https://github.com/apache/mesos/blob/077f122d52671412a2ab5d992d535712cc154002/src/slave/containerizer/mesos/containerizer.cpp#L1930-L1958]
> by the agent before spawning the containerizer launcher process, because the
> process was running back then.
> The issue can be reproduced using the following test (pseudocode):
> {code:java}
> launchTask("sleep 1000")
> parentContainerId = containerizer.containers().begin()
> outputs = []
> for i in range(10):
> ContainerId containerId
> containerId.parent = parentContainerId
> containerId.id = UUID.random()
> LAUNCH_NESTED_CONTAINER_SESSION(containerId, "echo echo")
> response = ATTACH_CONTAINER_OUTPUT(containerId)
> outputs.append(response.reader)
> for output in outputs:
> stdout, stderr = getProcessIOData(output)
> assert("echo" == stdout + stderr){code}
> When we start the very first nested container, `getMountNamespaceTarget()`
> returns a PID of the task (`sleep 1000`), because it's the only process whose
> `mnt` namespace differs from the parent container. This nested container
> becomes a child of PID 1 process, which is also a parent of the command
> executor. It's not an executor's child! It can be seen in attached
> `pstree.png`.
> When we start a second nested container, `getMountNamespaceTarget()` might
> return PID of the previous nested container (`echo echo`) instead of the
> task's PID (`sleep 1000`). It happens because the first nested container
> entered `mnt` namespace of the task. Then, the containerizer launcher
> ("nanny" process) attempts to enter `mnt` namespace using the PID of a
> terminated process, so we get this error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
