[
https://issues.apache.org/jira/browse/MESOS-10126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118250#comment-17118250
]
Qian Zhang edited comment on MESOS-10126 at 5/29/20, 12:09 PM:
---------------------------------------------------------------
Master branch:
commit 2845330fbd78a80fb7e71c6101724655fa254392
Author: Qian Zhang
Date: Fri May 15 10:23:51 2020 +0800
Erased `Info` struct before unmouting volumes in Docker volume isolator.
Currently when `DockerVolumeIsolatorProcess::cleanup()` is called, we will
unmount the volume first, and if the unmount operation fails we will NOT
erase the container's `Info` struct from `infos`. This is problematic
because the remaining `Info` in `infos` will cause the reference count of
the volume is greater than 0, but actually the volume is not being used by
any containers. That means we may never get a chance to unmount this volume
on this agent, furthermore if it is an EBS volume, it cannot be used by any
tasks launched on any other agents since a EBS volume can only be attached
to one node at a time. The only workaround would manually unmount the volume.
So in this patch `DockerVolumeIsolatorProcess::cleanup()` is updated to erase
container's `Info` struct before unmounting volumes.
Review: [https://reviews.apache.org/r/72516]
commit b7c3da5a28fb46b4517d52872aec504fff098967
Author: Qian Zhang
Date: Sun May 17 23:30:38 2020 +0800
Added a test `ROOT_CommandTaskNoRootfsWithUnmountVolumeFailure`.
Review: [https://reviews.apache.org/r/72523]
was (Author: qianzhang):
commit 2845330fbd78a80fb7e71c6101724655fa254392
Author: Qian Zhang
Date: Fri May 15 10:23:51 2020 +0800
Erased `Info` struct before unmouting volumes in Docker volume isolator.
Currently when `DockerVolumeIsolatorProcess::cleanup()` is called, we will
unmount the volume first, and if the unmount operation fails we will NOT
erase the container's `Info` struct from `infos`. This is problematic
because the remaining `Info` in `infos` will cause the reference count of
the volume is greater than 0, but actually the volume is not being used by
any containers. That means we may never get a chance to unmount this volume
on this agent, furthermore if it is an EBS volume, it cannot be used by any
tasks launched on any other agents since a EBS volume can only be attached
to one node at a time. The only workaround would manually unmount the volume.
So in this patch `DockerVolumeIsolatorProcess::cleanup()` is updated to erase
container's `Info` struct before unmounting volumes.
Review: [https://reviews.apache.org/r/72516]
commit b7c3da5a28fb46b4517d52872aec504fff098967
Author: Qian Zhang
Date: Sun May 17 23:30:38 2020 +0800
Added a test `ROOT_CommandTaskNoRootfsWithUnmountVolumeFailure`.
Review: [https://reviews.apache.org/r/72523]
> Docker volume isolator needs to clean up the `info` struct regardless the
> result of unmount operation
> -----------------------------------------------------------------------------------------------------
>
> Key: MESOS-10126
> URL: https://issues.apache.org/jira/browse/MESOS-10126
> Project: Mesos
> Issue Type: Bug
> Components: containerization
> Reporter: Qian Zhang
> Assignee: Qian Zhang
> Priority: Critical
> Fix For: 1.4.4, 1.5.4, 1.6.3, 1.8.2, 1.9.1, 1.7.4, 1.11.0, 1.10.1
>
>
> Currently when
> [DockerVolumeIsolatorProcess::cleanup()|https://github.com/apache/mesos/blob/1.9.0/src/slave/containerizer/mesos/isolators/docker/volume/isolator.cpp#L610]
> is called, we will unmount the volume first, but if the unmount operation
> fails we will not remove the container's checkpoint directory and NOT erase
> the container's `info` struct from `infos`. This is problematic, because the
> remaining `info` in the `infos` will cause the reference count of the volume
> is larger than 0, but actually the volume is not being used by any
> containers. And next time when another container using this volume is
> destroyed, we will NOT unmount the volume since its reference count will be
> larger than 1 (see
> [here|https://github.com/apache/mesos/blob/1.9.0/src/slave/containerizer/mesos/isolators/docker/volume/isolator.cpp#L631:L651]
> for details) which should be 2, so we will never have chance to unmount this
> volume.
> We have this issue since Mesos 1.0.0 release when Docker volume isolator was
> introduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
