Repository: mesos Updated Branches: refs/heads/1.1.x 2f910790f -> 3da5fa963
Lazily unmount persistent volumes in MesosContainerizer. Use MNT_DETACH when unmounting persistent volumes in Linux filesystem isolator to workaround an issue of incorrect handling of container destroy failures. Currently, if isolator cleanup returns a failure, the slave will treat the container as terminated, and will schedule the cleanup of the container's sandbox. Since the mount hasn't been removed in the sandbox (e.g., due to EBUSY), that'll result in data in the persistent volume being incorrectly deleted. Use MNT_DETACH so that the mount point in the sandbox will be removed immediately. See MESOS-7366 for more details. Review: https://reviews.apache.org/r/58278 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/389b5aaf Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/389b5aaf Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/389b5aaf Branch: refs/heads/1.1.x Commit: 389b5aaf8e30ef47546bb419ed2cbd9e616ea8a2 Parents: 2f91079 Author: Jie Yu <[email protected]> Authored: Fri Apr 7 16:33:53 2017 -0700 Committer: Jie Yu <[email protected]> Committed: Tue Apr 11 16:42:55 2017 -0700 ---------------------------------------------------------------------- .../mesos/isolators/filesystem/linux.cpp | 22 ++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/mesos/blob/389b5aaf/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp ---------------------------------------------------------------------- diff --git a/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp b/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp index df16b8f..0c57757 100644 --- a/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp +++ b/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp @@ -800,6 +800,8 @@ Future<Nothing> LinuxFilesystemIsolatorProcess::cleanup( return Failure("Failed to get mount table: " + table.error()); } + vector<string> unmountErrors; + // Reverse unmount order to handle nested mount points. foreach (const fs::MountInfoTable::Entry& entry, adaptor::reverse(table->entries)) { @@ -810,15 +812,31 @@ Future<Nothing> LinuxFilesystemIsolatorProcess::cleanup( LOG(INFO) << "Unmounting volume '" << entry.target << "' for container " << containerId; - Try<Nothing> unmount = fs::unmount(entry.target); + // TODO(jieyu): Use MNT_DETACH here to workaround an issue of + // incorrect handling of container destroy failures. Currently, + // if isolator cleanup returns a failure, the slave will treat + // the container as terminated, and will schedule the cleanup of + // the container's sandbox. Since the mount hasn't been removed + // in the sandbox, that'll result in data in the persistent + // volume being incorrectly deleted. Use MNT_DETACH here so that + // the mount point in the sandbox will be removed immediately. + // See MESOS-7366 for more details. + Try<Nothing> unmount = fs::unmount(entry.target, MNT_DETACH); if (unmount.isError()) { - return Failure( + // NOTE: Instead of short circuit, we try to perform as many + // unmount as possible. We'll accumulate the errors together + // in the end. + unmountErrors.push_back( "Failed to unmount volume '" + entry.target + "': " + unmount.error()); } } } + if (!unmountErrors.empty()) { + return Failure(strings::join(", ", unmountErrors)); + } + return Nothing(); }
