Joseph Wu created MESOS-9899:
Summary: Using a symlink as the agent's work directory results in
non-removal of persistent volume mounts.
Key: MESOS-9899
URL: https://issues.apache.org/jira/browse/MESOS-9899
Project: Mesos
Issue Type: Bug
Affects Versions: 1.8.0, 1.7.0, 1.6.0
Reporter: Joseph Wu
The directory layout of the agent's information places created persistent
volumes under the agent's {{--work_dir}}:
{code}
// root ('--work_dir' flag)
// |-- volumes
// | |-- roles
// | |--
// | |-- (persistent volume)
{code}
When these persistent volumes are used, they will (on Linux) generally be
mounted underneath the sandbox directory (also located under {{--work_dir}}).
Upon termination of use, persistent volumes are unmounted by reading the mount
table, and checking if any mount targets are under the sandbox:
{code}
// Reverse unmount order to handle nested mount points.
foreach (const fs::MountInfoTable::Entry& entry,
adaptor::reverse(table->entries)) {
// NOTE: All persistent volumes are mounted at targets under the
// container's work directory. We unmount all the persistent
// volumes before unmounting the sandbox/work directory mount.
if (strings::startsWith(entry.target, sandbox)) {
LOG(INFO) << "Unmounting volume '" << entry.target
<< "' for container " << containerId;
{code}
---
However, when an agent's work directory is placed under a symlink, the same
code above might not find any persistent volumes to remove. This is because
the mount table shows the real location on disk, but the sandbox expects the
symlinked location.
For example, suppose:
* The {{--work_dir}} is {{/var/run/mesos}}.
* {{/var/run/mesos}} is a symlink pointing to {{/tmp/link}}.
The agent will create sandboxes under paths like
{{/var/run/mesos/slave/.../framework/.../...}}. The mount table however, will
show mount targets like {{/tmp/link/slave/.../framework/.../...}}. Since the
mount table target does not start with the sandbox path, the
{{filesystem/linux}} isolator will not find any persistent volumes to clean up.
The agent's garbage collector will also fail here, because it tries to unmount
any persistent volumes under the agent's work directory.
{code}
foreach (const fs::MountInfoTable::Entry& entry,
adaptor::reverse(mountTable->entries)) {
// Ignore mounts whose targets are not under `workDir`.
if (!strings::startsWith(
path::join(entry.target, ""),
path::join(_workDir, ""))) {
continue;
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
