James Peach created MESOS-9300:
----------------------------------
Summary: XFS isolator can mislabel project IDs on persistence
volumes.
Key: MESOS-9300
URL: https://issues.apache.org/jira/browse/MESOS-9300
Project: Mesos
Issue Type: Bug
Components: agent
Reporter: James Peach
Assignee: James Peach
What happens here is that we are erroneously applying the sandbox's project ID
to the persistent volume.
First, the filesystem/linux isolator bind mounts the persistent volume into the
sandbox:
{noformat}
I1003 06:49:21.907644 2812466 linux.cpp:593] Mounting
'/srv/mesos/work/volumes/roles/pie.mobius/21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f'
to
'/srv/mesos/work/slaves/909cff92-8e17-41bf-a251-9b5eb6186c35-S0/frameworks/363e6d80-8c38-46cf-815f-2fbf60a62628-0309/executors/mobius-mloop-1538549013_438156792-v2-shared-volume.pod1.writer-job.0.e93hs3uips2i9_1/runs/9e5770a7-9f78-46dc-9264-3e80be0e40cc/shared'
for persistent volume disk(allocated: pie.mobius)(reservations:
[(DYNAMIC,pie.mobius,jarvis-principal,\{podInstance: e93hs3uips2i9, pod: pod1,
service:
mobius-mloop-1538549013_438156792-v2-shared-volume})])[21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f:shared]<SHARED>:1
of container 9e5770a7-9f78-46dc-9264-3e80be0e40cc
{noformat}
Next, the `disk/xfs` isolator assigns a project ID to the sandbox:
{noformat}
I1003 06:49:21.920197 2812452 disk.cpp:402] Assigned project 6806 to
'/srv/mesos/work/slaves/909cff92-8e17-41bf-a251-9b5eb6186c35-S0/frameworks/363e6d80-8c38-46cf-815f-2fbf60a62628-0309/executors/mobius-mloop-1538549013_438156792-v2-shared-volume.pod1.writer-job.0.e93hs3uips2i9_1/runs/9e5770a7-9f78-46dc-9264-3e80be0e40cc'
{noformat}
Note, that when this happens, the isolator recursively applies the project ID
to the contents of the sandbox. It doesn't follow symlinks or cross devices
when it does this, but on Linux, a bind mount would not trigger either of these
conditions.
Finally, the `disk/xfs` isolator tries to assign a project ID to the persistent
volume as it is used by the task:
{noformat}
F1003 06:49:21.920577 2812452 disk.cpp:532] Check failed:
scheduledProjects.contains(projectId.get()) untracked project ID 6806 for
volume ID 21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f on
/srv/mesos/work/volumes/roles/pie.mobius/21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f
{noformat}
This check fails, because if the persistent volume has a project ID, we expect
that is had already be scheduled for reclaimation. However, it's project ID is
the one we assigned to the sandbox. We don't scheduled the ssandbox for
reclaimation until cleanup, so (fortunately) the invariant check triggers.
So, apart from triggering the CHECK, the root cause of this is that we are
altering the project ID of the persistent volume, which permanently
misattributes the corresponding quote.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
