James Peach created MESOS-9300: ---------------------------------- Summary: XFS isolator can mislabel project IDs on persistence volumes. Key: MESOS-9300 URL: https://issues.apache.org/jira/browse/MESOS-9300 Project: Mesos Issue Type: Bug Components: agent Reporter: James Peach Assignee: James Peach
What happens here is that we are erroneously applying the sandbox's project ID to the persistent volume. First, the filesystem/linux isolator bind mounts the persistent volume into the sandbox: {noformat} I1003 06:49:21.907644 2812466 linux.cpp:593] Mounting '/srv/mesos/work/volumes/roles/pie.mobius/21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f' to '/srv/mesos/work/slaves/909cff92-8e17-41bf-a251-9b5eb6186c35-S0/frameworks/363e6d80-8c38-46cf-815f-2fbf60a62628-0309/executors/mobius-mloop-1538549013_438156792-v2-shared-volume.pod1.writer-job.0.e93hs3uips2i9_1/runs/9e5770a7-9f78-46dc-9264-3e80be0e40cc/shared' for persistent volume disk(allocated: pie.mobius)(reservations: [(DYNAMIC,pie.mobius,jarvis-principal,\{podInstance: e93hs3uips2i9, pod: pod1, service: mobius-mloop-1538549013_438156792-v2-shared-volume})])[21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f:shared]<SHARED>:1 of container 9e5770a7-9f78-46dc-9264-3e80be0e40cc {noformat} Next, the `disk/xfs` isolator assigns a project ID to the sandbox: {noformat} I1003 06:49:21.920197 2812452 disk.cpp:402] Assigned project 6806 to '/srv/mesos/work/slaves/909cff92-8e17-41bf-a251-9b5eb6186c35-S0/frameworks/363e6d80-8c38-46cf-815f-2fbf60a62628-0309/executors/mobius-mloop-1538549013_438156792-v2-shared-volume.pod1.writer-job.0.e93hs3uips2i9_1/runs/9e5770a7-9f78-46dc-9264-3e80be0e40cc' {noformat} Note, that when this happens, the isolator recursively applies the project ID to the contents of the sandbox. It doesn't follow symlinks or cross devices when it does this, but on Linux, a bind mount would not trigger either of these conditions. Finally, the `disk/xfs` isolator tries to assign a project ID to the persistent volume as it is used by the task: {noformat} F1003 06:49:21.920577 2812452 disk.cpp:532] Check failed: scheduledProjects.contains(projectId.get()) untracked project ID 6806 for volume ID 21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f on /srv/mesos/work/volumes/roles/pie.mobius/21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f {noformat} This check fails, because if the persistent volume has a project ID, we expect that is had already be scheduled for reclaimation. However, it's project ID is the one we assigned to the sandbox. We don't scheduled the ssandbox for reclaimation until cleanup, so (fortunately) the invariant check triggers. So, apart from triggering the CHECK, the root cause of this is that we are altering the project ID of the persistent volume, which permanently misattributes the corresponding quote. -- This message was sent by Atlassian JIRA (v7.6.3#76005)