Gilbert Song created MESOS-8280:
-----------------------------------
Summary: Mesos Containerizer GC should set 'layers' after
checkpointing layer ids in provisioner.
Key: MESOS-8280
URL: https://issues.apache.org/jira/browse/MESOS-8280
Project: Mesos
Issue Type: Bug
Components: image-gc, provisioner
Reporter: Gilbert Song
Priority: Critical
{noformat}
11111
222222
333333
444444
11111
222222
333333
444444
I1129 23:24:45.469543 6592 registry_puller.cpp:395] Extracting layer tar ball
'/tmp/mesos/store/docker/staging/MVgVC7/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
to rootfs
'/tmp/mesos/store/docker/staging/MVgVC7/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e/rootfs.overlay'
I1129 23:24:45.473287 6592 registry_puller.cpp:395] Extracting layer tar ball
'/tmp/mesos/store/docker/staging/MVgVC7/sha256:b56ae66c29370df48e7377c8f9baa744a3958058a766793f821dadcb144a4647
to rootfs
'/tmp/mesos/store/docker/staging/MVgVC7/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3/rootfs.overlay'
I1129 23:24:45.582002 6594 registry_puller.cpp:395] Extracting layer tar ball
'/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
to rootfs
'/tmp/mesos/store/docker/staging/6Zbc17/e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6/rootfs.overlay'
I1129 23:24:45.589404 6595 metadata_manager.cpp:167] Successfully cached image
'alpine'
I1129 23:24:45.590204 6594 registry_puller.cpp:395] Extracting layer tar ball
'/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
to rootfs
'/tmp/mesos/store/docker/staging/6Zbc17/be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e/rootfs.overlay'
I1129 23:24:45.595190 6594 registry_puller.cpp:395] Extracting layer tar ball
'/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
to rootfs
'/tmp/mesos/store/docker/staging/6Zbc17/53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f/rootfs.overlay'
I1129 23:24:45.599500 6594 registry_puller.cpp:395] Extracting layer tar ball
'/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
to rootfs
'/tmp/mesos/store/docker/staging/6Zbc17/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721/rootfs.overlay'
I1129 23:24:45.602047 6597 provisioner.cpp:506] Provisioning image rootfs
'/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/rootfses/b5d48445-848d-4274-a4f8-e909351ebc35'
for container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 using
overlay backend
I1129 23:24:45.602751 6594 registry_puller.cpp:395] Extracting layer tar ball
'/tmp/mesos/store/docker/staging/6Zbc17/sha256:1db09adb5ddd7f1a07b6d585a7db747a51c7bd17418d47e91f901bdf420abd66
to rootfs
'/tmp/mesos/store/docker/staging/6Zbc17/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16/rootfs.overlay'
I1129 23:24:45.603054 6596 overlay.cpp:168] Created symlink
'/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/scratch/b5d48445-848d-4274-a4f8-e909351ebc35/links'
-> '/tmp/xAWQ8y'
I1129 23:24:45.604398 6596 overlay.cpp:196] Provisioning image rootfs with
overlayfs:
'lowerdir=/tmp/xAWQ8y/1:/tmp/xAWQ8y/0,upperdir=/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/scratch/b5d48445-848d-4274-a4f8-e909351ebc35/upperdir,workdir=/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/scratch/b5d48445-848d-4274-a4f8-e909351ebc35/workdir'
I1129 23:24:45.607802 6594 registry_puller.cpp:395] Extracting layer tar ball
'/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
to rootfs
'/tmp/mesos/store/docker/staging/6Zbc17/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229/rootfs.overlay'
I1129 23:24:45.612139 6594 registry_puller.cpp:395] Extracting layer tar ball
'/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
to rootfs
'/tmp/mesos/store/docker/staging/6Zbc17/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/rootfs.overlay'
I1129 23:24:45.612253 6593 containerizer.cpp:1369] Checkpointed
ContainerConfig at
'/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/config'
I1129 23:24:45.612298 6593 containerizer.cpp:2926] Transitioning the state of
container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from
PROVISIONING to PREPARING
I1129 23:24:45.625658 6596 containerizer.cpp:1838] Launching
'mesos-containerizer' with flags '--help="false"
--launch_info="{"clone_namespaces":[131072],"command":{"shell":true,"value":"sleep
1"},"environment":{"variables":[{"name":"MESOS_SANDBOX","type":"VALUE","value":"\/mnt\/mesos\/sandbox"},{"name":"PATH","type":"VALUE","value":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"},{"name":"MESOS_CONTAINER_IP","type":"VALUE","value":"10.0.2.15"}]},"pre_exec_commands":[{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/vagrant\/mesos\/build\/src\/mesos-containerizer"},{"arguments":["mount","-n","--rbind","\/tmp\/slaves\/65fa58c5-48c6-4998-b336-ceb9bcd2ec43-S0\/frameworks\/55b825d4-c922-46bb-ab8e-4e01abf1a756-0000\/executors\/default-executor\/runs\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/01de09c5-d8e9-412e-8825-a592d2c875e5","\/tmp\/provisioner\/containers\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/01de09c5-d8e9-412e-8825-a592d2c875e5\/backends\/overlay\/rootfses\/b5d48445-848d-4274-a4f8-e909351ebc35\/mnt\/mesos\/sandbox"],"shell":false,"value":"mount"}],"rootfs":"\/tmp\/provisioner\/containers\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/01de09c5-d8e9-412e-8825-a592d2c875e5\/backends\/overlay\/rootfses\/b5d48445-848d-4274-a4f8-e909351ebc35","task_environment":{},"user":"root","working_directory":"\/mnt\/mesos\/sandbox"}"
--pipe_read="12" --pipe_write="15"
--runtime_directory="/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5"
--unshare_namespace_mnt="false"'
I1129 23:24:45.626317 6598 linux_launcher.cpp:438] Launching nested container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 and
cloning with namespaces CLONE_NEWNS
I1129 23:24:45.633211 6598 systemd.cpp:96] Assigned child process '6745' to
'mesos_executors.slice'
I1129 23:24:45.636270 6596 containerizer.cpp:2926] Transitioning the state of
container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from
PREPARING to ISOLATING
I1129 23:24:45.691830 6597 metadata_manager.cpp:167] Successfully cached image
'mesosphere/inky'
I1129 23:24:45.694399 6594 provisioner.cpp:506] Provisioning image rootfs
'/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/backends/overlay/rootfses/1187cc83-a23a-4390-9c28-092a7b7690b5'
for container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 using
overlay backend
I1129 23:24:45.694919 6596 overlay.cpp:168] Created symlink
'/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/backends/overlay/scratch/1187cc83-a23a-4390-9c28-092a7b7690b5/links'
-> '/tmp/GXhXiT'
I1129 23:24:45.695103 6596 overlay.cpp:196] Provisioning image rootfs with
overlayfs:
'lowerdir=/tmp/GXhXiT/6:/tmp/GXhXiT/5:/tmp/GXhXiT/4:/tmp/GXhXiT/3:/tmp/GXhXiT/2:/tmp/GXhXiT/1:/tmp/GXhXiT/0,upperdir=/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/backends/overlay/scratch/1187cc83-a23a-4390-9c28-092a7b7690b5/upperdir,workdir=/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/backends/overlay/scratch/1187cc83-a23a-4390-9c28-092a7b7690b5/workdir'
I1129 23:24:45.696255 6596 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has
no checkpointed layers
I1129 23:24:45.696349 6596 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has
no checkpointed layers
I1129 23:24:45.696449 6593 containerizer.cpp:1369] Checkpointed
ContainerConfig at
'/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/config'
I1129 23:24:45.696506 6593 containerizer.cpp:2926] Transitioning the state of
container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 from
PROVISIONING to PREPARING
I1129 23:24:45.697865 6595 store.cpp:531] Layer
'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained
by image store cache
I1129 23:24:45.697918 6595 store.cpp:531] Layer
'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained
by image store cache
I1129 23:24:45.697968 6595 store.cpp:531] Layer
'53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained
by image store cache
I1129 23:24:45.697999 6595 store.cpp:531] Layer
'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained
by image store cache
I1129 23:24:45.698025 6595 store.cpp:531] Layer
'120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained
by image store cache
I1129 23:24:45.698050 6595 store.cpp:531] Layer
'38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained
by image store cache
I1129 23:24:45.698076 6595 store.cpp:531] Layer
'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained
by image store cache
I1129 23:24:45.698104 6595 store.cpp:531] Layer
'42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained
by image store cache
I1129 23:24:45.698129 6595 store.cpp:531] Layer
'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained
by image store cache
I1129 23:24:45.698894 6596 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has
no checkpointed layers
I1129 23:24:45.698966 6596 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has
no checkpointed layers
I1129 23:24:45.700333 6596 store.cpp:531] Layer
'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained
by image store cache
I1129 23:24:45.700394 6596 store.cpp:531] Layer
'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained
by image store cache
I1129 23:24:45.700412 6596 store.cpp:531] Layer
'53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained
by image store cache
I1129 23:24:45.700428 6596 store.cpp:531] Layer
'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained
by image store cache
I1129 23:24:45.700441 6596 store.cpp:531] Layer
'120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained
by image store cache
I1129 23:24:45.700454 6596 store.cpp:531] Layer
'38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained
by image store cache
I1129 23:24:45.700467 6596 store.cpp:531] Layer
'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained
by image store cache
I1129 23:24:45.700480 6596 store.cpp:531] Layer
'42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained
by image store cache
I1129 23:24:45.700495 6596 store.cpp:531] Layer
'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained
by image store cache
I1129 23:24:45.702491 6594 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has
no checkpointed layers
I1129 23:24:45.702554 6594 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has
no checkpointed layers
I1129 23:24:45.703707 6592 store.cpp:531] Layer
'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained
by image store cache
I1129 23:24:45.703783 6592 store.cpp:531] Layer
'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained
by image store cache
I1129 23:24:45.703812 6592 store.cpp:531] Layer
'53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained
by image store cache
I1129 23:24:45.703816 6592 store.cpp:531] Layer
'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained
by image store cache
I1129 23:24:45.703816 6592 store.cpp:531] Layer
'120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained
by image store cache
I1129 23:24:45.704164 6592 store.cpp:531] Layer
'38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained
by image store cache
I1129 23:24:45.704208 6592 store.cpp:531] Layer
'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained
by image store cache
I1129 23:24:45.704255 6592 store.cpp:531] Layer
'42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained
by image store cache
I1129 23:24:45.704285 6592 store.cpp:531] Layer
'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained
by image store cache
I1129 23:24:45.704814 6595 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has
no checkpointed layers
I1129 23:24:45.704861 6595 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has
no checkpointed layers
I1129 23:24:45.708112 6592 store.cpp:531] Layer
'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained
by image store cache
I1129 23:24:45.708204 6592 store.cpp:531] Layer
'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained
by image store cache
I1129 23:24:45.708238 6592 store.cpp:531] Layer
'53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained
by image store cache
I1129 23:24:45.708264 6592 store.cpp:531] Layer
'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained
by image store cache
I1129 23:24:45.708375 6592 store.cpp:531] Layer
'120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained
by image store cache
I1129 23:24:45.708407 6592 store.cpp:531] Layer
'38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained
by image store cache
I1129 23:24:45.708472 6592 store.cpp:531] Layer
'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained
by image store cache
I1129 23:24:45.708514 6592 store.cpp:531] Layer
'42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained
by image store cache
I1129 23:24:45.708545 6592 store.cpp:531] Layer
'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained
by image store cache
I1129 23:24:45.709048 6596 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has
no checkpointed layers
I1129 23:24:45.709161 6596 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has
no checkpointed layers
I1129 23:24:45.710321 6594 store.cpp:531] Layer
'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained
by image store cache
I1129 23:24:45.710382 6594 store.cpp:531] Layer
'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained
by image store cache
I1129 23:24:45.710412 6594 store.cpp:531] Layer
'53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained
by image store cache
I1129 23:24:45.710436 6594 store.cpp:531] Layer
'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained
by image store cache
I1129 23:24:45.710458 6594 store.cpp:531] Layer
'120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained
by image store cache
I1129 23:24:45.710480 6594 store.cpp:531] Layer
'38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained
by image store cache
I1129 23:24:45.710522 6594 store.cpp:531] Layer
'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained
by image store cache
I1129 23:24:45.710551 6594 store.cpp:531] Layer
'42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained
by image store cache
I1129 23:24:45.710590 6594 store.cpp:531] Layer
'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained
by image store cache
I1129 23:24:45.713176 6594 containerizer.cpp:1838] Launching
'mesos-containerizer' with flags '--help="false"
--launch_info="{"clone_namespaces":[131072],"command":{"shell":true,"value":"sleep
100000"},"environment":{"variables":[{"name":"MESOS_SANDBOX","type":"VALUE","value":"\/mnt\/mesos\/sandbox"},{"name":"HOME","type":"VALUE","value":"\/"},{"name":"PATH","type":"VALUE","value":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"},{"name":"MESOS_CONTAINER_IP","type":"VALUE","value":"10.0.2.15"}]},"pre_exec_commands":[{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/vagrant\/mesos\/build\/src\/mesos-containerizer"},{"arguments":["mount","-n","--rbind","\/tmp\/slaves\/65fa58c5-48c6-4998-b336-ceb9bcd2ec43-S0\/frameworks\/55b825d4-c922-46bb-ab8e-4e01abf1a756-0000\/executors\/default-executor\/runs\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/7be37efd-7ec4-459a-a51a-97f6be99c3c3","\/tmp\/provisioner\/containers\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/7be37efd-7ec4-459a-a51a-97f6be99c3c3\/backends\/overlay\/rootfses\/1187cc83-a23a-4390-9c28-092a7b7690b5\/mnt\/mesos\/sandbox"],"shell":false,"value":"mount"}],"rootfs":"\/tmp\/provisioner\/containers\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/7be37efd-7ec4-459a-a51a-97f6be99c3c3\/backends\/overlay\/rootfses\/1187cc83-a23a-4390-9c28-092a7b7690b5","task_environment":{},"user":"root","working_directory":"\/mnt\/mesos\/sandbox"}"
--pipe_read="13" --pipe_write="14"
--runtime_directory="/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3"
--unshare_namespace_mnt="false"'
I1129 23:24:45.713954 6597 linux_launcher.cpp:438] Launching nested container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 and
cloning with namespaces CLONE_NEWNS
I1129 23:24:45.721781 6597 systemd.cpp:96] Assigned child process '6775' to
'mesos_executors.slice'
I1129 23:24:45.725494 6594 containerizer.cpp:2926] Transitioning the state of
container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 from
PREPARING to ISOLATING
I1129 23:24:45.791635 6595 containerizer.cpp:2926] Transitioning the state of
container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from
ISOLATING to FETCHING
I1129 23:24:45.791880 6591 fetcher.cpp:379] Starting to fetch URIs for
container:
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5,
directory:
/tmp/slaves/65fa58c5-48c6-4998-b336-ceb9bcd2ec43-S0/frameworks/55b825d4-c922-46bb-ab8e-4e01abf1a756-0000/executors/default-executor/runs/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5
I1129 23:24:45.792626 6591 containerizer.cpp:2926] Transitioning the state of
container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from
FETCHING to RUNNING
11111
222222
333333
444444
I1129 23:24:45.807262 6592 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has
no checkpointed layers
I1129 23:24:45.807375 6592 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has
no checkpointed layers
I1129 23:24:45.808658 6591 store.cpp:531] Layer
'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained
by image store cache
I1129 23:24:45.808843 6591 store.cpp:531] Layer
'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained
by image store cache
I1129 23:24:45.808869 6591 store.cpp:531] Layer
'53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained
by image store cache
I1129 23:24:45.808897 6591 store.cpp:531] Layer
'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained
by image store cache
I1129 23:24:45.808962 6591 store.cpp:531] Layer
'120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained
by image store cache
I1129 23:24:45.808990 6591 store.cpp:531] Layer
'38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained
by image store cache
I1129 23:24:45.809012 6591 store.cpp:531] Layer
'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained
by image store cache
I1129 23:24:45.809036 6591 store.cpp:531] Layer
'42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained
by image store cache
I1129 23:24:45.809057 6591 store.cpp:531] Layer
'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained
by image store cache
I1129 23:24:45.893280 6596 containerizer.cpp:2926] Transitioning the state of
container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 from
ISOLATING to FETCHING
I1129 23:24:45.893523 6596 fetcher.cpp:379] Starting to fetch URIs for
container:
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3,
directory:
/tmp/slaves/65fa58c5-48c6-4998-b336-ceb9bcd2ec43-S0/frameworks/55b825d4-c922-46bb-ab8e-4e01abf1a756-0000/executors/default-executor/runs/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3
I1129 23:24:45.894335 6594 containerizer.cpp:2926] Transitioning the state of
container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 from
FETCHING to RUNNING
I1129 23:24:45.902606 6598 process.cpp:3503] Handling HTTP event for process
'slave(1)' with path: '/slave(1)/api/v1/executor'
I1129 23:24:45.903908 6597 process.cpp:3503] Handling HTTP event for process
'slave(1)' with path: '/slave(1)/api/v1'
I1129 23:24:45.904618 6597 process.cpp:3503] Handling HTTP event for process
'slave(1)' with path: '/slave(1)/api/v1'
I1129 23:24:45.908681 6597 http.cpp:1185] HTTP POST for /slave(1)/api/v1 from
10.0.2.15:57620
I1129 23:24:45.909113 6597 http.cpp:1185] HTTP POST for /slave(1)/api/v1 from
10.0.2.15:57622
I1129 23:24:45.909708 6597 http.cpp:2589] Processing WAIT_NESTED_CONTAINER
call for container
'3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5'
I1129 23:24:45.910148 6597 http.cpp:2589] Processing WAIT_NESTED_CONTAINER
call for container
'3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3'
I1129 23:24:45.938350 6596 process.cpp:3503] Handling HTTP event for process
'slave(1)' with path: '/slave(1)/api/v1/executor'
I1129 23:24:45.938781 6596 http.cpp:1185] HTTP POST for
/slave(1)/api/v1/executor from 10.0.2.15:57564
I1129 23:24:45.939141 6596 slave.cpp:4584] Handling status update TASK_RUNNING
(UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework
55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.940809 6591 http.cpp:1185] HTTP POST for
/slave(1)/api/v1/executor from 10.0.2.15:57564
I1129 23:24:45.941130 6591 slave.cpp:4584] Handling status update TASK_RUNNING
(UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework
55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.942906 6591 task_status_update_manager.cpp:328] Received task
status update TASK_RUNNING (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for
task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.943076 6591 task_status_update_manager.cpp:383] Forwarding task
status update TASK_RUNNING (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for
task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to the agent
I1129 23:24:45.943322 6595 slave.cpp:5067] Forwarding the update TASK_RUNNING
(UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework
55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to [email protected]:5050
I1129 23:24:45.943332 6591 task_status_update_manager.cpp:328] Received task
status update TASK_RUNNING (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for
task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.943506 6591 task_status_update_manager.cpp:383] Forwarding task
status update TASK_RUNNING (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for
task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to the agent
I1129 23:24:45.943717 6595 slave.cpp:4960] Task status update manager
successfully handled status update TASK_RUNNING (UUID:
6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework
55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.943905 6595 slave.cpp:5067] Forwarding the update TASK_RUNNING
(UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework
55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to [email protected]:5050
I1129 23:24:45.943905 6595 slave.cpp:4960] Task status update manager
successfully handled status update TASK_RUNNING (UUID:
f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework
55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.992866 6591 task_status_update_manager.cpp:401] Received task
status update acknowledgement (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for
task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.993261 6595 slave.cpp:3868] Task status update manager
successfully handled status update acknowledgement (UUID:
6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework
55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.993968 6593 task_status_update_manager.cpp:401] Received task
status update acknowledgement (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for
task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.994295 6598 slave.cpp:3868] Task status update manager
successfully handled status update acknowledgement (UUID:
f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework
55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
11111
222222
333333
444444
I1129 23:24:46.808684 6592 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has
no checkpointed layers
I1129 23:24:46.808876 6592 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has
no checkpointed layers
I1129 23:24:46.810683 6593 store.cpp:531] Layer
'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained
by image store cache
I1129 23:24:46.810751 6593 store.cpp:531] Layer
'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained
by image store cache
I1129 23:24:46.810781 6593 store.cpp:531] Layer
'53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained
by image store cache
I1129 23:24:46.810808 6593 store.cpp:531] Layer
'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained
by image store cache
I1129 23:24:46.810834 6593 store.cpp:531] Layer
'120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained
by image store cache
I1129 23:24:46.810860 6593 store.cpp:531] Layer
'38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained
by image store cache
I1129 23:24:46.810885 6593 store.cpp:531] Layer
'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained
by image store cache
I1129 23:24:46.810911 6593 store.cpp:531] Layer
'42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained
by image store cache
I1129 23:24:46.810937 6593 store.cpp:531] Layer
'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained
by image store cache
I1129 23:24:47.057590 6596 containerizer.cpp:2775] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has
exited
I1129 23:24:47.057667 6596 containerizer.cpp:2324] Destroying container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 in
RUNNING state
I1129 23:24:47.057695 6596 containerizer.cpp:2926] Transitioning the state of
container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from
RUNNING to DESTROYING
I1129 23:24:47.058082 6596 linux_launcher.cpp:514] Asked to destroy container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5
I1129 23:24:47.059027 6596 linux_launcher.cpp:560] Using freezer to destroy
cgroup
mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5
I1129 23:24:47.060667 6596 cgroups.cpp:3058] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5
I1129 23:24:47.062700 6597 cgroups.cpp:1413] Successfully froze cgroup
/sys/fs/cgroup/freezer/mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5
after 1.838336ms
I1129 23:24:47.064627 6592 cgroups.cpp:3076] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5
I1129 23:24:47.066498 6598 cgroups.cpp:1442] Successfully thawed cgroup
/sys/fs/cgroup/freezer/mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5
after 1.642752ms
I1129 23:24:47.071521 6592 provisioner.cpp:648] Destroying container rootfs at
'/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/rootfses/b5d48445-848d-4274-a4f8-e909351ebc35'
for container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5
I1129 23:24:47.098203 6596 overlay.cpp:296] Removed temporary directory
'/tmp/xAWQ8y' pointed by
'/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/scratch/b5d48445-848d-4274-a4f8-e909351ebc35/links'
I1129 23:24:47.100265 6591 containerizer.cpp:2613] Checkpointing termination
state to nested container's runtime directory
'/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/termination'
I1129 23:24:47.107206 6594 process.cpp:3503] Handling HTTP event for process
'slave(1)' with path: '/slave(1)/api/v1/executor'
I1129 23:24:47.151911 6594 http.cpp:1185] HTTP POST for
/slave(1)/api/v1/executor from 10.0.2.15:57564
I1129 23:24:47.152243 6594 slave.cpp:4584] Handling status update
TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of
framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:47.154391 6594 task_status_update_manager.cpp:328] Received task
status update TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for
task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:47.154578 6594 task_status_update_manager.cpp:383] Forwarding task
status update TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for
task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to the agent
I1129 23:24:47.154810 6593 slave.cpp:5067] Forwarding the update TASK_FINISHED
(UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework
55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to [email protected]:5050
I1129 23:24:47.157249 6593 slave.cpp:4960] Task status update manager
successfully handled status update TASK_FINISHED (UUID:
288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework
55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:47.385977 6592 task_status_update_manager.cpp:401] Received task
status update acknowledgement (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for
task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:47.386334 6592 task_status_update_manager.cpp:538] Cleaning up
status update stream for task 1 of framework
55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:47.387459 6592 slave.cpp:3868] Task status update manager
successfully handled status update acknowledgement (UUID:
288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework
55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:47.387568 6592 slave.cpp:8457] Completing task 1
11111
222222
333333
444444
I1129 23:24:47.818768 6595 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has
no checkpointed layers
I1129 23:24:47.824591 6598 store.cpp:531] Layer
'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained
by image store cache
I1129 23:24:47.824724 6598 store.cpp:531] Layer
'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained
by image store cache
I1129 23:24:47.824753 6598 store.cpp:531] Layer
'53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained
by image store cache
I1129 23:24:47.824767 6598 store.cpp:531] Layer
'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained
by image store cache
I1129 23:24:47.824782 6598 store.cpp:531] Layer
'120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained
by image store cache
I1129 23:24:47.824861 6598 store.cpp:550] Marking layer
'38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' to gc by
renaming
'/tmp/mesos/store/docker/layers/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e'
to
'/tmp/mesos/store/docker/gc/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e.1511997887824822784'
I1129 23:24:47.824918 6598 store.cpp:531] Layer
'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained
by image store cache
I1129 23:24:47.824960 6598 store.cpp:531] Layer
'42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained
by image store cache
I1129 23:24:47.825088 6598 store.cpp:550] Marking layer
'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' to gc by
renaming
'/tmp/mesos/store/docker/layers/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3'
to
'/tmp/mesos/store/docker/gc/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3.1511997887825015040'
I1129 23:24:47.825389 6598 store.cpp:577] Deleting path
'/tmp/mesos/store/docker/gc/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3.1511997887825015040'
I1129 23:24:47.829903 6598 store.cpp:584] Deleted
'/tmp/mesos/store/docker/gc/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3.1511997887825015040'
I1129 23:24:47.829980 6598 store.cpp:577] Deleting path
'/tmp/mesos/store/docker/gc/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e.1511997887824822784'
I1129 23:24:47.830047 6598 store.cpp:584] Deleted
'/tmp/mesos/store/docker/gc/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e.1511997887824822784'
11111
222222
333333
444444
I1129 23:24:48.829519 6595 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has
no checkpointed layers
I1129 23:24:48.831161 6598 store.cpp:531] Layer
'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained
by image store cache
I1129 23:24:48.831225 6598 store.cpp:531] Layer
'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained
by image store cache
I1129 23:24:48.831248 6598 store.cpp:531] Layer
'53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained
by image store cache
I1129 23:24:48.831266 6598 store.cpp:531] Layer
'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained
by image store cache
I1129 23:24:48.831284 6598 store.cpp:531] Layer
'120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained
by image store cache
I1129 23:24:48.831302 6598 store.cpp:531] Layer
'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained
by image store cache
I1129 23:24:48.831321 6598 store.cpp:531] Layer
'42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained
by image store cache
11111
222222
333333
444444
I1129 23:24:49.830904 6597 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has
no checkpointed layers
I1129 23:24:49.832487 6597 store.cpp:531] Layer
'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained
by image store cache
I1129 23:24:49.832584 6597 store.cpp:531] Layer
'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained
by image store cache
I1129 23:24:49.833329 6597 store.cpp:531] Layer
'53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained
by image store cache
I1129 23:24:49.833367 6597 store.cpp:531] Layer
'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained
by image store cache
I1129 23:24:49.833387 6597 store.cpp:531] Layer
'120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained
by image store cache
I1129 23:24:49.833406 6597 store.cpp:531] Layer
'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained
by image store cache
I1129 23:24:49.833425 6597 store.cpp:531] Layer
'42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained
by image store cache
{noformat}
Please neglect the debugging logs like '111111'. To reproduce this issue, just
continuously trigger image gc. The log above was from a scenario that we launch
two nested containers. One sleeps 1 second, another sleep forever.
This is related to this patch:
https://github.com/apache/mesos/commit/e273efe6976434858edb85bbcf367a02e963a467#diff-a3593ed0ebd2b205775f7f04d9b5afe7
The root cause is that we did not set the 'layers' after we checkpoint the
layer ids in provisioner. The log below is the prove:
{noformat}
I1129 23:24:45.698894 6596 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has
no checkpointed layers
I1129 23:24:45.698966 6596 provisioner.cpp:714] Container
3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has
no checkpointed layers
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
