[
https://issues.apache.org/jira/browse/MESOS-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932151#comment-16932151
]
Qian Zhang edited comment on MESOS-9966 at 9/18/19 7:19 AM:
------------------------------------------------------------
Actually I have reproduced this issue, just launch Mesos agent with
`--gc_non_executor_container_sandboxes` enabled, launch a task group, and then
restart Mesos agent which will crash.
was (Author: qianzhang):
Actually I have reproduced this issue, just launch Mesos agent with
`--gc_non_executor_container_sandboxes` enabled, launched a task group, and
then restart Mesos agent which will crash.
> Agent crashes when trying to destroy orphaned nested container if root
> container is orphaned as well
> ----------------------------------------------------------------------------------------------------
>
> Key: MESOS-9966
> URL: https://issues.apache.org/jira/browse/MESOS-9966
> Project: Mesos
> Issue Type: Bug
> Components: containerization
> Affects Versions: 1.7.3
> Reporter: Jan Schlicht
> Assignee: Qian Zhang
> Priority: Major
>
> Noticed an agent crash-looping when trying to recover. It recognized a
> container and its nested container as orphaned. When trying to destroy the
> nested container, the agent crashes. Probably when trying to [get the sandbox
> path of the root
> container|https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/containerizer.cpp#L2966].
> {noformat}
> 2019-09-09 05:04:26: I0909 05:04:26.382326 89950 linux_launcher.cpp:286]
> Recovering Linux launcher
> 2019-09-09 05:04:26: I0909 05:04:26.383162 89950 linux_launcher.cpp:331] Not
> recovering cgroup mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.383199 89950 linux_launcher.cpp:343]
> Recovered container
> a127917b-96fe-4100-b73d-5f876ce9ffc1.9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> 2019-09-09 05:04:26: I0909 05:04:26.383216 89950 linux_launcher.cpp:331] Not
> recovering cgroup
> mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.383229 89950 linux_launcher.cpp:343]
> Recovered container 2ee154e2-3cc4-420a-99fb-065e740f3091
> 2019-09-09 05:04:26: I0909 05:04:26.383237 89950 linux_launcher.cpp:343]
> Recovered container a127917b-96fe-4100-b73d-5f876ce9ffc1
> 2019-09-09 05:04:26: I0909 05:04:26.383249 89950 linux_launcher.cpp:343]
> Recovered container
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436
> 2019-09-09 05:04:26: I0909 05:04:26.383260 89950 linux_launcher.cpp:331] Not
> recovering cgroup mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.383271 89950 linux_launcher.cpp:331] Not
> recovering cgroup
> mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.383280 89950 linux_launcher.cpp:437]
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436 is
> a known orphaned container
> 2019-09-09 05:04:26: I0909 05:04:26.383289 89950 linux_launcher.cpp:437]
> a127917b-96fe-4100-b73d-5f876ce9ffc1 is a known orphaned container
> 2019-09-09 05:04:26: I0909 05:04:26.383296 89950 linux_launcher.cpp:437]
> 2ee154e2-3cc4-420a-99fb-065e740f3091 is a known orphaned container
> 2019-09-09 05:04:26: I0909 05:04:26.383304 89950 linux_launcher.cpp:437]
> a127917b-96fe-4100-b73d-5f876ce9ffc1.9783e2bb-7c2e-4930-9d39-4225bb6f1b97 is
> a known orphaned container
> 2019-09-09 05:04:26: I0909 05:04:26.383414 89950 containerizer.cpp:1092]
> Recovering isolators
> 2019-09-09 05:04:26: I0909 05:04:26.385931 89977 memory.cpp:478] Started
> listening for OOM events for container a127917b-96fe-4100-b73d-5f876ce9ffc1
> 2019-09-09 05:04:26: I0909 05:04:26.386118 89977 memory.cpp:590] Started
> listening on 'low' memory pressure events for container
> a127917b-96fe-4100-b73d-5f876ce9ffc1
> 2019-09-09 05:04:26: I0909 05:04:26.386152 89977 memory.cpp:590] Started
> listening on 'medium' memory pressure events for container
> a127917b-96fe-4100-b73d-5f876ce9ffc1
> 2019-09-09 05:04:26: I0909 05:04:26.386175 89977 memory.cpp:590] Started
> listening on 'critical' memory pressure events for container
> a127917b-96fe-4100-b73d-5f876ce9ffc1
> 2019-09-09 05:04:26: I0909 05:04:26.386227 89977 memory.cpp:478] Started
> listening for OOM events for container 2ee154e2-3cc4-420a-99fb-065e740f3091
> 2019-09-09 05:04:26: I0909 05:04:26.386248 89977 memory.cpp:590] Started
> listening on 'low' memory pressure events for container
> 2ee154e2-3cc4-420a-99fb-065e740f3091
> 2019-09-09 05:04:26: I0909 05:04:26.386270 89977 memory.cpp:590] Started
> listening on 'medium' memory pressure events for container
> 2ee154e2-3cc4-420a-99fb-065e740f3091
> 2019-09-09 05:04:26: I0909 05:04:26.386376 89977 memory.cpp:590] Started
> listening on 'critical' memory pressure events for container
> 2ee154e2-3cc4-420a-99fb-065e740f3091
> 2019-09-09 05:04:26: I0909 05:04:26.386694 89921 containerizer.cpp:1131]
> Recovering provisioner
> 2019-09-09 05:04:26: I0909 05:04:26.388226 90010 metadata_manager.cpp:286]
> Successfully loaded 64 Docker images
> 2019-09-09 05:04:26: I0909 05:04:26.388420 89932 provisioner.cpp:494]
> Provisioner recovery complete
> 2019-09-09 05:04:26: I0909 05:04:26.388530 90003 containerizer.cpp:1203]
> Cleaning up orphan container
> a127917b-96fe-4100-b73d-5f876ce9ffc1.9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> 2019-09-09 05:04:26: I0909 05:04:26.388562 90003 containerizer.cpp:2520]
> Destroying container
> a127917b-96fe-4100-b73d-5f876ce9ffc1.9783e2bb-7c2e-4930-9d39-4225bb6f1b97 in
> RUNNING state
> 2019-09-09 05:04:26: I0909 05:04:26.388576 90003 containerizer.cpp:3187]
> Transitioning the state of container
> a127917b-96fe-4100-b73d-5f876ce9ffc1.9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> from RUNNING to DESTROYING
> 2019-09-09 05:04:26: I0909 05:04:26.388640 90003 containerizer.cpp:1203]
> Cleaning up orphan container a127917b-96fe-4100-b73d-5f876ce9ffc1
> 2019-09-09 05:04:26: I0909 05:04:26.388650 90003 containerizer.cpp:2520]
> Destroying container a127917b-96fe-4100-b73d-5f876ce9ffc1 in RUNNING state
> 2019-09-09 05:04:26: I0909 05:04:26.388659 90003 containerizer.cpp:3187]
> Transitioning the state of container a127917b-96fe-4100-b73d-5f876ce9ffc1
> from RUNNING to DESTROYING
> 2019-09-09 05:04:26: I0909 05:04:26.388689 90003 containerizer.cpp:1203]
> Cleaning up orphan container
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436
> 2019-09-09 05:04:26: I0909 05:04:26.388698 90003 containerizer.cpp:2520]
> Destroying container
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436 in
> RUNNING state
> 2019-09-09 05:04:26: I0909 05:04:26.388706 90003 containerizer.cpp:3187]
> Transitioning the state of container
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436
> from RUNNING to DESTROYING
> 2019-09-09 05:04:26: I0909 05:04:26.388720 90003 containerizer.cpp:1203]
> Cleaning up orphan container 2ee154e2-3cc4-420a-99fb-065e740f3091
> 2019-09-09 05:04:26: I0909 05:04:26.388729 90003 containerizer.cpp:2520]
> Destroying container 2ee154e2-3cc4-420a-99fb-065e740f3091 in RUNNING state
> 2019-09-09 05:04:26: I0909 05:04:26.388737 90003 containerizer.cpp:3187]
> Transitioning the state of container 2ee154e2-3cc4-420a-99fb-065e740f3091
> from RUNNING to DESTROYING
> 2019-09-09 05:04:26: I0909 05:04:26.388783 90003 containerizer.cpp:3026]
> Container
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436 has
> exited
> 2019-09-09 05:04:26: I0909 05:04:26.388837 89929 linux_launcher.cpp:576]
> Asked to destroy container
> a127917b-96fe-4100-b73d-5f876ce9ffc1.9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> 2019-09-09 05:04:26: I0909 05:04:26.388904 89929 linux_launcher.cpp:618]
> Destroying cgroup
> '/sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97'
> 2019-09-09 05:04:26: I0909 05:04:26.389147 89929 linux_launcher.cpp:576]
> Asked to destroy container
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436
> 2019-09-09 05:04:26: I0909 05:04:26.389173 89929 linux_launcher.cpp:618]
> Destroying cgroup
> '/sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436'
> 2019-09-09 05:04:26: I0909 05:04:26.389261 89947 cgroups.cpp:2854] Freezing
> cgroup
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> 2019-09-09 05:04:26: I0909 05:04:26.389269 89948 cgroups.cpp:2854] Freezing
> cgroup
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.389454 89953 cgroups.cpp:2854] Freezing
> cgroup
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436
> 2019-09-09 05:04:26: I0909 05:04:26.389530 89956 cgroups.cpp:1242]
> Successfully froze cgroup
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97/mesos
> after 166912ns
> 2019-09-09 05:04:26: I0909 05:04:26.389582 89965 cgroups.cpp:2854] Freezing
> cgroup
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.389605 89937 cgroups.cpp:1242]
> Successfully froze cgroup
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> after 269056ns
> 2019-09-09 05:04:26: I0909 05:04:26.389679 89964 cgroups.cpp:1242]
> Successfully froze cgroup
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436
> after 145920ns
> 2019-09-09 05:04:26: I0909 05:04:26.389761 89963 cgroups.cpp:2872] Thawing
> cgroup
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.389888 89969 cgroups.cpp:1242]
> Successfully froze cgroup
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436/mesos
> after 219136ns
> 2019-09-09 05:04:26: I0909 05:04:26.389904 89974 cgroups.cpp:2872] Thawing
> cgroup
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436
> 2019-09-09 05:04:26: I0909 05:04:26.390111 89980 cgroups.cpp:2872] Thawing
> cgroup
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.390151 89987 cgroups.cpp:1271]
> Successfully thawed cgroup
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436
> after 128us
> 2019-09-09 05:04:26: I0909 05:04:26.390199 89980 cgroups.cpp:1271]
> Successfully thawed cgroup
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436/mesos
> after 47104ns
> 2019-09-09 05:04:26: I0909 05:04:26.390290 89956 cgroups.cpp:2872] Thawing
> cgroup
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> 2019-09-09 05:04:26: I0909 05:04:26.390463 89983 linux_launcher.cpp:650]
> Destroying cgroup
> '/sys/fs/cgroup/systemd/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436'
> 2019-09-09 05:04:26: I0909 05:04:26.392710 89995 cgroups.cpp:1271]
> Successfully thawed cgroup
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> after 2.397184ms
> 2019-09-09 05:04:26: I0909 05:04:26.394942 89976 containerizer.cpp:2812]
> Checkpointing termination state to nested container's runtime directory
> '/var/run/mesos/containers/2ee154e2-3cc4-420a-99fb-065e740f3091/containers/49fe2bf9-17af-415f-92b6-92a4db619436/termination'
> 2019-09-09 05:04:26: mesos-agent:
> /pkg/src/mesos/3rdparty/stout/include/stout/option.hpp:119: T&
> Option<T>::get() & [with T = std::basic_string<char>]: Assertion `isSome()'
> failed.
> 2019-09-09 05:04:26: *** Aborted at 1568019866 (unix time) try "date -d
> @1568019866" if you are using GNU date ***
> 2019-09-09 05:04:26: PC: @ 0x7f8229cc02c7 __GI_raise
> 2019-09-09 05:04:26: *** SIGABRT (@0x15f32) received by PID 89906 (TID
> 0x7f820c148700) from PID 89906; stack trace: ***
> 2019-09-09 05:04:26: @ 0x7f822a066680 (unknown)
> 2019-09-09 05:04:26: @ 0x7f8229cc02c7 __GI_raise
> 2019-09-09 05:04:26: @ 0x7f8229cc19b8 __GI_abort
> 2019-09-09 05:04:26: @ 0x7f8229cb90e6 __assert_fail_base
> 2019-09-09 05:04:26: @ 0x7f8229cb9192 __GI___assert_fail
> 2019-09-09 05:04:26: @ 0x7f822d306e33 _ZNR6OptionISsE3getEv.part.137
> 2019-09-09 05:04:26: @ 0x7f822d317c4f
> mesos::internal::slave::MesosContainerizerProcess::______destroy()
> 2019-09-09 05:04:26: I0909 05:04:26.418018 89974 token_retriever.cpp:422]
> Successfuly acquired token with expiration set at 2019-09-09 09:09:26+00:00
> 2019-09-09 05:04:26: I0909 05:04:26.418375 89974 token_retriever.cpp:280]
> Scheduling token refresh tu run at 2019-09-09 09:08:56.041828249+00:00
> 2019-09-09 05:04:26: @ 0x7f822de72fc1 process::ProcessBase::consume()
> 2019-09-09 05:04:26: @ 0x7f822de899ac process::ProcessManager::resume()
> 2019-09-09 05:04:26: @ 0x7f822de8f466
> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
> 2019-09-09 05:04:26: @ 0x7f822a840070 (unknown)
> 2019-09-09 05:04:26: @ 0x7f822a05edd5 start_thread
> 2019-09-09 05:04:26: @ 0x7f8229d88bfd __clone
> 2019-09-09 05:04:26: dcos-mesos-slave.service: main process exited,
> code=killed, status=6/ABRT
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
