[ 
https://issues.apache.org/jira/browse/MESOS-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Dellar reassigned MESOS-9966:
----------------------------------

    Assignee: Gilbert Song

> Agent crashes when trying to destroy orphaned nested container if root 
> container is orphaned as well
> ----------------------------------------------------------------------------------------------------
>
>                 Key: MESOS-9966
>                 URL: https://issues.apache.org/jira/browse/MESOS-9966
>             Project: Mesos
>          Issue Type: Bug
>          Components: containerization
>    Affects Versions: 1.7.3
>            Reporter: Jan Schlicht
>            Assignee: Gilbert Song
>            Priority: Major
>
> Noticed an agent crash-looping when trying to recover. It recognized a 
> container and its nested container as orphaned. When trying to destroy the 
> nested container, the agent crashes. Probably when trying to [get the sandbox 
> path of the root 
> container|https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/containerizer.cpp#L2966].
> {noformat}
> 2019-09-09 05:04:26: I0909 05:04:26.382326 89950 linux_launcher.cpp:286] 
> Recovering Linux launcher
> 2019-09-09 05:04:26: I0909 05:04:26.383162 89950 linux_launcher.cpp:331] Not 
> recovering cgroup mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.383199 89950 linux_launcher.cpp:343] 
> Recovered container 
> a127917b-96fe-4100-b73d-5f876ce9ffc1.9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> 2019-09-09 05:04:26: I0909 05:04:26.383216 89950 linux_launcher.cpp:331] Not 
> recovering cgroup 
> mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.383229 89950 linux_launcher.cpp:343] 
> Recovered container 2ee154e2-3cc4-420a-99fb-065e740f3091
> 2019-09-09 05:04:26: I0909 05:04:26.383237 89950 linux_launcher.cpp:343] 
> Recovered container a127917b-96fe-4100-b73d-5f876ce9ffc1
> 2019-09-09 05:04:26: I0909 05:04:26.383249 89950 linux_launcher.cpp:343] 
> Recovered container 
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436
> 2019-09-09 05:04:26: I0909 05:04:26.383260 89950 linux_launcher.cpp:331] Not 
> recovering cgroup mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.383271 89950 linux_launcher.cpp:331] Not 
> recovering cgroup 
> mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.383280 89950 linux_launcher.cpp:437] 
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436 is 
> a known orphaned container
> 2019-09-09 05:04:26: I0909 05:04:26.383289 89950 linux_launcher.cpp:437] 
> a127917b-96fe-4100-b73d-5f876ce9ffc1 is a known orphaned container
> 2019-09-09 05:04:26: I0909 05:04:26.383296 89950 linux_launcher.cpp:437] 
> 2ee154e2-3cc4-420a-99fb-065e740f3091 is a known orphaned container
> 2019-09-09 05:04:26: I0909 05:04:26.383304 89950 linux_launcher.cpp:437] 
> a127917b-96fe-4100-b73d-5f876ce9ffc1.9783e2bb-7c2e-4930-9d39-4225bb6f1b97 is 
> a known orphaned container
> 2019-09-09 05:04:26: I0909 05:04:26.383414 89950 containerizer.cpp:1092] 
> Recovering isolators
> 2019-09-09 05:04:26: I0909 05:04:26.385931 89977 memory.cpp:478] Started 
> listening for OOM events for container a127917b-96fe-4100-b73d-5f876ce9ffc1
> 2019-09-09 05:04:26: I0909 05:04:26.386118 89977 memory.cpp:590] Started 
> listening on 'low' memory pressure events for container 
> a127917b-96fe-4100-b73d-5f876ce9ffc1
> 2019-09-09 05:04:26: I0909 05:04:26.386152 89977 memory.cpp:590] Started 
> listening on 'medium' memory pressure events for container 
> a127917b-96fe-4100-b73d-5f876ce9ffc1
> 2019-09-09 05:04:26: I0909 05:04:26.386175 89977 memory.cpp:590] Started 
> listening on 'critical' memory pressure events for container 
> a127917b-96fe-4100-b73d-5f876ce9ffc1
> 2019-09-09 05:04:26: I0909 05:04:26.386227 89977 memory.cpp:478] Started 
> listening for OOM events for container 2ee154e2-3cc4-420a-99fb-065e740f3091
> 2019-09-09 05:04:26: I0909 05:04:26.386248 89977 memory.cpp:590] Started 
> listening on 'low' memory pressure events for container 
> 2ee154e2-3cc4-420a-99fb-065e740f3091
> 2019-09-09 05:04:26: I0909 05:04:26.386270 89977 memory.cpp:590] Started 
> listening on 'medium' memory pressure events for container 
> 2ee154e2-3cc4-420a-99fb-065e740f3091
> 2019-09-09 05:04:26: I0909 05:04:26.386376 89977 memory.cpp:590] Started 
> listening on 'critical' memory pressure events for container 
> 2ee154e2-3cc4-420a-99fb-065e740f3091
> 2019-09-09 05:04:26: I0909 05:04:26.386694 89921 containerizer.cpp:1131] 
> Recovering provisioner
> 2019-09-09 05:04:26: I0909 05:04:26.388226 90010 metadata_manager.cpp:286] 
> Successfully loaded 64 Docker images
> 2019-09-09 05:04:26: I0909 05:04:26.388420 89932 provisioner.cpp:494] 
> Provisioner recovery complete
> 2019-09-09 05:04:26: I0909 05:04:26.388530 90003 containerizer.cpp:1203] 
> Cleaning up orphan container 
> a127917b-96fe-4100-b73d-5f876ce9ffc1.9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> 2019-09-09 05:04:26: I0909 05:04:26.388562 90003 containerizer.cpp:2520] 
> Destroying container 
> a127917b-96fe-4100-b73d-5f876ce9ffc1.9783e2bb-7c2e-4930-9d39-4225bb6f1b97 in 
> RUNNING state
> 2019-09-09 05:04:26: I0909 05:04:26.388576 90003 containerizer.cpp:3187] 
> Transitioning the state of container 
> a127917b-96fe-4100-b73d-5f876ce9ffc1.9783e2bb-7c2e-4930-9d39-4225bb6f1b97 
> from RUNNING to DESTROYING
> 2019-09-09 05:04:26: I0909 05:04:26.388640 90003 containerizer.cpp:1203] 
> Cleaning up orphan container a127917b-96fe-4100-b73d-5f876ce9ffc1
> 2019-09-09 05:04:26: I0909 05:04:26.388650 90003 containerizer.cpp:2520] 
> Destroying container a127917b-96fe-4100-b73d-5f876ce9ffc1 in RUNNING state
> 2019-09-09 05:04:26: I0909 05:04:26.388659 90003 containerizer.cpp:3187] 
> Transitioning the state of container a127917b-96fe-4100-b73d-5f876ce9ffc1 
> from RUNNING to DESTROYING
> 2019-09-09 05:04:26: I0909 05:04:26.388689 90003 containerizer.cpp:1203] 
> Cleaning up orphan container 
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436
> 2019-09-09 05:04:26: I0909 05:04:26.388698 90003 containerizer.cpp:2520] 
> Destroying container 
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436 in 
> RUNNING state
> 2019-09-09 05:04:26: I0909 05:04:26.388706 90003 containerizer.cpp:3187] 
> Transitioning the state of container 
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436 
> from RUNNING to DESTROYING
> 2019-09-09 05:04:26: I0909 05:04:26.388720 90003 containerizer.cpp:1203] 
> Cleaning up orphan container 2ee154e2-3cc4-420a-99fb-065e740f3091
> 2019-09-09 05:04:26: I0909 05:04:26.388729 90003 containerizer.cpp:2520] 
> Destroying container 2ee154e2-3cc4-420a-99fb-065e740f3091 in RUNNING state
> 2019-09-09 05:04:26: I0909 05:04:26.388737 90003 containerizer.cpp:3187] 
> Transitioning the state of container 2ee154e2-3cc4-420a-99fb-065e740f3091 
> from RUNNING to DESTROYING
> 2019-09-09 05:04:26: I0909 05:04:26.388783 90003 containerizer.cpp:3026] 
> Container 
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436 has 
> exited
> 2019-09-09 05:04:26: I0909 05:04:26.388837 89929 linux_launcher.cpp:576] 
> Asked to destroy container 
> a127917b-96fe-4100-b73d-5f876ce9ffc1.9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> 2019-09-09 05:04:26: I0909 05:04:26.388904 89929 linux_launcher.cpp:618] 
> Destroying cgroup 
> '/sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97'
> 2019-09-09 05:04:26: I0909 05:04:26.389147 89929 linux_launcher.cpp:576] 
> Asked to destroy container 
> 2ee154e2-3cc4-420a-99fb-065e740f3091.49fe2bf9-17af-415f-92b6-92a4db619436
> 2019-09-09 05:04:26: I0909 05:04:26.389173 89929 linux_launcher.cpp:618] 
> Destroying cgroup 
> '/sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436'
> 2019-09-09 05:04:26: I0909 05:04:26.389261 89947 cgroups.cpp:2854] Freezing 
> cgroup 
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> 2019-09-09 05:04:26: I0909 05:04:26.389269 89948 cgroups.cpp:2854] Freezing 
> cgroup 
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.389454 89953 cgroups.cpp:2854] Freezing 
> cgroup 
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436
> 2019-09-09 05:04:26: I0909 05:04:26.389530 89956 cgroups.cpp:1242] 
> Successfully froze cgroup 
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97/mesos
>  after 166912ns
> 2019-09-09 05:04:26: I0909 05:04:26.389582 89965 cgroups.cpp:2854] Freezing 
> cgroup 
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.389605 89937 cgroups.cpp:1242] 
> Successfully froze cgroup 
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97
>  after 269056ns
> 2019-09-09 05:04:26: I0909 05:04:26.389679 89964 cgroups.cpp:1242] 
> Successfully froze cgroup 
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436
>  after 145920ns
> 2019-09-09 05:04:26: I0909 05:04:26.389761 89963 cgroups.cpp:2872] Thawing 
> cgroup 
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.389888 89969 cgroups.cpp:1242] 
> Successfully froze cgroup 
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436/mesos
>  after 219136ns
> 2019-09-09 05:04:26: I0909 05:04:26.389904 89974 cgroups.cpp:2872] Thawing 
> cgroup 
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436
> 2019-09-09 05:04:26: I0909 05:04:26.390111 89980 cgroups.cpp:2872] Thawing 
> cgroup 
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436/mesos
> 2019-09-09 05:04:26: I0909 05:04:26.390151 89987 cgroups.cpp:1271] 
> Successfully thawed cgroup 
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436
>  after 128us
> 2019-09-09 05:04:26: I0909 05:04:26.390199 89980 cgroups.cpp:1271] 
> Successfully thawed cgroup 
> /sys/fs/cgroup/freezer/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436/mesos
>  after 47104ns
> 2019-09-09 05:04:26: I0909 05:04:26.390290 89956 cgroups.cpp:2872] Thawing 
> cgroup 
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97
> 2019-09-09 05:04:26: I0909 05:04:26.390463 89983 linux_launcher.cpp:650] 
> Destroying cgroup 
> '/sys/fs/cgroup/systemd/mesos/2ee154e2-3cc4-420a-99fb-065e740f3091/mesos/49fe2bf9-17af-415f-92b6-92a4db619436'
> 2019-09-09 05:04:26: I0909 05:04:26.392710 89995 cgroups.cpp:1271] 
> Successfully thawed cgroup 
> /sys/fs/cgroup/freezer/mesos/a127917b-96fe-4100-b73d-5f876ce9ffc1/mesos/9783e2bb-7c2e-4930-9d39-4225bb6f1b97
>  after 2.397184ms
> 2019-09-09 05:04:26: I0909 05:04:26.394942 89976 containerizer.cpp:2812] 
> Checkpointing termination state to nested container's runtime directory 
> '/var/run/mesos/containers/2ee154e2-3cc4-420a-99fb-065e740f3091/containers/49fe2bf9-17af-415f-92b6-92a4db619436/termination'
> 2019-09-09 05:04:26: mesos-agent: 
> /pkg/src/mesos/3rdparty/stout/include/stout/option.hpp:119: T& 
> Option<T>::get() & [with T = std::basic_string<char>]: Assertion `isSome()' 
> failed.
> 2019-09-09 05:04:26: *** Aborted at 1568019866 (unix time) try "date -d 
> @1568019866" if you are using GNU date ***
> 2019-09-09 05:04:26: PC: @     0x7f8229cc02c7 __GI_raise
> 2019-09-09 05:04:26: *** SIGABRT (@0x15f32) received by PID 89906 (TID 
> 0x7f820c148700) from PID 89906; stack trace: ***
> 2019-09-09 05:04:26: @     0x7f822a066680 (unknown)
> 2019-09-09 05:04:26: @     0x7f8229cc02c7 __GI_raise
> 2019-09-09 05:04:26: @     0x7f8229cc19b8 __GI_abort
> 2019-09-09 05:04:26: @     0x7f8229cb90e6 __assert_fail_base
> 2019-09-09 05:04:26: @     0x7f8229cb9192 __GI___assert_fail
> 2019-09-09 05:04:26: @     0x7f822d306e33 _ZNR6OptionISsE3getEv.part.137
> 2019-09-09 05:04:26: @     0x7f822d317c4f 
> mesos::internal::slave::MesosContainerizerProcess::______destroy()
> 2019-09-09 05:04:26: I0909 05:04:26.418018 89974 token_retriever.cpp:422] 
> Successfuly acquired token with expiration set at 2019-09-09 09:09:26+00:00
> 2019-09-09 05:04:26: I0909 05:04:26.418375 89974 token_retriever.cpp:280] 
> Scheduling token refresh tu run at 2019-09-09 09:08:56.041828249+00:00
> 2019-09-09 05:04:26: @     0x7f822de72fc1 process::ProcessBase::consume()
> 2019-09-09 05:04:26: @     0x7f822de899ac process::ProcessManager::resume()
> 2019-09-09 05:04:26: @     0x7f822de8f466 
> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
> 2019-09-09 05:04:26: @     0x7f822a840070 (unknown)
> 2019-09-09 05:04:26: @     0x7f822a05edd5 start_thread
> 2019-09-09 05:04:26: @     0x7f8229d88bfd __clone
> 2019-09-09 05:04:26: dcos-mesos-slave.service: main process exited, 
> code=killed, status=6/ABRT
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Reply via email to