[
https://issues.apache.org/jira/browse/MESOS-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157937#comment-16157937
]
Chun-Hung Hsiao edited comment on MESOS-5482 at 9/8/17 1:01 AM:
----------------------------------------------------------------
My superficial guess is that someone (Marathon?) asked to shutdown framework
{{f853458f-b07b-4b79-8192-24953f474369-0000}}, but the framework itself didn't
know that and asked to launch task
{{metrics_statsd.70dff634-7dce-11e7-bea2-0242f4eb80ac}} (possibly due to some
race condition?). From the agent's perspective, the framework was already
shutdown, so the task never existed; but from the framework's perspective, it
thought the task was launched so kept asking to kill it but got no response.
[~gengmao] Can you attach the complete master/agent logs, and the sandbox log
of Marathon related to launching/shutting
{{f853458f-b07b-4b79-8192-24953f474369-0000}} if possible, so we can
confirm/reject the above guess?
was (Author: chhsia0):
My superficial guess is that someone (Marathon?) asked to shutdown framework
{{f853458f-b07b-4b79-8192-24953f474369-0000}}, but the framework itself didn't
know that and asked to launch task
{{metrics_statsd.70dff634-7dce-11e7-bea2-0242f4eb80ac}} (possibly due to some
race condition?). From the agent's perspective, the framework was already
shutdown, so the task never existed; but from the framework's perspective, it
thought the task was launched so kept asking to kill it but got no response.
[~gengmao] Can you attach the complete master/agent logs, and the sandbox log
of Marathon related to launching/shutting
{{f853458f-b07b-4b79-8192-24953f474369-0000}} if possible?
> mesos/marathon task stuck in staging after slave reboot
> -------------------------------------------------------
>
> Key: MESOS-5482
> URL: https://issues.apache.org/jira/browse/MESOS-5482
> Project: Mesos
> Issue Type: Bug
> Reporter: lutful karim
> Labels: tech-debt
> Attachments: marathon-mesos-masters_after-reboot.log,
> mesos-masters_mesos.log, mesos_slaves_after_reboot.log,
> tasks_running_before_rebooot.marathon
>
>
> The main idea of mesos/marathon is to sleep well, but after node reboot mesos
> task gets stuck in staging for about 4 hours.
> To reproduce the issue:
> - setup a mesos cluster in HA mode with systemd enabled mesos-master and
> mesos-slave service.
> - run docker registry (https://hub.docker.com/_/registry/ ) with mesos
> constraint (hostname:LIKE:mesos-slave-1) in one node. Reboot the node and
> notice that task getting stuck in staging.
> Possible workaround: service mesos-slave restart fixes the issue.
> OS: centos 7.2
> mesos version: 0.28.1
> marathon: 1.1.1
> zookeeper: 3.4.8
> docker: 1.9.1 dockerAPIversion: 1.21
> error message:
> May 30 08:38:24 euca-10-254-237-140 mesos-slave[832]: W0530 08:38:24.120013
> 909 slave.cpp:2018] Ignoring kill task
> docker-registry.066fb448-2628-11e6-bedd-d00d0ef81dc3 because the executor
> 'docker-registry.066fb448-2628-11e6-bedd-d00d0ef81dc3' of framework
> 8517fcb7-f2d0-47ad-ae02-837570bef929-0000 is terminating/terminated
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
