[
https://issues.apache.org/jira/browse/MESOS-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768514#comment-16768514
]
Alexander Rukletsov commented on MESOS-9322:
--------------------------------------------
[~guoshiwei] I agree and think it is a bug, too. We recently have at least two
bugs related to "zombie executors":
* MESOS-9502, stuck IOSwitchboard
* MESOS-8125, MESOS-9501 pid reusal
I would like to ask you to provide us with executor and agent logs, so we can
determine whether you see one of the aforementioned issues or this is a
separate bug.
> Executor exited accidentally, but mesos-agent did not report TASK_FAILED
> event.
> -------------------------------------------------------------------------------
>
> Key: MESOS-9322
> URL: https://issues.apache.org/jira/browse/MESOS-9322
> Project: Mesos
> Issue Type: Bug
> Components: agent
> Affects Versions: 1.4.1
> Environment: Linux n14-068-081 4.4.0-33.bm.1-amd64 #1 SMP Fri, 01 Sep
> 2017 18:36:21 +0800 x86_64 GNU/Linux
> OS: debion 8.10
> mesos version: 1.4.1
> Reporter: Shiwei Guo
> Priority: Major
>
> The log about this executor:
> executorid:
> 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz'
>
> {noformat}
> I0914 10:40:36.448287 2505 slave.cpp:7336] Recovering executor
> 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz'
> of framework ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000
> I0914 10:40:36.479209 2511 gc.cpp:58] Scheduling
> '/opt/tiger/mesos_deploy/mesos_titan/slave/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/189e4b23-c892-4c87-9069-dfc98ca5edc8'
> for gc 3.1546935280563days in the future
> I0914 10:40:36.479287 2511 gc.cpp:58] Scheduling
> '/opt/tiger/mesos_deploy/mesos_titan/slave/meta/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/189e4b23-c892-4c87-9069-dfc98ca5edc8'
> for gc 3.15469352761481days in the future
> I0914 10:40:36.479310 2511 gc.cpp:58] Scheduling
> '/opt/tiger/mesos_deploy/mesos_titan/slave/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/4b27d1d4-fe67-4475-88bc-14e994acfb85'
> for gc -1.02171850967407days in the future
> I0914 10:40:36.479337 2511 gc.cpp:58] Scheduling
> '/opt/tiger/mesos_deploy/mesos_titan/slave/meta/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/4b27d1d4-fe67-4475-88bc-14e994acfb85'
> for gc -1.02171850987259days in the future
> I0914 10:40:36.480459 2514 gc.cpp:169] Deleting
> /opt/tiger/mesos_deploy/mesos_titan/slave/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/4b27d1d4-fe67-4475-88bc-14e994acfb85
> I0914 10:40:36.552492 2516 status_update_manager.cpp:211] Recovering executor
> 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz'
> of framework ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000
> I0914 10:40:36.553234 2519 containerizer.cpp:665] Recovering container
> 106c7257-fabb-4d58-8fcb-89b15bb9d404 for executor
> 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz'
> of framework ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000
> I0914 10:40:36.591421 2514 gc.cpp:177] Deleted
> '/opt/tiger/mesos_deploy/mesos_titan/slave/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/4b27d1d4-fe67-4475-88bc-14e994acfb85'
> I0914 10:40:36.705005 2534 gc.cpp:169] Deleting
> /opt/tiger/mesos_deploy/mesos_titan/slave/meta/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/4b27d1d4-fe67-4475-88bc-14e994acfb85
> I0914 10:40:36.705201 2534 gc.cpp:177] Deleted
> '/opt/tiger/mesos_deploy/mesos_titan/slave/meta/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/4b27d1d4-fe67-4475-88bc-14e994acfb85'
> I0914 10:40:36.764660 2514 slave.cpp:6180] Sending reconnect request to
> executor
> 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz'
> of framework ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000 at
> executor(1)@10.14.68.81:26383
> I0914 10:40:38.765029 2533 slave.cpp:4346] Killing un-reregistered executor
> 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz'
> of framework ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000 at
> executor(1)@10.14.68.81:26383
> I0916 06:09:40.390676 2530 gc.cpp:169] Deleting
> /opt/tiger/mesos_deploy/mesos_titan/slave/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/189e4b23-c892-4c87-9069-dfc98ca5edc8
> I0916 06:09:40.432540 2530 gc.cpp:177] Deleted
> '/opt/tiger/mesos_deploy/mesos_titan/slave/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/189e4b23-c892-4c87-9069-dfc98ca5edc8'
> I0916 06:09:40.432620 2530 gc.cpp:169] Deleting
> /opt/tiger/mesos_deploy/mesos_titan/slave/meta/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/189e4b23-c892-4c87-9069-dfc98ca5edc8
> I0916 06:09:40.437860 2530 gc.cpp:177] Deleted
> '/opt/tiger/mesos_deploy/mesos_titan/slave/meta/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/189e4b23-c892-4c87-9069-dfc98ca5edc8'
> W1012 17:14:24.485033 2513 slave.cpp:3100] Ignoring kill task
> gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;ts:1536560605651
> because the executor
> 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz'
> of framework ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000 at
> executor(1)@10.14.68.81:26383 is terminating
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
