[ 
https://issues.apache.org/jira/browse/MESOS-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380688#comment-16380688
 ] 

Joseph Wu commented on MESOS-8623:
----------------------------------

This CHECK can be hit whenever the master attempts to send any message to a 
framework recovered via Agent registration (i.e. Agent reports that the 
framework is running tasks on it), but before the framework has reconnected to 
the master.  

The master should be guarding against sending messages to disconnected 
frameworks, so we'll have to track down which code path is responsible for 
sending this message.

A cursory {{git blame}} suggests that this crash could have happened from 1.2.0 
onwards, but this also depends on which message is being sent:
https://github.com/apache/mesos/commit/0dbceafa3b7caba9e541cd64bce3d5421e3b1262

Note: This commit removed a {{return;}} that would have hidden this bug.
https://github.com/apache/mesos/commit/65efb347301f90e638361a50282cb74980c4c081

> Crashed framework brings down the whole Mesos cluster
> -----------------------------------------------------
>
>                 Key: MESOS-8623
>                 URL: https://issues.apache.org/jira/browse/MESOS-8623
>             Project: Mesos
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 1.4.1
>         Environment: Debian 8
> Mesos 1.4.1
>            Reporter: Tomas Barton
>            Priority: Critical
>
> It might be hard to replicate, but when you do your Mesos cluster is gone. 
> The issue was caused by an unresponsive Docker engine on a single agent node. 
> Unfortunately even after fixing Docker issues, all Mesos masters repeatedly 
> failed to start. In despair I've deleted all {{replicated_log}} data from 
> Master and ZooKeeper. Even after that messages agent's {{replicated_log}} got 
> replayed and the master crashed again. Average lifetime for Mesos master was 
> less than 1min.
> {code}
> mesos-master[3814]: I0228 00:25:55.269835  3828 network.hpp:436] ZooKeeper 
> group memberships changed
> mesos-master[3814]: I0228 00:25:55.269979  3832 group.cpp:700] Trying to get 
> '/mesos/log_replicas/0000002519' in ZooKeeper
> mesos-master[3814]: I0228 00:25:55.271117  3832 group.cpp:700] Trying to get 
> '/mesos/log_replicas/0000002520' in ZooKeeper
> mesos-master[3814]: I0228 00:25:55.277971  3832 group.cpp:700] Trying to get 
> '/mesos/log_replicas/0000002521' in ZooKeeper
> mesos-master[3814]: I0228 00:25:55.279296  3827 network.hpp:484] ZooKeeper 
> group PIDs: { log-replica(1)
> mesos-master[3814]: W0228 00:26:15.261255  3831 master.hpp:2372] Master 
> attempted to send message to disconnected framework 
> 911c4b47-2ba7-4959-b59e-c48d896fe210-0005 (kafka)
> mesos-master[3814]: F0228 00:26:15.261318  3831 master.hpp:2382] 
> CHECK_SOME(pid): is NONE
> mesos-master[3814]: *** Check failure stack trace: ***
> mesos-master[3814]: @     0x7f7187ca073d  google::LogMessage::Fail()
> mesos-master[3814]: @     0x7f7187ca23bd  google::LogMessage::SendToLog()
> mesos-master[3814]: @     0x7f7187ca0302  google::LogMessage::Flush()
> mesos-master[3814]: @     0x7f7187ca2da9  
> google::LogMessageFatal::~LogMessageFatal()
> mesos-master[3814]: @     0x7f7186d6d769  _CheckFatal::~_CheckFatal()
> mesos-master[3814]: @     0x7f71870465d5  
> mesos::internal::master::Framework::send<>()
> mesos-master[3814]: @     0x7f7186fcfe8a  
> mesos::internal::master::Master::executorMessage()
> mesos-master[3814]: @     0x7f718706b1a1  ProtobufProcess<>::handler4<>()
> mesos-master[3814]: @     0x7f7187008e36  
> std::_Function_handler<>::_M_invoke()
> mesos-master[3814]: @     0x7f71870293d1  ProtobufProcess<>::visit()
> mesos-master[3814]: @     0x7f7186fb7ee4  
> mesos::internal::master::Master::_visit()
> mesos-master[3814]: @     0x7f7186fd0d5d  
> mesos::internal::master::Master::visit()
> mesos-master[3814]: @     0x7f7187c02e22  process::ProcessManager::resume()
> mesos-master[3814]: @     0x7f7187c08d46  
> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vE
> mesos-master[3814]: @     0x7f7185babca0  (unknown)
> mesos-master[3814]: @     0x7f71853c6064  start_thread
> mesos-master[3814]: @     0x7f71850fb62d  (unknown)
> systemd[1]: mesos-master.service: main process exited, code=killed, 
> status=6/ABRT
> systemd[1]: Unit mesos-master.service entered failed state.
> systemd[1]: mesos-master.service holdoff time over, scheduling restart.
> systemd[1]: Stopping Mesos Master...
> systemd[1]: Starting Mesos Master...
> systemd[1]: Started Mesos Master.
> mesos-master[27840]: WARNING: Logging before InitGoogleLogging() is written 
> to STDERR
> mesos-master[27840]: I0228 01:32:38.294122 27829 main.cpp:232] Build: 
> 2017-11-18 02:15:41 by admin
> mesos-master[27840]: I0228 01:32:38.294168 27829 main.cpp:233] Version: 1.4.1
> mesos-master[27840]: I0228 01:32:38.294178 27829 main.cpp:236] Git tag: 1.4.1
> mesos-master[27840]: I0228 01:32:38.294186 27829 main.cpp:240] Git SHA: 
> c844db9ac7c0cef59be87438c6781bfb71adcc42
> mesos-master[27840]: I0228 01:32:38.296067 27829 main.cpp:340] Using 
> 'HierarchicalDRF' allocator
> mesos-master[27840]: I0228 01:32:38.411576 27829 replica.cpp:779] Replica 
> recovered with log positions 13 -> 14 with 0 holes and 0 unlearned
> mesos-master[27840]: 2018-02-28 
> 01:32:38,412:27829(0x7fab44755700):ZOO_INFO@log_env@726: Client 
> environment:zookeeper.version=zookeeper C client 3.4.8
> mesos-master[27840]: 2018-02-28 
> 01:32:38,412:27829(0x7fab44755700):ZOO_INFO@log_env@730: Client 
> environment:host.name=svc01
> mesos-master[27840]: 2018-02-28 
> 01:32:38,412:27829(0x7fab44755700):ZOO_INFO@log_env@737: Client 
> environment:os.name=Linux
> mesos-master[27840]: I0228 01:32:38.412711 27841 log.cpp:107] Attempting to 
> join replica to ZooKeeper group
> mesos-master[27840]: 2018-02-28 
> 01:32:38,412:27829(0x7fab4775b700):ZOO_INFO@log_env@726: Client 
> environment:zookeeper.version=zookeeper C client 3.4.8
> mesos-master[27840]: 2018-02-28 
> 01:32:38,412:27829(0x7fab4775b700):ZOO_INFO@log_env@730: Client 
> environment:host.name=svc01
> mesos-master[27840]: I0228 01:32:38.412932 27841 recover.cpp:451] Starting 
> replica recovery
> mesos-master[27840]: I0228 01:32:38.413024 27846 recover.cpp:477] Replica is 
> in VOTING status
> mesos-master[27840]: 2018-02-28 
> 01:32:38,412:27829(0x7fab4775b700):ZOO_INFO@log_env@737: Client 
> environment:os.name=Linux
> mesos-master[27840]: 2018-02-28 
> 01:32:38,413:27829(0x7fab4775b700):ZOO_INFO@log_env@738: Client 
> environment:os.arch=3.16.0-5-amd64
> mesos-master[27840]: 2018-02-28 
> 01:32:38,413:27829(0x7fab4775b700):ZOO_INFO@log_env@739: Client 
> environment:os.version=#1 SMP Debian 3.16.51-3+deb8u1 (2018-01-08)
> mesos-master[27840]: 2018-02-28 
> 01:32:38,413:27829(0x7fab47f5c700):ZOO_INFO@log_env@726: Client 
> environment:zookeeper.version=zookeeper C client 3.4.8
> {code}
> on slave Mesos is trying many times to kill all instances of kafka framework:
> {code}
> I0227 23:14:57.993875 26049 slave.cpp:2931] Asked to kill task 
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f of framework 
> ecd3a4be-d34
> W0227 23:14:57.993914 26049 slave.cpp:3099] Ignoring kill task 
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f because the 
> executor 'ser
> I0227 23:15:02.993985 26048 slave.cpp:2931] Asked to kill task 
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f of framework 
> ecd3a4be-d34
> W0227 23:15:02.994027 26048 slave.cpp:3099] Ignoring kill task 
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f because the 
> executor 'ser
> I0227 23:15:07.992681 26041 slave.cpp:2931] Asked to kill task 
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f of framework 
> ecd3a4be-d34
> W0227 23:15:07.992720 26041 slave.cpp:3099] Ignoring kill task 
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f because the 
> executor 'ser
> I0227 23:15:12.992703 26039 slave.cpp:2931] Asked to kill task 
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f of framework 
> ecd3a4be-d34
> {code}
> the Docker daemon stopped to respond, thus any tasks were failing:
> {code}
> E0227 23:06:20.440865 26044 slave.cpp:5292] Container 
> '10763986-e133-4483-b34e-cfe10903a46f' for executor 
> 'api_rapi-a.d25fc1e0-1c12-11e8-a38c-fef9d3423
> *** Aborted at 1519772780 (unix time) try "date -d @1519772780" if you are 
> using GNU date ***
> PC: @     0x7f1326961067 (unknown)
> PC: @     0x7f1326961067 (unknown)
> *** SIGABRT (@0x47c0) received by PID 18368 (TID 0x7f131c27b700) from PID 
> 18368; stack trace: ***
> @     0x7f1326ce6890 (unknown)
> @     0x7f1326961067 (unknown)
> @     0x7f1326962448 (unknown)
> @     0x560218e2f740 (unknown)
> @     0x560218e2f77c (unknown)
> @     0x7f1329261ff9 (unknown)
> @     0x7f1329261c81 (unknown)
> @     0x7f1329261c41 (unknown)
> @     0x7f1329263ecf (unknown)
> @     0x7f1329261561 (unknown)
> @     0x7f1328422509 (unknown)
> @     0x7f13284253ff (unknown)
> @     0x7f1328433134 (unknown)
> @     0x7f132844ab26 (unknown)
> @     0x7f1328398a47 (unknown)
> @     0x7f132839a9ed (unknown)
> @     0x7f1329260bcc (unknown)
> @     0x7f1328398a47 (unknown)
> @     0x7f132839a9ed (unknown)
> @     0x7f13289ec817 (unknown)
> @     0x7f13289ec91f (unknown)
> @     0x7f1329259818 (unknown)
> @     0x7f1329259ac3 (unknown)
> @     0x7f1329218e29 (unknown)
> @     0x7f132921ed16 (unknown)
> @     0x7f13271c3ca0 (unknown)
> @     0x7f1326cdf064 start_thread
> @     0x7f1326a1462d (unknown)
> '
> I0227 23:06:20.441429 26043 slave.cpp:5405] Executor 
> 'api_rapi-a.d25fc1e0-1c12-11e8-a38c-fef9d3423c7f' of framework 
> ecd3a4be-d34c-46f3-b358-c4e26ac0d13
> I0227 23:06:20.441520 26043 slave.cpp:4399] Handling status update 
> TASK_FAILED (UUID: 003e072e-341b-48ce-abb5-79849128ba6f) for task 
> api_rapi-a.d25fc1e
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to