I0721 19:30:01.449234 11585 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (363)@10.10.186.76:5050 I0721 19:30:01.518342 11590 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (335)@10.10.151.44:5050
this message is saw in the third master logs ------------------ ???????? ------------------ ??????: "Joseph Wu";<[email protected]>; ????????: 2016??7??22??(??????) ????1:26 ??????: "user"<[email protected]>; ????: Re: mesos crash You should check that all masters can talk to one another. This must be bi-directional. From the logs, there are only two masters: 10.10.186.76, 10.10.151.44. Presumably, one of those IPs is the master printing those logs. So there's a third master that is not reachable. Also, you should see the following message in the logs if your masters have connectivity: Received a recover response from a replica in VOTING status On Thu, Jul 21, 2016 at 4:33 AM, ???????????? <[email protected]> wrote: I set MESOS_quorum to 2,start 3 masters,masters still crashed. I0721 19:30:01.449234 11585 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (363)@10.10.186.76:5050 I0721 19:30:01.518342 11590 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (335)@10.10.151.44:5050 2016-07-21 19:30:02,086:11583(0x7fb9a5db3700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 13ms 2016-07-21 19:30:02,086:11583(0x7fb977fff700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 13ms I0721 19:30:02.252887 11588 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (366)@10.10.186.76:5050 I0721 19:30:02.338348 11584 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (338)@10.10.151.44:5050 I0721 19:30:03.000632 11589 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (369)@10.10.186.76:5050 I0721 19:30:03.257617 11590 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (341)@10.10.151.44:5050 I0721 19:30:03.752945 11588 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (372)@10.10.186.76:5050 I0721 19:30:03.965860 11587 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (344)@10.10.151.44:5050 I0721 19:30:04.260133 11591 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (375)@10.10.186.76:5050 I0721 19:30:04.554836 11588 http.cpp:381] HTTP GET for /master/state from 10.10.159.106:49056 with User-Agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36' I0721 19:30:04.619936 11585 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (347)@10.10.151.44:5050 I0721 19:30:05.023617 11588 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (378)@10.10.186.76:5050 I0721 19:30:05.424495 11587 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (350)@10.10.151.44:5050 I0721 19:30:05.791437 11591 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (381)@10.10.186.76:5050 I0721 19:30:06.310353 11585 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (353)@10.10.151.44:5050 I0721 19:30:06.675050 11584 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (384)@10.10.186.76:5050 I0721 19:30:06.816815 11584 http.cpp:381] HTTP GET for /master/state from 10.10.159.106:49058 with User-Agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36' I0721 19:30:07.194869 11584 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (387)@10.10.186.76:5050 I0721 19:30:07.303427 11590 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (356)@10.10.151.44:5050 I0721 19:30:07.812754 11588 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (359)@10.10.151.44:5050 I0721 19:30:08.055387 11591 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (390)@10.10.186.76:5050 I0721 19:30:08.687959 11589 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (362)@10.10.151.44:5050 I0721 19:30:08.796779 11585 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (393)@10.10.186.76:5050 I0721 19:30:09.521941 11588 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (396)@10.10.186.76:5050 I0721 19:30:09.563043 11584 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (365)@10.10.151.44:5050 I0721 19:30:10.208359 11591 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (368)@10.10.151.44:5050 I0721 19:30:10.412394 11589 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (399)@10.10.186.76:5050 I0721 19:30:10.904234 11590 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (371)@10.10.151.44:5050 I0721 19:30:11.211045 11585 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (402)@10.10.186.76:5050 I0721 19:30:11.734627 11584 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (374)@10.10.151.44:5050 I0721 19:30:12.056349 11586 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (405)@10.10.186.76:5050 I0721 19:30:12.596413 11591 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (377)@10.10.151.44:5050 I0721 19:30:12.789674 11588 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (408)@10.10.186.76:5050 I0721 19:30:13.380175 11587 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (380)@10.10.151.44:5050 I0721 19:30:13.691694 11585 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (411)@10.10.186.76:5050 I0721 19:30:14.069072 11589 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (383)@10.10.151.44:5050 I0721 19:30:14.225723 11588 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (414)@10.10.186.76:5050 I0721 19:30:14.570282 11587 http.cpp:381] HTTP GET for /master/state from 10.10.159.106:49063 with User-Agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36' I0721 19:30:14.711906 11587 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (386)@10.10.151.44:5050 I0721 19:30:15.060392 11586 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (417)@10.10.186.76:5050 I0721 19:30:15.424607 11588 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (389)@10.10.151.44:5050 2016-07-21 19:30:15,433:11583(0x7fb977fff700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 13ms 2016-07-21 19:30:15,433:11583(0x7fb9a5db3700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 13ms I0721 19:30:15.849879 11587 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (420)@10.10.186.76:5050 I0721 19:30:16.205354 11584 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (392)@10.10.151.44:5050 I0721 19:30:16.578645 11590 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (423)@10.10.186.76:5050 I0721 19:30:16.882959 11589 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (395)@10.10.151.44:5050 I0721 19:30:17.388262 11591 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (426)@10.10.186.76:5050 I0721 19:30:17.475071 11591 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (398)@10.10.151.44:5050 I0721 19:30:17.817046 11585 http.cpp:381] HTTP GET for /master/state from 10.10.159.106:49065 with User-Agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36' I0721 19:30:17.922546 11588 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (429)@10.10.186.76:5050 I0721 19:30:18.241506 11591 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (401)@10.10.151.44:5050 I0721 19:30:18.562672 11589 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (432)@10.10.186.76:5050 I0721 19:30:19.093329 11587 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (404)@10.10.151.44:5050 I0721 19:30:19.356179 11585 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (435)@10.10.186.76:5050 I0721 19:30:19.612740 11584 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (407)@10.10.151.44:5050 I0721 19:30:20.352881 11590 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (438)@10.10.186.76:5050 I0721 19:30:20.410377 11591 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (410)@10.10.151.44:5050 I0721 19:30:21.153241 11586 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (413)@10.10.151.44:5050 I0721 19:30:21.256703 11588 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (441)@10.10.186.76:5050 F0721 19:30:21.685487 11586 master.cpp:1662] Recovery failed: Failed to recover registrar: Failed to perform fetch within 2mins *** Check failure stack trace: *** @ 0x7fb9b735237c google::LogMessage::Fail() @ 0x7fb9b73522d8 google::LogMessage::SendToLog() @ 0x7fb9b7351cce google::LogMessage::Flush() @ 0x7fb9b7354a88 google::LogMessageFatal::~LogMessageFatal() @ 0x7fb9b62b064c mesos::internal::master::fail() @ 0x7fb9b6384ffb _ZNSt5_BindIFPFvRKSsS1_EPKcSt12_PlaceholderILi1EEEE6__callIvJS1_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE @ 0x7fb9b635f8df _ZNSt5_BindIFPFvRKSsS1_EPKcSt12_PlaceholderILi1EEEEclIJS1_EvEET0_DpOT_ @ 0x7fb9b632c783 _ZZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvRKSsS6_EPKcSt12_PlaceholderILi1EEEEvEERKS2_OT_NS2_6PreferEENUlS6_E_clES6_ @ 0x7fb9b63850cd _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvS1_S1_EPKcSt12_PlaceholderILi1EEEEvEERKS6_OT_NS6_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_ @ 0x4a4833 std::function<>::operator()() @ 0x49f0eb _ZN7process8internal3runISt8functionIFvRKSsEEJS4_EEEvRKSt6vectorIT_SaIS8_EEDpOT0_ @ 0x4997c2 process::Future<>::fail() @ 0x7fb9b5f75a22 process::Promise<>::fail() @ 0x7fb9b63824f0 process::internal::thenf<>() @ 0x7fb9b63c6bd9 _ZNSt5_BindIFPFvRKSt8functionIFN7process6FutureI7NothingEERKN5mesos8internal8RegistryEEERKSt10shared_ptrINS1_7PromiseIS3_EEERKNS2_IS7_EEESB_SH_St12_PlaceholderILi1EEEE6__callIvISM_EILm0ELm1ELm2EEEET_OSt5tupleIIDpT0_EESt12_Index_tupleIIXspT1_EEE @ 0x7fb9b63bd8cd std::_Bind<>::operator()<>() @ 0x7fb9b63a4821 std::_Function_handler<>::_M_invoke() @ 0x7fb9b63bdaff std::function<>::operator()() @ 0x7fb9b63a4955 _ZZNK7process6FutureIN5mesos8internal8RegistryEE5onAnyIRSt8functionIFvRKS4_EEvEES8_OT_NS4_6PreferEENUlS8_E_clES8_ @ 0x7fb9b63c6c85 _ZNSt17_Function_handlerIFvRKN7process6FutureIN5mesos8internal8RegistryEEEEZNKS5_5onAnyIRSt8functionIS8_EvEES7_OT_NS5_6PreferEEUlS7_E_E9_M_invokeERKSt9_Any_dataS7_ @ 0x7fb9b63bdaff std::function<>::operator()() @ 0x7fb9b64267c4 process::internal::run<>() @ 0x7fb9b641cef4 process::Future<>::fail() @ 0x7fb9b64572de std::_Mem_fn<>::operator()<>() @ 0x7fb9b64526c7 _ZNSt5_BindIFSt7_Mem_fnIMN7process6FutureIN5mesos8internal8RegistryEEEFbRKSsEES6_St12_PlaceholderILi1EEEE6__callIbIS8_EILm0ELm1EEEET_OSt5tupleIIDpT0_EESt12_Index_tupleIIXspT1_EEE @ 0x7fb9b644ad23 _ZNSt5_BindIFSt7_Mem_fnIMN7process6FutureIN5mesos8internal8RegistryEEEFbRKSsEES6_St12_PlaceholderILi1EEEEclIJS8_EbEET0_DpOT_ @ 0x7fb9b6440c63 _ZZNK7process6FutureIN5mesos8internal8RegistryEE8onFailedISt5_BindIFSt7_Mem_fnIMS4_FbRKSsEES4_St12_PlaceholderILi1EEEEbEERKS4_OT_NS4_6PreferEENUlS9_E_clES9_ @ 0x7fb9b6452752 _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureIN5mesos8internal8RegistryEE8onFailedISt5_BindIFSt7_Mem_fnIMS8_FbS1_EES8_St12_PlaceholderILi1EEEEbEERKS8_OT_NS8_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_ @ 0x4a4833 std::function<>::operator()() @ 0x49f0eb _ZN7process8internal3runISt8functionIFvRKSsEEJS4_EEEvRKSt6vectorIT_SaIS8_EEDpOT0_ @ 0x7fb9b641cecc process::Future<>::fail() @ 0x7fb9b6415eac process::Promise<>::fail() ------------------ ???????? ------------------ ??????: "Joseph Wu";<[email protected]>; ????????: 2016??7??20??(??????) ????2:15 ??????: "user"<[email protected]>; ????: Re: mesos crash When you start a new group of masters, the masters will not initialize their replicated log (from the EMPTY state) until all masters are present. This means (quorum * 2 - 1) masters must be up and reachable. We enforce this behavior because the replicated log can get into a inconsistent state otherwise. Consider a simple case where you have an existing group of 3 masters: 1) Each master's replicated log is up-to-date with the leader. 2) Two masters are completely destroyed (their disks blow up, or something). 3) You bring two new masters up. If we allow a quorum of new masters to rejoin an existing cluster, the old master's data becomes the source of truth because it has the highest log position. This is not necessarily correct. By dis-allowing a quorum of new masters to rejoin an existing cluster, it becomes the operator's job to recover after catastrophic failures. On Tue, Jul 19, 2016 at 2:57 AM, ???????????? <[email protected]> wrote: yes,I start 1 master ,work fined,I try 3 masters later,thanks ------------------ ???????? ------------------ ??????: "haosdent";<[email protected]>; ????????: 2016??7??19??(??????) ????5:39 ??????: "user"<[email protected]>; ????: Re: mesos crash I think may because you start 2 masters and set MESOS_quorum to 2, then the election could not finish successfully. May you start 3 masters? Or remove zookeeper and just start 1 master. On Tue, Jul 19, 2016 at 5:26 PM, ???????????? <[email protected]> wrote: No,I deployed in two different server ------------------ ???????? ------------------ ??????: "haosdent";<[email protected]>; ????????: 2016??7??19??(??????) ????5:05 ??????: "user"<[email protected]>; ????: Re: mesos crash Hi, >I start two master node : Did you start them in a same server with same work dir? On Tue, Jul 19, 2016 at 12:18 PM, ???????????? <[email protected]> wrote: Hello,I deploy mesos on centos,kernel is 3.14.73,mesos version 1.0.0, this is my master config: export MESOS_log_dir=/apps/mesos/logs/ export MESOS_ip=0.0.0.0 export MESOS_hostname=`hostname` export MESOS_logging_level=INFO export MESOS_quorum=2 export MESOS_work_dir=/apps/mesos/master export MESOS_zk=zk://zk1:2181,zk2:2181,zk3:2181/oss-mesos export MESOS_allocator=HierarchicalDRF export MESOS_cluster=oss-mesos export MESOS_credentials=/apps/mesos/etc/mesos/credentials.txt export MESOS_registry=replicated_log export MESOS_webui_dir=/apps/mesos/share/mesos/webui export MESOS_zk_session_timeout=90secs export MESOS_max_executors_per_slave=10 export MESOS_registry_fetch_timeout=2mins I start two master node : but master nodes will crash in a few minute the log message is I0719 11:50:22.673280 5376 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (287)@10.10.186.76:5050 I0719 11:50:23.154119 5381 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (504)@10.10.179.252:5050 I0719 11:50:23.154749 5376 recover.cpp:197] Received a recover response from a replica in EMPTY status I0719 11:50:23.156838 5378 recover.cpp:197] Received a recover response from a replica in EMPTY status I0719 11:50:23.563072 5382 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (289)@10.10.186.76:5050 I0719 11:50:23.883855 5376 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (507)@10.10.179.252:5050 I0719 11:50:23.884414 5380 recover.cpp:197] Received a recover response from a replica in EMPTY status I0719 11:50:23.886569 5375 recover.cpp:197] Received a recover response from a replica in EMPTY status I0719 11:50:24.163056 5379 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (291)@10.10.186.76:5050 I0719 11:50:24.425379 5378 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (510)@10.10.179.252:5050 I0719 11:50:24.425864 5379 recover.cpp:197] Received a recover response from a replica in EMPTY status I0719 11:50:24.428951 5375 recover.cpp:197] Received a recover response from a replica in EMPTY status I0719 11:50:24.935673 5379 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (293)@10.10.186.76:5050 F0719 11:50:25.262277 5381 master.cpp:1662] Recovery failed: Failed to recover registrar: Failed to perform fetch within 2mins *** Check failure stack trace: *** @ 0x7fe6fa0ac37c google::LogMessage::Fail() @ 0x7fe6fa0ac2d8 google::LogMessage::SendToLog() @ 0x7fe6fa0abcce google::LogMessage::Flush() @ 0x7fe6fa0aea88 google::LogMessageFatal::~LogMessageFatal() @ 0x7fe6f900a64c mesos::internal::master::fail() @ 0x7fe6f90deffb _ZNSt5_BindIFPFvRKSsS1_EPKcSt12_PlaceholderILi1EEEE6__callIvJS1_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE @ 0x7fe6f90b98df _ZNSt5_BindIFPFvRKSsS1_EPKcSt12_PlaceholderILi1EEEEclIJS1_EvEET0_DpOT_ @ 0x7fe6f9086783 _ZZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvRKSsS6_EPKcSt12_PlaceholderILi1EEEEvEERKS2_OT_NS2_6PreferEENUlS6_E_clES6_ @ 0x7fe6f90df0cd _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvS1_S1_EPKcSt12_PlaceholderILi1EEEEvEERKS6_OT_NS6_6PreferEEUlS1_E _E9_M_invokeERKSt9_Any_dataS1_ @ 0x4a4833 std::function<>::operator()() @ 0x49f0eb _ZN7process8internal3runISt8functionIFvRKSsEEJS4_EEEvRKSt6vectorIT_SaIS8_EEDpOT0_ @ 0x4997c2 process::Future<>::fail() @ 0x7fe6f8ccfa22 process::Promise<>::fail() @ 0x7fe6f90dc4f0 process::internal::thenf<>() @ 0x7fe6f9120bd9 _ZNSt5_BindIFPFvRKSt8functionIFN7process6FutureI7NothingEERKN5mesos8internal8RegistryEEERKSt10shared_ptrINS1_7PromiseIS3_EEERKNS2_IS7_EEESB_SH_St12 _PlaceholderILi1EEEE6__callIvISM_EILm0ELm1ELm2EEEET_OSt5tupleIIDpT0_EESt12_Index_tupleIIXspT1_EEE @ 0x7fe6f91178cd std::_Bind<>::operator()<>() @ 0x7fe6f90fe821 std::_Function_handler<>::_M_invoke() @ 0x7fe6f9117aff std::function<>::operator()() @ 0x7fe6f90fe955 _ZZNK7process6FutureIN5mesos8internal8RegistryEE5onAnyIRSt8functionIFvRKS4_EEvEES8_OT_NS4_6PreferEENUlS8_E_clES8_ @ 0x7fe6f9120c85 _ZNSt17_Function_handlerIFvRKN7process6FutureIN5mesos8internal8RegistryEEEEZNKS5_5onAnyIRSt8functionIS8_EvEES7_OT_NS5_6PreferEEUlS7_E_E9_M_invokeER KSt9_Any_dataS7_ @ 0x7fe6f9117aff std::function<>::operator()() @ 0x7fe6f91807c4 process::internal::run<>() @ 0x7fe6f9176ef4 process::Future<>::fail() @ 0x7fe6f91b12de std::_Mem_fn<>::operator()<>() I0719 11:50:25.414069 5382 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (513)@10.10.179.252:5050 I0719 11:50:25.414718 5376 recover.cpp:197] Received a recover response from a replica in EMPTY status @ 0x7fe6f91ac6c7 _ZNSt5_BindIFSt7_Mem_fnIMN7process6FutureIN5mesos8internal8RegistryEEEFbRKSsEES6_St12_PlaceholderILi1EEEE6__callIbIS8_EILm0ELm1EEEET_OSt5tupleIIDpT 0_EESt12_Index_tupleIIXspT1_EEE I0719 11:50:25.416431 5377 recover.cpp:197] Received a recover response from a replica in EMPTY status I0719 11:50:25.418115 5379 http.cpp:381] HTTP GET for /master/state from 10.10.159.106:3363 with User-Agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36' @ 0x7fe6f91a4d23 _ZNSt5_BindIFSt7_Mem_fnIMN7process6FutureIN5mesos8internal8RegistryEEEFbRKSsEES6_St12_PlaceholderILi1EEEEclIJS8_EbEET0_DpOT_ @ 0x7fe6f919ac63 _ZZNK7process6FutureIN5mesos8internal8RegistryEE8onFailedISt5_BindIFSt7_Mem_fnIMS4_FbRKSsEES4_St12_PlaceholderILi1EEEEbEERKS4_OT_NS4_6PreferEENUlS9 _E_clES9_ @ 0x7fe6f91ac752 _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureIN5mesos8internal8RegistryEE8onFailedISt5_BindIFSt7_Mem_fnIMS8_FbS1_EES8_St12_PlaceholderILi1EEEE bEERKS8_OT_NS8_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_ @ 0x4a4833 std::function<>::operator()() @ 0x49f0eb _ZN7process8internal3runISt8functionIFvRKSsEEJS4_EEEvRKSt6vectorIT_SaIS8_EEDpOT0_ @ 0x7fe6f9176ecc process::Future<>::fail() @ 0x7fe6f916feac process::Promise<>::fail() error logs is Log file created at: 2016/07/19 11:50:25 Running on machine: oss-mesos-master-bjc-001 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg F0719 11:50:25.262277 5381 master.cpp:1662] Recovery failed: Failed to recover registrar: Failed to perform fetch within 2mins can you help me.thanks -- Best Regards, Haosdent Huang -- Best Regards, Haosdent Huang
