You should check that all masters can talk to one another. This must be bi-directional. From the logs, there are only two masters: 10.10.186.76, 10.10.151.44. Presumably, one of those IPs is the master printing those logs. So there's a third master that is not reachable.
Also, you should see the following message in the logs if your masters have connectivity: > Received a recover response from a replica in VOTING status > On Thu, Jul 21, 2016 at 4:33 AM, 梦开始的地方 <[email protected]> wrote: > I set MESOS_quorum to 2,start 3 masters,masters still crashed. > > I0721 19:30:01.449234 11585 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (363)@10.10.186.76:5050 > I0721 19:30:01.518342 11590 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (335)@10.10.151.44:5050 > 2016-07-21 > 19:30:02,086:11583(0x7fb9a5db3700):ZOO_WARN@zookeeper_interest@1570: > Exceeded deadline by 13ms > 2016-07-21 > 19:30:02,086:11583(0x7fb977fff700):ZOO_WARN@zookeeper_interest@1570: > Exceeded deadline by 13ms > I0721 19:30:02.252887 11588 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (366)@10.10.186.76:5050 > I0721 19:30:02.338348 11584 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (338)@10.10.151.44:5050 > I0721 19:30:03.000632 11589 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (369)@10.10.186.76:5050 > I0721 19:30:03.257617 11590 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (341)@10.10.151.44:5050 > I0721 19:30:03.752945 11588 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (372)@10.10.186.76:5050 > I0721 19:30:03.965860 11587 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (344)@10.10.151.44:5050 > I0721 19:30:04.260133 11591 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (375)@10.10.186.76:5050 > I0721 19:30:04.554836 11588 http.cpp:381] HTTP GET for /master/state from > 10.10.159.106:49056 with User-Agent='Mozilla/5.0 (X11; Linux x86_64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36' > I0721 19:30:04.619936 11585 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (347)@10.10.151.44:5050 > I0721 19:30:05.023617 11588 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (378)@10.10.186.76:5050 > I0721 19:30:05.424495 11587 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (350)@10.10.151.44:5050 > I0721 19:30:05.791437 11591 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (381)@10.10.186.76:5050 > I0721 19:30:06.310353 11585 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (353)@10.10.151.44:5050 > I0721 19:30:06.675050 11584 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (384)@10.10.186.76:5050 > I0721 19:30:06.816815 11584 http.cpp:381] HTTP GET for /master/state from > 10.10.159.106:49058 with User-Agent='Mozilla/5.0 (X11; Linux x86_64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36' > I0721 19:30:07.194869 11584 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (387)@10.10.186.76:5050 > I0721 19:30:07.303427 11590 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (356)@10.10.151.44:5050 > I0721 19:30:07.812754 11588 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (359)@10.10.151.44:5050 > I0721 19:30:08.055387 11591 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (390)@10.10.186.76:5050 > I0721 19:30:08.687959 11589 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (362)@10.10.151.44:5050 > I0721 19:30:08.796779 11585 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (393)@10.10.186.76:5050 > I0721 19:30:09.521941 11588 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (396)@10.10.186.76:5050 > I0721 19:30:09.563043 11584 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (365)@10.10.151.44:5050 > I0721 19:30:10.208359 11591 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (368)@10.10.151.44:5050 > I0721 19:30:10.412394 11589 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (399)@10.10.186.76:5050 > I0721 19:30:10.904234 11590 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (371)@10.10.151.44:5050 > I0721 19:30:11.211045 11585 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (402)@10.10.186.76:5050 > I0721 19:30:11.734627 11584 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (374)@10.10.151.44:5050 > I0721 19:30:12.056349 11586 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (405)@10.10.186.76:5050 > I0721 19:30:12.596413 11591 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (377)@10.10.151.44:5050 > I0721 19:30:12.789674 11588 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (408)@10.10.186.76:5050 > I0721 19:30:13.380175 11587 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (380)@10.10.151.44:5050 > I0721 19:30:13.691694 11585 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (411)@10.10.186.76:5050 > I0721 19:30:14.069072 11589 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (383)@10.10.151.44:5050 > I0721 19:30:14.225723 11588 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (414)@10.10.186.76:5050 > I0721 19:30:14.570282 11587 http.cpp:381] HTTP GET for /master/state from > 10.10.159.106:49063 with User-Agent='Mozilla/5.0 (X11; Linux x86_64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36' > I0721 19:30:14.711906 11587 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (386)@10.10.151.44:5050 > I0721 19:30:15.060392 11586 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (417)@10.10.186.76:5050 > I0721 19:30:15.424607 11588 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (389)@10.10.151.44:5050 > 2016-07-21 > 19:30:15,433:11583(0x7fb977fff700):ZOO_WARN@zookeeper_interest@1570: > Exceeded deadline by 13ms > 2016-07-21 > 19:30:15,433:11583(0x7fb9a5db3700):ZOO_WARN@zookeeper_interest@1570: > Exceeded deadline by 13ms > I0721 19:30:15.849879 11587 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (420)@10.10.186.76:5050 > I0721 19:30:16.205354 11584 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (392)@10.10.151.44:5050 > I0721 19:30:16.578645 11590 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (423)@10.10.186.76:5050 > I0721 19:30:16.882959 11589 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (395)@10.10.151.44:5050 > I0721 19:30:17.388262 11591 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (426)@10.10.186.76:5050 > I0721 19:30:17.475071 11591 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (398)@10.10.151.44:5050 > I0721 19:30:17.817046 11585 http.cpp:381] HTTP GET for /master/state from > 10.10.159.106:49065 with User-Agent='Mozilla/5.0 (X11; Linux x86_64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36' > I0721 19:30:17.922546 11588 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (429)@10.10.186.76:5050 > I0721 19:30:18.241506 11591 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (401)@10.10.151.44:5050 > I0721 19:30:18.562672 11589 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (432)@10.10.186.76:5050 > I0721 19:30:19.093329 11587 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (404)@10.10.151.44:5050 > I0721 19:30:19.356179 11585 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (435)@10.10.186.76:5050 > I0721 19:30:19.612740 11584 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (407)@10.10.151.44:5050 > I0721 19:30:20.352881 11590 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (438)@10.10.186.76:5050 > I0721 19:30:20.410377 11591 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (410)@10.10.151.44:5050 > I0721 19:30:21.153241 11586 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (413)@10.10.151.44:5050 > I0721 19:30:21.256703 11588 replica.cpp:673] Replica in VOTING status > received a broadcasted recover request from (441)@10.10.186.76:5050 > F0721 19:30:21.685487 11586 master.cpp:1662] Recovery failed: Failed to > recover registrar: Failed to perform fetch within 2mins > *** Check failure stack trace: *** > @ 0x7fb9b735237c google::LogMessage::Fail() > @ 0x7fb9b73522d8 google::LogMessage::SendToLog() > @ 0x7fb9b7351cce google::LogMessage::Flush() > @ 0x7fb9b7354a88 google::LogMessageFatal::~LogMessageFatal() > @ 0x7fb9b62b064c mesos::internal::master::fail() > @ 0x7fb9b6384ffb > > _ZNSt5_BindIFPFvRKSsS1_EPKcSt12_PlaceholderILi1EEEE6__callIvJS1_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE > @ 0x7fb9b635f8df > _ZNSt5_BindIFPFvRKSsS1_EPKcSt12_PlaceholderILi1EEEEclIJS1_EvEET0_DpOT_ > @ 0x7fb9b632c783 > > _ZZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvRKSsS6_EPKcSt12_PlaceholderILi1EEEEvEERKS2_OT_NS2_6PreferEENUlS6_E_clES6_ > @ 0x7fb9b63850cd > > _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvS1_S1_EPKcSt12_PlaceholderILi1EEEEvEERKS6_OT_NS6_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_ > @ 0x4a4833 std::function<>::operator()() > @ 0x49f0eb > > _ZN7process8internal3runISt8functionIFvRKSsEEJS4_EEEvRKSt6vectorIT_SaIS8_EEDpOT0_ > @ 0x4997c2 process::Future<>::fail() > @ 0x7fb9b5f75a22 process::Promise<>::fail() > @ 0x7fb9b63824f0 process::internal::thenf<>() > @ 0x7fb9b63c6bd9 > > _ZNSt5_BindIFPFvRKSt8functionIFN7process6FutureI7NothingEERKN5mesos8internal8RegistryEEERKSt10shared_ptrINS1_7PromiseIS3_EEERKNS2_IS7_EEESB_SH_St12_PlaceholderILi1EEEE6__callIvISM_EILm0ELm1ELm2EEEET_OSt5tupleIIDpT0_EESt12_Index_tupleIIXspT1_EEE > @ 0x7fb9b63bd8cd std::_Bind<>::operator()<>() > @ 0x7fb9b63a4821 std::_Function_handler<>::_M_invoke() > @ 0x7fb9b63bdaff std::function<>::operator()() > @ 0x7fb9b63a4955 > > _ZZNK7process6FutureIN5mesos8internal8RegistryEE5onAnyIRSt8functionIFvRKS4_EEvEES8_OT_NS4_6PreferEENUlS8_E_clES8_ > @ 0x7fb9b63c6c85 > > _ZNSt17_Function_handlerIFvRKN7process6FutureIN5mesos8internal8RegistryEEEEZNKS5_5onAnyIRSt8functionIS8_EvEES7_OT_NS5_6PreferEEUlS7_E_E9_M_invokeERKSt9_Any_dataS7_ > @ 0x7fb9b63bdaff std::function<>::operator()() > @ 0x7fb9b64267c4 process::internal::run<>() > @ 0x7fb9b641cef4 process::Future<>::fail() > @ 0x7fb9b64572de std::_Mem_fn<>::operator()<>() > @ 0x7fb9b64526c7 > > _ZNSt5_BindIFSt7_Mem_fnIMN7process6FutureIN5mesos8internal8RegistryEEEFbRKSsEES6_St12_PlaceholderILi1EEEE6__callIbIS8_EILm0ELm1EEEET_OSt5tupleIIDpT0_EESt12_Index_tupleIIXspT1_EEE > @ 0x7fb9b644ad23 > > _ZNSt5_BindIFSt7_Mem_fnIMN7process6FutureIN5mesos8internal8RegistryEEEFbRKSsEES6_St12_PlaceholderILi1EEEEclIJS8_EbEET0_DpOT_ > @ 0x7fb9b6440c63 > > _ZZNK7process6FutureIN5mesos8internal8RegistryEE8onFailedISt5_BindIFSt7_Mem_fnIMS4_FbRKSsEES4_St12_PlaceholderILi1EEEEbEERKS4_OT_NS4_6PreferEENUlS9_E_clES9_ > @ 0x7fb9b6452752 > > _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureIN5mesos8internal8RegistryEE8onFailedISt5_BindIFSt7_Mem_fnIMS8_FbS1_EES8_St12_PlaceholderILi1EEEEbEERKS8_OT_NS8_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_ > @ 0x4a4833 std::function<>::operator()() > @ 0x49f0eb > > _ZN7process8internal3runISt8functionIFvRKSsEEJS4_EEEvRKSt6vectorIT_SaIS8_EEDpOT0_ > @ 0x7fb9b641cecc process::Future<>::fail() > @ 0x7fb9b6415eac process::Promise<>::fail() > > ------------------ 原始邮件 ------------------ > *发件人:* "Joseph Wu";<[email protected]>; > *发送时间:* 2016年7月20日(星期三) 凌晨2:15 > *收件人:* "user"<[email protected]>; > *主题:* Re: mesos crash > > When you start a new group of masters, the masters will not initialize > their replicated log (from the EMPTY state) until all masters are present. > This means (quorum * 2 - 1) masters must be up and reachable. > > We enforce this behavior because the replicated log can get into a > inconsistent state otherwise. Consider a simple case where you have an > existing group of 3 masters: > 1) Each master's replicated log is up-to-date with the leader. > 2) Two masters are completely destroyed (their disks blow up, or > something). > 3) You bring two new masters up. > > If we allow a quorum of new masters to rejoin an existing cluster, the old > master's data becomes the source of truth because it has the highest log > position. This is not necessarily correct. > By dis-allowing a quorum of new masters to rejoin an existing cluster, it > becomes the operator's job to recover after catastrophic failures. > > On Tue, Jul 19, 2016 at 2:57 AM, 梦开始的地方 <[email protected]> wrote: > >> >> yes,I start 1 master ,work fined,I try 3 masters later,thanks >> >> ------------------ 原始邮件 ------------------ >> *发件人:* "haosdent";<[email protected]>; >> *发送时间:* 2016年7月19日(星期二) 下午5:39 >> *收件人:* "user"<[email protected]>; >> *主题:* Re: mesos crash >> >> I think may because you start 2 masters and set MESOS_quorum to 2, then >> the election could not finish successfully. May you start 3 masters? Or >> remove zookeeper and just start 1 master. >> >> On Tue, Jul 19, 2016 at 5:26 PM, 梦开始的地方 <[email protected]> wrote: >> >>> No,I deployed in two different server >>> >>> >>> ------------------ 原始邮件 ------------------ >>> *发件人:* "haosdent";<[email protected]>; >>> *发送时间:* 2016年7月19日(星期二) 下午5:05 >>> *收件人:* "user"<[email protected]>; >>> *主题:* Re: mesos crash >>> >>> Hi, >>> >I start two master node : >>> Did you start them in a same server with same work dir? >>> >>> On Tue, Jul 19, 2016 at 12:18 PM, 梦开始的地方 <[email protected]> wrote: >>> >>>> >>>> >>>> Hello,I deploy mesos on centos,kernel is 3.14.73,mesos version 1.0.0, >>>> this is my master config: >>>> export MESOS_log_dir=/apps/mesos/logs/ >>>> export MESOS_ip=0.0.0.0 >>>> export MESOS_hostname=`hostname` >>>> export MESOS_logging_level=INFO >>>> export MESOS_quorum=2 >>>> export MESOS_work_dir=/apps/mesos/master >>>> export MESOS_zk=zk://zk1:2181,zk2:2181,zk3:2181/oss-mesos >>>> export MESOS_allocator=HierarchicalDRF >>>> export MESOS_cluster=oss-mesos >>>> export MESOS_credentials=/apps/mesos/etc/mesos/credentials.txt >>>> export MESOS_registry=replicated_log >>>> export MESOS_webui_dir=/apps/mesos/share/mesos/webui >>>> export MESOS_zk_session_timeout=90secs >>>> export MESOS_max_executors_per_slave=10 >>>> export MESOS_registry_fetch_timeout=2mins >>>> >>>> I start two master node : >>>> but master nodes will crash in a few minute >>>> the log message is >>>> >>>> I0719 11:50:22.673280 5376 replica.cpp:673] Replica in EMPTY status >>>> received a broadcasted recover request from (287)@10.10.186.76:5050 >>>> I0719 11:50:23.154119 5381 replica.cpp:673] Replica in EMPTY status >>>> received a broadcasted recover request from (504)@10.10.179.252:5050 >>>> I0719 11:50:23.154749 5376 recover.cpp:197] Received a recover >>>> response from a replica in EMPTY status >>>> I0719 11:50:23.156838 5378 recover.cpp:197] Received a recover >>>> response from a replica in EMPTY status >>>> I0719 11:50:23.563072 5382 replica.cpp:673] Replica in EMPTY status >>>> received a broadcasted recover request from (289)@10.10.186.76:5050 >>>> I0719 11:50:23.883855 5376 replica.cpp:673] Replica in EMPTY status >>>> received a broadcasted recover request from (507)@10.10.179.252:5050 >>>> I0719 11:50:23.884414 5380 recover.cpp:197] Received a recover >>>> response from a replica in EMPTY status >>>> I0719 11:50:23.886569 5375 recover.cpp:197] Received a recover >>>> response from a replica in EMPTY status >>>> I0719 11:50:24.163056 5379 replica.cpp:673] Replica in EMPTY status >>>> received a broadcasted recover request from (291)@10.10.186.76:5050 >>>> I0719 11:50:24.425379 5378 replica.cpp:673] Replica in EMPTY status >>>> received a broadcasted recover request from (510)@10.10.179.252:5050 >>>> I0719 11:50:24.425864 5379 recover.cpp:197] Received a recover >>>> response from a replica in EMPTY status >>>> I0719 11:50:24.428951 5375 recover.cpp:197] Received a recover >>>> response from a replica in EMPTY status >>>> I0719 11:50:24.935673 5379 replica.cpp:673] Replica in EMPTY status >>>> received a broadcasted recover request from (293)@10.10.186.76:5050 >>>> F0719 11:50:25.262277 5381 master.cpp:1662] Recovery failed: Failed >>>> to recover registrar: Failed to perform fetch within 2mins >>>> *** Check failure stack trace: *** >>>> @ 0x7fe6fa0ac37c google::LogMessage::Fail() >>>> @ 0x7fe6fa0ac2d8 google::LogMessage::SendToLog() >>>> @ 0x7fe6fa0abcce google::LogMessage::Flush() >>>> @ 0x7fe6fa0aea88 google::LogMessageFatal::~LogMessageFatal() >>>> @ 0x7fe6f900a64c mesos::internal::master::fail() >>>> @ 0x7fe6f90deffb >>>> >>>> _ZNSt5_BindIFPFvRKSsS1_EPKcSt12_PlaceholderILi1EEEE6__callIvJS1_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE >>>> @ 0x7fe6f90b98df >>>> _ZNSt5_BindIFPFvRKSsS1_EPKcSt12_PlaceholderILi1EEEEclIJS1_EvEET0_DpOT_ >>>> @ 0x7fe6f9086783 >>>> _ZZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvRKSsS6_EPKcSt12_PlaceholderILi1EEEEvEERKS2_OT_NS2_6PreferEENUlS6_E_clES6_ >>>> @ 0x7fe6f90df0cd >>>> >>>> _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvS1_S1_EPKcSt12_PlaceholderILi1EEEEvEERKS6_OT_NS6_6PreferEEUlS1_E >>>> _E9_M_invokeERKSt9_Any_dataS1_ >>>> @ 0x4a4833 std::function<>::operator()() >>>> @ 0x49f0eb >>>> >>>> _ZN7process8internal3runISt8functionIFvRKSsEEJS4_EEEvRKSt6vectorIT_SaIS8_EEDpOT0_ >>>> @ 0x4997c2 process::Future<>::fail() >>>> @ 0x7fe6f8ccfa22 process::Promise<>::fail() >>>> @ 0x7fe6f90dc4f0 process::internal::thenf<>() >>>> @ 0x7fe6f9120bd9 >>>> >>>> _ZNSt5_BindIFPFvRKSt8functionIFN7process6FutureI7NothingEERKN5mesos8internal8RegistryEEERKSt10shared_ptrINS1_7PromiseIS3_EEERKNS2_IS7_EEESB_SH_St12 >>>> >>>> _PlaceholderILi1EEEE6__callIvISM_EILm0ELm1ELm2EEEET_OSt5tupleIIDpT0_EESt12_Index_tupleIIXspT1_EEE >>>> @ 0x7fe6f91178cd std::_Bind<>::operator()<>() >>>> @ 0x7fe6f90fe821 std::_Function_handler<>::_M_invoke() >>>> @ 0x7fe6f9117aff std::function<>::operator()() >>>> @ 0x7fe6f90fe955 >>>> >>>> _ZZNK7process6FutureIN5mesos8internal8RegistryEE5onAnyIRSt8functionIFvRKS4_EEvEES8_OT_NS4_6PreferEENUlS8_E_clES8_ >>>> @ 0x7fe6f9120c85 >>>> >>>> _ZNSt17_Function_handlerIFvRKN7process6FutureIN5mesos8internal8RegistryEEEEZNKS5_5onAnyIRSt8functionIS8_EvEES7_OT_NS5_6PreferEEUlS7_E_E9_M_invokeER >>>> KSt9_Any_dataS7_ >>>> @ 0x7fe6f9117aff std::function<>::operator()() >>>> @ 0x7fe6f91807c4 process::internal::run<>() >>>> @ 0x7fe6f9176ef4 process::Future<>::fail() >>>> @ 0x7fe6f91b12de std::_Mem_fn<>::operator()<>() >>>> I0719 11:50:25.414069 5382 replica.cpp:673] Replica in EMPTY status >>>> received a broadcasted recover request from (513)@10.10.179.252:5050 >>>> I0719 11:50:25.414718 5376 recover.cpp:197] Received a recover >>>> response from a replica in EMPTY status >>>> @ 0x7fe6f91ac6c7 >>>> >>>> _ZNSt5_BindIFSt7_Mem_fnIMN7process6FutureIN5mesos8internal8RegistryEEEFbRKSsEES6_St12_PlaceholderILi1EEEE6__callIbIS8_EILm0ELm1EEEET_OSt5tupleIIDpT >>>> 0_EESt12_Index_tupleIIXspT1_EEE >>>> I0719 11:50:25.416431 5377 recover.cpp:197] Received a recover >>>> response from a replica in EMPTY status >>>> I0719 11:50:25.418115 5379 http.cpp:381] HTTP GET for /master/state >>>> from 10.10.159.106:3363 with User-Agent='Mozilla/5.0 (X11; Linux >>>> x86_64) AppleWebKit/537.36 (KHTML, like >>>> Gecko) Chrome/48.0.2564.116 Safari/537.36' >>>> @ 0x7fe6f91a4d23 >>>> >>>> _ZNSt5_BindIFSt7_Mem_fnIMN7process6FutureIN5mesos8internal8RegistryEEEFbRKSsEES6_St12_PlaceholderILi1EEEEclIJS8_EbEET0_DpOT_ >>>> @ 0x7fe6f919ac63 >>>> >>>> _ZZNK7process6FutureIN5mesos8internal8RegistryEE8onFailedISt5_BindIFSt7_Mem_fnIMS4_FbRKSsEES4_St12_PlaceholderILi1EEEEbEERKS4_OT_NS4_6PreferEENUlS9 >>>> _E_clES9_ >>>> @ 0x7fe6f91ac752 >>>> >>>> _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureIN5mesos8internal8RegistryEE8onFailedISt5_BindIFSt7_Mem_fnIMS8_FbS1_EES8_St12_PlaceholderILi1EEEE >>>> bEERKS8_OT_NS8_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_ >>>> @ 0x4a4833 std::function<>::operator()() >>>> @ 0x49f0eb >>>> >>>> _ZN7process8internal3runISt8functionIFvRKSsEEJS4_EEEvRKSt6vectorIT_SaIS8_EEDpOT0_ >>>> @ 0x7fe6f9176ecc process::Future<>::fail() >>>> @ 0x7fe6f916feac process::Promise<>::fail() >>>> >>>> >>>> >>>> error logs is >>>> Log file created at: 2016/07/19 11:50:25 >>>> Running on machine: oss-mesos-master-bjc-001 >>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg >>>> F0719 11:50:25.262277 5381 master.cpp:1662] Recovery failed: Failed >>>> to recover registrar: Failed to perform fetch within 2mins >>>> can you help me.thanks >>>> >>> >>> >>> >>> -- >>> Best Regards, >>> Haosdent Huang >>> >> >> >> >> -- >> Best Regards, >> Haosdent Huang >> > >
