JFYI. I finished the mesos-master migrate and all works fine as expectd.
-- Thanks, Chengwei On Wed, Nov 25, 2015 at 06:29:54PM +0800, Chengwei Yang wrote: > OOPS, > > We forgot to disable firewalld on the new centos7 VM, once firewalld disabled, > replicate finished in seconds. > > as below. > > ``` > I1125 18:27:33.737843 2490 replica.cpp:369] Replica ignoring promise request > as it is in RECOVERING status > I1125 18:27:33.740927 2489 replica.cpp:655] Replica received learned notice > for position 984 > I1125 18:27:33.741539 2489 leveldb.cpp:343] Persisting action (20 bytes) to > leveldb took 572913ns > I1125 18:27:33.741628 2489 replica.cpp:676] Persisted action at 984 > I1125 18:27:33.741673 2489 replica.cpp:661] Replica learned TRUNCATE action > at position 984 > I1125 18:27:33.742463 2491 recover.cpp:554] Updating replica status to VOTING > I1125 18:27:33.743335 2490 leveldb.cpp:306] Persisting metadata (8 bytes) to > leveldb took 739871ns > I1125 18:27:33.743413 2490 replica.cpp:320] Persisted replica status to > VOTING > I1125 18:27:33.743522 2490 recover.cpp:568] Successfully joined the Paxos > group > I1125 18:27:33.743727 2490 recover.cpp:452] Recover process terminated > ``` > > -- > Thanks, > Chengwei > > On Wed, Nov 25, 2015 at 06:25:51PM +0800, Chengwei Yang wrote: > > while the other 2 mesos-master (one leader and one follower) both repeat > > below > > log. > > > > I1125 18:06:33.315208 28401 replica.cpp:638] Replica in VOTING status > > received a broadcasted recover request > > I1125 18:06:43.316341 28404 replica.cpp:638] Replica in VOTING status > > received a broadcasted recover request > > I1125 18:06:53.318739 28399 replica.cpp:638] Replica in VOTING status > > received a broadcasted recover request > > I1125 18:07:03.321287 28403 replica.cpp:638] Replica in VOTING status > > received a broadcasted recover request > > > > Seems the new mesos-master can not catch up and continuously retry, is this > > a > > bug? > > > > I'm using mesos-0.21.0 on centos7, the vanilla rpm released by mesosphere. > > > > -- > > Thanks, > > Chengwei > > > > > > On Wed, Nov 25, 2015 at 05:45:56PM +0800, Chengwei Yang wrote: > > > Hi All, > > > > > > I did step 1 below and check logs from the new started mesos-master, and > > > it > > > continuously complaint like below. > > > > > > ``` > > > I1125 17:42:59.066706 2330 recover.cpp:188] Received a recover response > > > from a replica in EMPTY status > > > I1125 17:43:09.065188 2331 recover.cpp:111] Unable to finish the recover > > > protocol in 10secs, retrying > > > I1125 17:43:09.066992 2330 replica.cpp:638] Replica in EMPTY status > > > received a broadcasted recover request > > > I1125 17:43:09.067425 2324 recover.cpp:188] Received a recover response > > > from a replica in EMPTY status > > > I1125 17:43:19.067332 2331 recover.cpp:111] Unable to finish the recover > > > protocol in 10secs, retrying > > > I1125 17:43:19.069587 2323 replica.cpp:638] Replica in EMPTY status > > > received a broadcasted recover request > > > I1125 17:43:19.069807 2323 recover.cpp:188] Received a recover response > > > from a replica in EMPTY status > > > ``` > > > > > > Seems it can not catch up the other replicas? > > > > > > -- > > > Thanks, > > > Chengwei > > > > > > On Tue, Nov 24, 2015 at 09:47:16AM +0800, Chengwei Yang wrote: > > > > Hi all, > > > > > > > > We're using mesos in product on CentOS 6 and plan to upgrade CentOS to > > > > 7.1, to > > > > avoid affect any tasks running on mesos. We're about to replace all > > > > mesos-masters in fly. > > > > > > > > The procedure listed below: > > > > > > > > 0. 3 mesos-masters running on CentOS 6 > > > > 1. shutdown 1 mesos-master(CentOS 6) and bring up 1 mesos-master(CentOS > > > > 7) > > > > wait the new master synced for some time(is there any simple way to > > > > know when?) > > > > 2. repeat step 1 > > > > > > > > NOTE: we plan to shutdown non-leader first, and shutdown the > > > > leader(CentOS 6) > > > > last. > > > > > > > > Can we do this in such way? Or any other better suggestions? > > > > > > > > -- > > > > Thanks, > > > > Chengwei > > > > > > > > >
signature.asc
Description: Digital signature
