Re: [ceph-users] ceph-mon leader election problem, should it be improved ?

2017-07-10 Thread Z Will
Hi Joao: > Basically, this would be something similar to heartbeats. If a monitor can't > reach all monitors in an existing quorum, then just don't do anything. Based on your solution, I make a little change : - send a probe to all monitors - if get a quorum ,

Re: [ceph-users] ceph-mon leader election problem, should it be improved ?

2017-07-08 Thread Z Will
Hi Sage: After these days consideration, and reading some related papers , I think we can just make a very little change to solve the problems above and make monitors to tolerate most of the network partition . The most of logic are still same as before except one : - send a probe to eac

Re: [ceph-users] ceph-mon leader election problem, should it be improved ?

2017-07-06 Thread Sage Weil
On Thu, 6 Jul 2017, Z Will wrote: > Hi Joao : > > Thanks for thorough analysis . My initial concern is that , I think > in some cases , network failure will make low rank monitor see little > siblings (not enough to form a quorum ) , but some high rank mointor > can see more siblings, so I want

Re: [ceph-users] ceph-mon leader election problem, should it be improved ?

2017-07-06 Thread Z Will
Hi Joao : Thanks for thorough analysis . My initial concern is that , I think in some cases , network failure will make low rank monitor see little siblings (not enough to form a quorum ) , but some high rank mointor can see more siblings, so I want to try to choose the one who can see the most

Re: [ceph-users] ceph-mon leader election problem, should it be improved ?

2017-07-05 Thread Joao Eduardo Luis
On 07/05/2017 08:01 AM, Z Will wrote: Hi Joao: I think this is all because we choose the monitor with the smallest rank number to be leader. For this kind of network error, no matter which mon has lost connection with the mon who has the smallest rank num , will be constantly calling an elec

Re: [ceph-users] ceph-mon leader election problem, should it be improved ?

2017-07-05 Thread Z Will
Hi Joao: I think this is all because we choose the monitor with the smallest rank number to be leader. For this kind of network error, no matter which mon has lost connection with the mon who has the smallest rank num , will be constantly calling an election, that say ,will constantly affact t

Re: [ceph-users] ceph-mon leader election problem, should it be improved ?

2017-07-04 Thread Joao Eduardo Luis
On 07/04/2017 06:57 AM, Z Will wrote: Hi: I am testing ceph-mon brain split . I have read the code . If I understand it right , I know it won't be brain split. But I think there is still another problem. My ceph version is 0.94.10. And here is my test detail : 3 ceph-mons , there ranks are 0,

Re: [ceph-users] ceph-mon leader election problem, should it be improved ?

2017-07-03 Thread Z Will
Hi Alvaro: From the code , I see unsigned need = monmap->size() / 2 + 1; So for 2 mons , the quorum must be 2 so that it can start election. That's why I use 3 mons. I know if I stop mon.0 or mon.1 , everything will work fine. And if this failure happens, it must be handled by human ? Is the

Re: [ceph-users] ceph-mon leader election problem, should it be improved ?

2017-07-03 Thread han vincent
I think it is really a bug, and I tested it. if the network between mon.0 and mon.1 is cut off, it is easy to reproduce. mon.0 \ \ \ \ mon.1 -- mon.2 mon.0 win the election

Re: [ceph-users] ceph-mon leader election problem, should it be improved ?

2017-07-03 Thread Alvaro Soto
Z, You are forcing a byzantine failure, the paxos implemented to form the consensus ring of the mon daemons does not support this kind of failures, that is why you get and erratic behaviour, I believe is the common paxos algorithm implemented in mon daemon code. If you just gracefully shutdown a m

[ceph-users] ceph-mon leader election problem, should it be improved ?

2017-07-03 Thread Z Will
Hi: I am testing ceph-mon brain split . I have read the code . If I understand it right , I know it won't be brain split. But I think there is still another problem. My ceph version is 0.94.10. And here is my test detail : 3 ceph-mons , there ranks are 0, 1, 2 respectively.I stop the rank 1 mon