Re: [ceph-users] monitor quorum

2014-09-19 Thread James Eckersall
I decided to remove mon-03 and re-create it. I copied the keyring and monmap from one of the other monitors, but the cluster is still reporting it as down (out of quorum). mon03 is now not in the electing state, but in the probing state. mon-03:~# ceph --admin-daemon

Re: [ceph-users] monitor quorum

2014-09-18 Thread James Eckersall
Is anyone able to offer any advice on how to fix this? I've tried re-injecting the monmap into mon03 as that was mentioned in the mon troubleshooting docs, but that has not helped at all. mon03 is still stuck in the same electing state :( I've increased the debug level on mon03 and it is

[ceph-users] monitor quorum

2014-09-17 Thread James Eckersall
Hi, I have a ceph cluster running 0.80.1 on Ubuntu 14.04. I have 3 monitors and 4 OSD nodes currently. Everything has been running great up until today where I've got an issue with the monitors. I moved mon03 to a different switchport so it would have temporarily lost connectivity. Since then,

Re: [ceph-users] monitor quorum

2014-09-17 Thread Florian Haas
On Wed, Sep 17, 2014 at 1:58 PM, James Eckersall james.eckers...@gmail.com wrote: Hi, I have a ceph cluster running 0.80.1 on Ubuntu 14.04. I have 3 monitors and 4 OSD nodes currently. Everything has been running great up until today where I've got an issue with the monitors. I moved

Re: [ceph-users] monitor quorum

2014-09-17 Thread James Eckersall
Hi, Thanks for the advice. I feel pretty dumb as it does indeed look like a simple networking issue. You know how you check things 5 times and miss the most obvious one... J On 17 September 2014 16:04, Florian Haas flor...@hastexo.com wrote: On Wed, Sep 17, 2014 at 1:58 PM, James Eckersall

Re: [ceph-users] monitor quorum

2014-09-17 Thread Florian Haas
On Wed, Sep 17, 2014 at 5:21 PM, James Eckersall james.eckers...@gmail.com wrote: Hi, Thanks for the advice. I feel pretty dumb as it does indeed look like a simple networking issue. You know how you check things 5 times and miss the most obvious one... J No worries at all .:) Cheers,

Re: [ceph-users] monitor quorum

2014-09-17 Thread James Eckersall
Hi, Now I feel dumb for jumping to the conclusion that it was a simple networking issue - it isn't. I've just checked connectivity properly and I can ping and telnet 6789 from all mon servers to all other mon servers. I've just restarted the mon03 service and the log is showing the following: