Re: [ceph-users] Cannot shutdown monitors

2017-02-10 Thread Michael Andersen
I definitely had all the rbd volumes unmounted. I am not sure if they were unmapped. I can try that. On Fri, Feb 10, 2017 at 9:10 PM, Brad Hubbard wrote: > On Sat, Feb 11, 2017 at 2:58 PM, Brad Hubbard wrote: > > Just making sure the list sees this for those that are following. > > > > On Sat,

Re: [ceph-users] Cannot shutdown monitors

2017-02-10 Thread Brad Hubbard
On Sat, Feb 11, 2017 at 2:58 PM, Brad Hubbard wrote: > Just making sure the list sees this for those that are following. > > On Sat, Feb 11, 2017 at 2:49 PM, Michael Andersen > wrote: >> Right, so yes libceph is loaded >> >> root@compound-7:~# lsmod | egrep "ceph|rbd" >> rbd6

Re: [ceph-users] Cannot shutdown monitors

2017-02-10 Thread Brad Hubbard
Just making sure the list sees this for those that are following. On Sat, Feb 11, 2017 at 2:49 PM, Michael Andersen wrote: > Right, so yes libceph is loaded > > root@compound-7:~# lsmod | egrep "ceph|rbd" > rbd69632 0 > libceph 245760 1 rbd > libcrc32c

Re: [ceph-users] Cannot shutdown monitors

2017-02-10 Thread Shinobu Kinjo
On Sat, Feb 11, 2017 at 1:08 PM, Michael Andersen wrote: > I believe I did shutdown mon process. Is that not done by the > > sudo systemctl stop ceph\*.service ceph\*.target Oh, that's I missed. > > command? Also, as I noted, the mon process does not show up in ps after I do > that, but I still

Re: [ceph-users] Cannot shutdown monitors

2017-02-10 Thread Michael Andersen
I believe I did shutdown mon process. Is that not done by the sudo systemctl stop ceph\*.service ceph\*.target command? Also, as I noted, the mon process does not show up in ps after I do that, but I still get the shutdown halting. The libceph kernel module may be installed. I did not do so deli

Re: [ceph-users] Cannot shutdown monitors

2017-02-10 Thread Brad Hubbard
That looks like dmesg output from the libceph kernel module. Do you have the libceph kernel module loaded? If the answer to that question is "yes" the follow-up question is "Why?" as it is not required for a MON or OSD host. On Sat, Feb 11, 2017 at 1:18 PM, Michael Andersen wrote: > Yeah, all th

Re: [ceph-users] Cannot shutdown monitors

2017-02-10 Thread Shinobu Kinjo
You may need to stop MON process (if it's acting as primary). And once you make sure that all OSDs sessions are moved to another MON, you would be able to shutdown physical host. Have you tried that? On Sat, Feb 11, 2017 at 12:18 PM, Michael Andersen wrote: > Yeah, all three mons have OSDs on

Re: [ceph-users] Cannot shutdown monitors

2017-02-10 Thread Michael Andersen
Yeah, all three mons have OSDs on the same machines. On Feb 10, 2017 7:13 PM, "Shinobu Kinjo" wrote: > Is your primary MON running on the host which some OSDs are running on? > > On Sat, Feb 11, 2017 at 11:53 AM, Michael Andersen > wrote: > > Hi > > > > I am running a small cluster of 8 machine

Re: [ceph-users] Cannot shutdown monitors

2017-02-10 Thread Shinobu Kinjo
Is your primary MON running on the host which some OSDs are running on? On Sat, Feb 11, 2017 at 11:53 AM, Michael Andersen wrote: > Hi > > I am running a small cluster of 8 machines (80 osds), with three monitors on > Ubuntu 16.04. Ceph version 10.2.5. > > I cannot reboot the monitors without phy

[ceph-users] Cannot shutdown monitors

2017-02-10 Thread Michael Andersen
Hi I am running a small cluster of 8 machines (80 osds), with three monitors on Ubuntu 16.04. Ceph version 10.2.5. I cannot reboot the monitors without physically going into the datacenter and power cycling them. What happens is that while shutting down, ceph gets stuck trying to contact the othe