Re: [ceph-users] rebooting nodes in a ceph cluster
It is also useful to mention that you can set the noout flag when doing maintenance of any given length needs to exceeds the 'mon osd down out interval'. $ ceph osd set noout ** no re-balancing will happen ** $ ceph osd unset noout ** normal re-balancing rules will resume ** - Mike Dawson On 12/19/2013 7:51 PM, Sage Weil wrote: On Thu, 19 Dec 2013, John-Paul Robinson wrote: What impact does rebooting nodes in a ceph cluster have on the health of the ceph cluster? Can it trigger rebalancing activities that then have to be undone once the node comes back up? I have a 4 node ceph cluster each node has 11 osds. There is a single pool with redundant storage. If it takes 15 minutes for one of my servers to reboot is there a risk that some sort of needless automatic processing will begin? By default, we start rebalancing data after 5 minutes. You can adjust this (to, say, 15 minutes) with mon osd down out interval = 900 in ceph.conf. sage I'm assuming that the ceph cluster can go into a not ok state but that in this particular configuration all the data is protected against the single node failure and there is no place for the data to migrate too so nothing bad will happen. Thanks for any feedback. ~jpr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rebooting nodes in a ceph cluster
I think my wording was a bit misleading in my last message. Instead of no re-balancing will happen, I should have said that no OSDs will be marked out of the cluster with the noout flag set. - Mike On 12/21/2013 2:06 PM, Mike Dawson wrote: It is also useful to mention that you can set the noout flag when doing maintenance of any given length needs to exceeds the 'mon osd down out interval'. $ ceph osd set noout ** no re-balancing will happen ** $ ceph osd unset noout ** normal re-balancing rules will resume ** - Mike Dawson On 12/19/2013 7:51 PM, Sage Weil wrote: On Thu, 19 Dec 2013, John-Paul Robinson wrote: What impact does rebooting nodes in a ceph cluster have on the health of the ceph cluster? Can it trigger rebalancing activities that then have to be undone once the node comes back up? I have a 4 node ceph cluster each node has 11 osds. There is a single pool with redundant storage. If it takes 15 minutes for one of my servers to reboot is there a risk that some sort of needless automatic processing will begin? By default, we start rebalancing data after 5 minutes. You can adjust this (to, say, 15 minutes) with mon osd down out interval = 900 in ceph.conf. sage I'm assuming that the ceph cluster can go into a not ok state but that in this particular configuration all the data is protected against the single node failure and there is no place for the data to migrate too so nothing bad will happen. Thanks for any feedback. ~jpr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rebooting nodes in a ceph cluster
David Clarke writes:
Not directly related to Ceph, but you may want to investigate kexec[0]
('kexec-tools' package in Debian derived distributions) in order to
get your machines rebooting quicker. It essentially re-loads the
kernel as the last step of the shutdown procedure, skipping over the
lengthy BIOS/UEFI/controller firmware etc boot stages.
[0]: http://en.wikipedia.org/wiki/Kexec
I'd like to second that recommendation - I only discovered this
recently, and on systems with long BIOS initialization, this cuts down
the time to reboot *dramatically*, like from 5 to 1 minute.
--
Simon.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rebooting nodes in a ceph cluster
On 12/19/13, 7:51 PM, Sage Weil wrote: If it takes 15 minutes for one of my servers to reboot is there a risk that some sort of needless automatic processing will begin? By default, we start rebalancing data after 5 minutes. You can adjust this (to, say, 15 minutes) with mon osd down out interval = 900 in ceph.conf. Will Ceph detect if the OSDs come back while it is re-balancing and stop? -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] rebooting nodes in a ceph cluster
What impact does rebooting nodes in a ceph cluster have on the health of the ceph cluster? Can it trigger rebalancing activities that then have to be undone once the node comes back up? I have a 4 node ceph cluster each node has 11 osds. There is a single pool with redundant storage. If it takes 15 minutes for one of my servers to reboot is there a risk that some sort of needless automatic processing will begin? I'm assuming that the ceph cluster can go into a not ok state but that in this particular configuration all the data is protected against the single node failure and there is no place for the data to migrate too so nothing bad will happen. Thanks for any feedback. ~jpr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rebooting nodes in a ceph cluster
On Thu, 19 Dec 2013, John-Paul Robinson wrote: What impact does rebooting nodes in a ceph cluster have on the health of the ceph cluster? Can it trigger rebalancing activities that then have to be undone once the node comes back up? I have a 4 node ceph cluster each node has 11 osds. There is a single pool with redundant storage. If it takes 15 minutes for one of my servers to reboot is there a risk that some sort of needless automatic processing will begin? By default, we start rebalancing data after 5 minutes. You can adjust this (to, say, 15 minutes) with mon osd down out interval = 900 in ceph.conf. sage I'm assuming that the ceph cluster can go into a not ok state but that in this particular configuration all the data is protected against the single node failure and there is no place for the data to migrate too so nothing bad will happen. Thanks for any feedback. ~jpr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rebooting nodes in a ceph cluster
On 20/12/13 13:51, Sage Weil wrote:
On Thu, 19 Dec 2013, John-Paul Robinson wrote:
What impact does rebooting nodes in a ceph cluster have on the health of
the ceph cluster? Can it trigger rebalancing activities that then have
to be undone once the node comes back up?
I have a 4 node ceph cluster each node has 11 osds. There is a single
pool with redundant storage.
If it takes 15 minutes for one of my servers to reboot is there a risk
that some sort of needless automatic processing will begin?
By default, we start rebalancing data after 5 minutes. You can adjust
this (to, say, 15 minutes) with
mon osd down out interval = 900
in ceph.conf.
sage
I'm assuming that the ceph cluster can go into a not ok state but that
in this particular configuration all the data is protected against the
single node failure and there is no place for the data to migrate too so
nothing bad will happen.
Thanks for any feedback.
Not directly related to Ceph, but you may want to investigate kexec[0]
('kexec-tools' package in
Debian derived distributions) in order to get your machines rebooting quicker.
It essentially
re-loads the kernel as the last step of the shutdown procedure, skipping over
the lengthy
BIOS/UEFI/controller firmware etc boot stages.
[0]: http://en.wikipedia.org/wiki/Kexec
--
David Clarke
Systems Architect
Catalyst IT
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rebooting nodes in a ceph cluster
So is it recommended to adjust the rebalance timeout to align with the time to reboot individual nodes? I didn't see this in my pass through the ops manual but maybe I'm not looking in the right place. Thanks, ~jpr On Dec 19, 2013, at 6:51 PM, Sage Weil s...@inktank.com wrote: On Thu, 19 Dec 2013, John-Paul Robinson wrote: What impact does rebooting nodes in a ceph cluster have on the health of the ceph cluster? Can it trigger rebalancing activities that then have to be undone once the node comes back up? I have a 4 node ceph cluster each node has 11 osds. There is a single pool with redundant storage. If it takes 15 minutes for one of my servers to reboot is there a risk that some sort of needless automatic processing will begin? By default, we start rebalancing data after 5 minutes. You can adjust this (to, say, 15 minutes) with mon osd down out interval = 900 in ceph.conf. sage I'm assuming that the ceph cluster can go into a not ok state but that in this particular configuration all the data is protected against the single node failure and there is no place for the data to migrate too so nothing bad will happen. Thanks for any feedback. ~jpr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
