The mons work best when they know absolutely everything. If they know that osd.3 was down 40 seconds before osd.2 that means that if a write was stilling happening while osd.2 was still up, the mons have record of it in the maps and when osd.3 comes up, it can get what it needs from the other osds. Mons are the keepers of maps, epochs, and everything important to know about a cluster. If you're using encryption on your OSDs, the mons keep track of the keys to decrypt the osds iirc.
Even if you aren't using encryption, the OSDs check with the Mons when they first start to know what the most recent map is. If they can't communicate with a Mon, they will fail to start and die. Last down/first up ensures that the mons know everything and is the safest way to handle a cluster shutdown. Yes Ceph can usually handle full system power-offs with no proper order or having too many of something shutdown while the rest of the cluster is running, but most people try to avoid disaster scenarios if they can help it. On Mon, May 7, 2018 at 9:48 PM Bryan Henderson <[email protected]> wrote: > There is a lot of advice around on shutting down a Ceph cluster that says > to shut down the OSDs before the monitors and bring up the monitors before > the OSDs, but no one explains why. > > I would have thought it would be better to shut down the monitors first and > bring them up last, so they don't have to witness all the interim states > with > OSDs down. And it should make the noout, nodown, etc. settings > unnecessary. > > So what am I missing? > > Also, how much difference does it really make? Ceph is obviously designed > to > tolerate any sequence of failures and recoveries of nodes, so how much risk > would I be taking if I just haphazardly killed everything instead of > orchestrating a shutdown? > > -- > Bryan Henderson San Jose, California > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
