Isn't the new leveldb tuning part of cuttlefish. Stefan
Am 18.04.2013 um 19:40 schrieb Joao Eduardo Luis <[email protected]>: > On 04/18/2013 05:28 PM, Gregory Farnum wrote: >> On Wed, Apr 17, 2013 at 7:40 AM, Guido Winkelmann >> <[email protected]> wrote: >>> Hi, >>> >>> I just tried upgrading parts of our experimental ceph cluster from 0.56.1 to >>> 0.60, and it looks like the new mon-daemon from 0.60 cannot talk to those >>> from >>> 0.56.1 at all. >>> >>> Long story short, we had to move some hardware around and during that time I >>> had to shrink the cluster to one single machine. My plan was to expand it to >>> three machines again, so that I would again have 3 mons and 3 osds, as >>> before. >>> I just installed the first new machine, going straight for 0.60, but leaving >>> the remaining old one at 0.56.1. I added the new mon to the mon map >>> according >>> to the documentation and started the new mon daemon, but the mon-cluster >>> wouldn't achieve quorum. In the logs for the new mon, I saw the following >>> line >>> repeated a lot: >>> >>> 0 -- 10.6.224.129:6789/0 >> 10.6.224.131:6789/0 pipe(0x2da5ec0 sd=20 :37863 >>> s=1 pgs=0 cs=0 l=0).connect protocol version mismatch, my 10 != 9 >>> >>> The old mon had no such lines in its log. >>> >>> I could only solve this by shutting down the old mon and upgrading it to >>> 0.60 >>> as well. >>> >>> It looks to me like this means rolling upgrades without downtime won't be >>> possible from bobtail to cuttlefish. Is that correct? >> >> If the cluster is in good shape, this shouldn't actually result in >> downtime. Do a rolling upgrade of your monitors, and then when a >> majority of them are on Cuttlefish they'll switch over to form the >> quorum — the "downtime" being the period a store requires to update, >> which shouldn't be long, and it will only be the monitors that are >> inaccessible (unless it takes a truly ridiculous time for the >> upgrade). All the rest of the daemons you can do rolling upgrades on >> just the same as before. > > Another potential source of delay would be the synchronization process > triggered when a majority of monitors have been upgraded. > > Say you have 5 monitors. > > You upgrade two while the cluster is happily running: the stores are > converted, which may take longer if the store is huge [1], but you get your > monitors ready to join the quorum as soon as a third member is upgraded. > > During this time, your cluster kept on going, with more versions being > created. > > And then you decide to upgrade the third monitor. It will go through the > same period of downtime as the other two monitors -- which as Greg said > shouldn't be long, but may be if your stores are huge [1] -- and this will be > the bulk of your downtime. > > However, as the cluster kept on going, there's a chance that the first two > monitors to be upgraded will have fallen out of sync with the more recent > cluster state. That will trigger a store sync, which shouldn't take long > either, but this is somewhat bound by the store size and the amount of > versions that were created in-between. You might even be lucky enough, and > go through with the whole thing in no time and the sync might not even be > necessary (there's another mechanism to handle catch-up when the monitors > haven't drifted that much). > > Anyway, when you are finally upgrading the third monitor (out of 5), that is > going to break quorum, so it would probably be wise to just upgrade the > remaining monitors all at once. > > > [1] - With the new leveldb tuning this might not even be an issue. > > -Joao > > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
