On Tue, Jan 16, 2018 at 6:07 AM Alex Gorbachev <[email protected]> wrote:
> I found a few WAN RBD cluster design discussions, but not a local one, > so was wonderinng if anyone has experience with a resilience-oriented > short distance (<10 km, redundant fiber connections) cluster in two > datacenters with a third site for quorum purposes only? > > I can see two types of scenarios: > > 1. Two (or even number) of OSD nodes at each site, 4x replication > (size 4, min_size 2). Three MONs, one at each site to handle split > brain. > > Question: How does the cluster handle the loss of communication > between the OSD sites A and B, while both can communicate with the > quorum site C? It seems, one of the sites should suspend, as OSDs > will not be able to communicate between sites. > Sadly this won't work — the OSDs on each side will report their peers on the other side down, but both will be able to connect to a live monitor. (Assuming the quorum site holds the leader monitor, anyway — if one of the main sites holds what should be the leader, you'll get into a monitor election storm instead.) You'll need your own netsplit monitoring to shut down one site if that kind of connection cut is a possibility. > > 2. 3x replication for performance or cost (size 3, min_size 2 - or > even min_size 1 and strict monitoring). Two replicas and two MONs at > one site and one replica and one MON at the other site. > > Question: in case of a permanent failure of the main site (with two > replicas), how to manually force the other site (with one replica and > MON) to provide storage? I would think a CRUSH map change and > modifying ceph.conf to include just one MON, then build two more MONs > locally and add? > Yep, pretty much that. You won't need to change ceph.conf to just one mon so much as to include the current set of mons and update the monmap. I believe that process is in the disaster recovery section of the docs. -Greg > > -- > Alex Gorbachev > Storcium > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
