Welcome to the addiction^h^h^hcommunity. answers in-line, and note that I am not an active pacemaker user, but am a RHCS user, so some of my answers might be wrong.
On 01/15/2013 10:27 PM, Alex Sudakar wrote: > Hello, > > Having done some reading over the past week on Pacemaker and Corosync > I'd like to ask a few questions about points on which I'm still hazy. > > I'm trying to set up a very simple active/passive cluster of two > nodes. The nodes will be KVM VM guests, each of the two running on > separate physical hosts/hypervisors. The operating system in all four > cases will be Redhat Enterprise Linux 6.2. I'll be using the RPMs > that are part of the RHEL6.2 distribution - Pacemaker 1.1.6 with > Corosync 1.4.1. In case it matters; Red Hat supports corosync + cman + rgmanager in RHEL 6.x. Pacemaker is scheduled to replace cman/rgmanager in RHEL 7, but until then, it's in tech-preview only and doesn't get updates between y-stream updates. Also; Why 6.2 when 6.3 has been out for a long time? > This will - at the start - be a "poor man's" cluster - no SBD > shared-disk fencing, no hardware controllers for STONITH, no dedicated > network links. Basically the two physical machines will have just > one, maybe two, network interfaces. I'm happy to accept that some > answers might be along the lines of "you can't have a serious > cluster/STONITH in such circumstances". That may be part of my > problem; I'm trying to determine when STONITH might make a difference, > and when it might be quite superfluous for my little cluster. > > Right now my cluster is operational with 'stonith-enabled="false"'. > I've worked with clusters before, but STONITH is new to me, hence my > indecision. STONITH (aka fencing) is used to avoid split-brains. These are possible when the two nodes lose contact with one another and continue providing services on their own. At best, you screw up your users. At worst, you destroy data. The docs are right; Fencing is really really important. I'd go so far as to say that your cluster is fatally flawed without proper fencing. > Q1. Is STONITH of any use in my simple cluster? Yes, always. > If each node has only one or two network connections to the other, and > they will be using those very same connections to employ STONITH to > bring down the other KVM node, is it worthwhile to set up STONITH? I > gather that, if the networks fail and then each node believes the > other is dead, they will both use STONITH to bring down the other. > But the STONITH will likewise fail. With fencing, a failed fence action will leave the cluster hung. This is bad, but a lot better than a split-brain. I personally avoid this by using Active/Passive bonds, with each link in a different switch, plus two fence devices. I put IPMI on the first switch and PDU fence devices on the other switch. This way, at least one fence device is available, no matter what. I understand this is outside your current resources, but I would still implement fencing. > But STONITH is also potentially of use in other (rarer?) cases; if > something is wrong with the Corosync configuration of one node, say; > or if a resource can't be STOPPED on one node for non-network reasons. > In which case - if there's no actual _network_ problem - STONITH will > function correctly. > > So should I set up STONITH anyway? Yes. always. > Q2. If a STONITH agent fails to bring down a node, will the > originating node continue to function as a cluster participant? > > In the example above, if the network connection(s) fail between the > two cluster nodes I believe that each will try and bring down the > other with STONITH. If each STONITH action _fails_ - because the > network is down, and the STONITH agents will be configured to use the > same network - will the nodes (both) try to run the cluster's > resources? You would have to lose the switch. The lose of one link alone, say the network cable or interface used by corosync on one node dies, the other node will still successfully reach the failed node's fence device and kill it. This is why fence devices must exist outside of the target node. > I understand that, if they do, the cluster will be running in a > split-brain condition. But that's actually what I would prefer; I'd > rather have both nodes running the application in split-brain mode > rather than _neither_ running the application. (The application is > largely 'read-only'; it's more important for readers to have access to > it during any outage than protect it from split-brain; reconciling the > split-brain data afterwards will be a simple case of picking one > node's data to supersede the other's). "Largely" read-only is not entirely read-only. If it was truly read-only, then why use a cluster at all? > If the answer to this is "a node will suicide if its STONITH agents > fail to bring down other cluster nodes" then I guess it will follow > that my answer to Q1 will be "don't set up STONITH, then". I suspect that not having fencing will end up badly for you. I can't imagine a case where a split-brain is preferable. Maybe others can. > Q3. What STONITH agent do I use to control KVM cluster nodes? If your nodes are KVM VMs, you can use either 'fence_xvm' or 'fence_virsh'. > Looking at agents that can be used to shoot KVM VM guests I believe I > have these options: > > 3.1 fence_virtd using its libvirt backend; except that only works for > a single physical host/hypervisor, so I can't use it. > > 3.2 fence_virtd using its libvirt-qpid backend. However I haven't > been able to find any examples of how this (a libvirt-qpid daemon is > involved? Configuration of qpid/QMF on the hosts?) is set up. I saw > that Adrian Allen posted a message to the Linux-HA mailing list a few > days ago (the 1/9) asking about setting up qmf/qpid with fence_virt; > like him I haven't found any examples of people using it. > > 3.3 I _have_ found people referencing an 'external/libvirt' agent. > Does anyone know where I can find this agent? In RHEL, all the fence agents are merged into the 'fence-agents' RPM. > 3.4 I've also seen references to the 'fence_virsh' agent, which comes > with the RHEL 6.2 fence RPMs. This agent seems to ssh from the > originating node to the to-be-killed node and then run virsh to reboot > the VM. All hypervisor-based fence devices talk to the host server, not the target node. > If I do go ahead with STONITH I assume I'll be using one of the above > agents, unless someone recommends another. Can anyone point me to > guidelines on configuring qpid/QMF with fence_virtd (3.2)? Or tell me > where I can find external/libvirt (3.3)? > > Many thanks for any help! Take a look at fence_xvm as well, it's multicast-based. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
