Sure, As a warning, it is stupid complicated, and I am not certain exactly why it works the way it does, multicast to a unicast broker still confuses me. ;-)
I have a guest cluster that is using the Centos HA packages (which are basically a clone of what RedHat provides). In this setup I needed to be able to do guest fencing, as I wanted my guest cluster members to reboot rather than being fenced with the fence_scsi option. In my config I have 8 hosts and 8 guests. Usually there is only 1 guest on a host, but sometimes the guests migrate as we manually balance things. The mapping from guest to host is not always the same. The end result is that any guest can run fence_xvm -H guest-name and the correct guest system will be rebooted regardless of what host it is running on. The command fence_xvm -o list does not work right for me, but my nodes should be able to fence if necessary so it is mostly a solution.... If that is interesting read on. I drew inspiration from http://clusterlabs.org/wiki/Guest_Fencing http://www.daemonzone.net/e/3/ http://oss.clusterlabs.org/pipermail/pacemaker/2011-November/012321.html This basically got luci and my /etc/cluster/cluster.conf to agree that multicast and guest fencing is what my cluster should use. I installed fence-virt fence-virtd fence-virtd-multicast fence-virtd-libvirt libvirt-qmf matahari-broker on the host systems and fence-virt on the guests. On the hosts (after getting iptables and multicast packets to play nicely) and getting libvirt-qmf running (for me this was just starting the init.d script for it). I configured fence-virtd to use the libvirt-qpid backend with a host and port configuration option that pointed at the matahari-broker (which I think is a more recent qpidd) libvirt-qpid { uri = "qemu:///system"; host = "192.168.20.20"; port = "49000"; } This gets libvirt-qpid talking to the broker running on the host. I tested a configuration where all hosts used the same broker and it lets fencing occur. This was not optimal as if the host with the broker failed.. bad things. I then got matahari-broker working on each host so each libvirt-qpid has something to talk to. I then looked at http://matahari-dev.blogspot.com/2010/11/federation.html Which shows how to get the matahari-brokers to send messages to each other in a federation configuration. I changed the init.d script for matahari-broker to setup a route from the host executing the script to each broker on the other hosts. So I think what happens (still not certain) is that the multicast gets the information about what needs to be fenced to fence-virtd on each host. Each hosts fence-virtd then passes the info to the hosts matahari-broker, which sends the information to all the other brokers in the federation. Each broker then tells its local libvirt-qmf the information and if that libvirt-qmf actually has the guest to be fenced it does so, otherwise it ignores the request because the desired guest is not running. This also has the potential issue of each libvirt-qmf will be told to fence a guest once for each broker this includes the one that really has the guest. I believe (I have looked at no source code so I am just speculating) fence_xvm -o list does not work as expected because it runs with the first response it receives so it becomes a race between all the hosts as to which one will get there first. I hope that helps Adrian. I am happy to provide the iptables, fence_virt.conf, and /etc/sysconfig/matahari and matahari-broker files and my init.d script for matahari-broker that sets up all the routes. On 02/14/13 17:43, Andrew Beekhof wrote: > On Fri, Feb 15, 2013 at 6:42 AM, Adrian Allen<[email protected]> wrote: >>> Not quite yet. >> >>> I had a chat with Lon today and the "checkpoint" backend is close to >>> what we want but requires cman and openais. >>> Those two dependancies need to be removed. >> >>> Essentially you'll basically just be running corosync (instead of >>> qpid) on the hosts (no need for pacemaker). >>> If you happen to have some C skills I'd be happy to point you in the >>> right direction. >> >> Fair enough, I'll continue to limp along. It's not unlikely that our devs >> will work out a safe active/active arrangement in the near future and at >> that point I won't really care about stonith anyway. > > Someone else on the list was able to get the qpid approach working. > Something to do with tweaking the brokers... Max can you elaborate for > Adrian please? > >> >> Unfortunately I haven't twiddled with C in about 17 years, or I might take >> a crack at it. >> >> I do eagerly anticipate this working at some point though as there are a >> few other places we could probably use it. >> _______________________________________________ >> Linux-HA mailing list >> [email protected] >> http://lists.linux-ha.org/mailman/listinfo/linux-ha >> See also: http://linux-ha.org/ReportingProblems -- ================================================================ Max Planck: [email protected] (575)835-5935 (Office) Technical Director (575)835-7683 (Office) (575)838-0884 (Home) (505)620-2323 (Cellular) Institute for Complex Additive Systems Analysis New Mexico Institute of Mining & Technology Note: Privacy Act of 1974 - as amended applies. This memo may contain For Official Use Only information which must be protected IAW DoD 5400-11R. ================================================================ _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
