Hi again, On Tue, Sep 22, 2009 at 12:06:06PM -0400, Karl W. Lewis wrote: > As attachments, then.
Any possibility to prepare also a short README on how to setup the blades for stonith? That would be great. You don't have to go into details which are not specific to Egenera. Cheers, Dejan > Karl > > > On Tue, Sep 22, 2009 at 10:51 AM, Dejan Muhamedagic > <[email protected]>wrote: > > > Hi, > > > > On Tue, Sep 22, 2009 at 09:39:00AM -0400, Karl W. Lewis wrote: > > > On Mon, Sep 7, 2009 at 11:23 AM, Karl W. Lewis <[email protected] > > >wrote: > > > > > > > > > > > > > > > On Mon, Sep 7, 2009 at 10:16 AM, Dejan Muhamedagic < > > [email protected]>wrote: > > > > > > > >> Hi, > > > >> > > > >> On Wed, Aug 26, 2009 at 01:11:24PM -0400, Karl W. Lewis wrote: > > > >> > On Wed, Aug 26, 2009 at 10:58 AM, Dejan Muhamedagic < > > > >> [email protected]>wrote: > > > >> > > > > >> > > Hi, > > > >> > > > > > >> > > On Mon, Aug 24, 2009 at 11:35:15AM -0400, Karl W. Lewis wrote: > > > >> > > > I don't know if anyone else is running Heartbeat/Pacemaker on an > > > >> Egenera > > > >> > > > BladeFrame, but if they are, these scripts might be useful to > > help > > > >> set up > > > >> > > > stonith. > > > >> > > > > > > >> > > > The Egeneras use an IPMI arrangement to control the various > > > >> processor > > > >> > > > blades, and these routines take advantage of that. > > > >> > > > > > >> > > Did you try ipmilan or external/ipmi too? > > > >> > > > > >> > > > > >> > I did not, but a quick review leads me to believe that the egenera > > frame > > > >> > does not count as an impi "device". One might be able to make that > > > >> work, > > > >> > but you'd have to modify the ipmi script until it looked much like > > this > > > >> > example, or, you'd have to play very. very hard with the egnera > > > >> c-blades, > > > >> > and I can't imagine anyone would want to do that. > > > >> > > > >> Well, either they are IPMI compliant or not, i.e. our ipmi > > > >> stonith modules should work without modifications. I think it's > > > >> worth trying. At any rate, I think that this is a different > > > >> solution worth pursuing, since it would reduce the configuration > > > >> quite a bit. > > > >> > > > > > > > > I've not forgotten this task, but it's been very busy at work and I'm > > doing > > > > more research. The ipmi commands do not work the way I expected them > > to, > > > > (writing to "off" just resets the blade), so I'm looking at the native > > > > Egenera PAN Manager commands. > > > > > > > >> > > > >> > > > This may not be the best method of doing it, but it does appear > > to > > > >> work. > > > >> > > > It is not fast. What it lacks in speed, however, it makes up > > for in > > > >> > > > flexibility. It can figure out which c-blade is the master, and > > > >> then > > > >> > > figure > > > >> > > > out which p-blade is running the server to be stonithed, and > > then > > > >> > > stonith, > > > >> > > > it. The framework for this was an existing external stonith > > script > > > >> that > > > >> > > I > > > >> > > > simply modified to meet my needs. > > > >> > > > > > > >> > > > The two short shell scripts at the top need to live in > > /root/bin/ on > > > >> both > > > >> > > > > > >> > > I guess you wanted to say $HOME/bin. Note that the root user may > > > >> > > not always have the same home directory. Perhaps it would be > > > >> > > better to go with an absolute path such as /usr/local/bin. > > > >> > > > > >> > > > > >> > I actually put them in root's directory with malice aforethought, > > but > > > >> now > > > >> > that you've asked the question... I guess I can't defend that > > decision. > > > >> I > > > >> > use the user root because that's the only user on the c-blades under > > > >> > egenera's setup. One could add another user, but I can't advocate > > > >> changing > > > >> > the config of the c-blades. > > > >> > > > > >> > > > > >> > > > > > >> > > > > > >> > > > control blades. The cluster servers need to have ssh keys and > > any > > > >> > > required > > > >> > > > firewall rules setup such that root on the cluster servers can > > ssh > > > >> to the > > > >> > > > c-blades. > > > >> > > > > > >> > > There can be more than one? Is that covered in any way by the > > > >> > > script, i.e. does it try all available? > > > >> > > > > >> > > > > >> > Yes. Just so. There are two c-baldes; one is master and one is > > > >> standby. I > > > >> > *think* that the master is the only one that can issue the ipmi > > commands > > > >> and > > > >> > get a reaction. I've not experimented with that, however. (I hit a > > > >> > solution that worked, and I stopped experimenting.) > > > >> > > > >> It would be good if you do some :) You don't want any surprises > > > >> when it comes to fencing. > > > >> > > > > > > > > Right. Surprises are for Birthday Parties, presents, and such, not so > > much > > > > for servers. > > > > > > > > Karl > > > > > > > > > > > > > > > >> > > > >> > The script tracks down > > > >> > which of the two is master and then talks to that server. One > > could > > > >> avoid > > > >> > that by giving both c-blades the same set of rsa host keys, and then > > > >> just > > > >> > talking to the virtual IP Address that the master blade, which ever > > of > > > >> the > > > >> > two it is, is answering on. Again, I am disinclined to modify the > > > >> egenera > > > >> > setup that way. I tried telling ssh to ignore the key verification, > > but > > > >> it > > > >> > still updates the knownhosts file when the script logs in, so it'll > > > >> throw > > > >> > errors the next time it tries if the master has moved. > > > >> > > > >> Right. Isn't there an option to tell ssh not to update the > > > >> knownhosts file. > > > >> > > > >> > > > If anyone else is already doing this and has a better way, I'd > > be > > > >> happy > > > >> > > to > > > >> > > > hear about it. > > > >> > > > > > > >> > > > Karl > > > >> > > > > > >> > > > ### THESE TWO SHELL SCRIPTS ARE EXPECTED IN /root/bin ON BOTH > > > >> c-blades > > > >> > > > ### THAT WAY THE SCRPIT THAT FOLLOWS WILL WORK REGARDLESS OF > > WHICH > > > >> > > > ### EGENERA c-blade IS PAN-MASTER > > > >> > > > > > > >> > > > #!/bin/bash > > > >> > > > # get_master.sh - returns the name of the master control blade > > in > > > >> the > > > >> > > > PAN. > > > >> > > > > > > >> > > > MASTER=`/opt/panmgr/bin/esh pan | grep ^c | grep Master | cut -c > > 2` > > > >> > > > echo "cblade-${MASTER}" > > > >> > > > #CUT HERE > > > >> ############################################################# > > > >> > > > > > > >> > > > #!/bin/bash > > > >> > > > > > > >> > > > # get_processor_blade.sh - identifies the blade running a given > > > >> pserver. > > > >> > > > > > >> > > This name doesn't exactly match the references in the script. > > > >> > > > > >> > > > > >> > > > > >> > :%s/_blade/-blade/ > > > >> > > > > >> > I'm a lousy typist.... > > > >> > > > > >> > > > > >> > > > > >> > > > > > >> > > > > > >> > > > BLADE=`/opt/panmgr/bin/esh pan | grep ^${1}` > > > >> > > > echo ${BLADE} | sed -e "s/^.*MyBladeFrame\/\(.\)\(.\) > > .*/a\1-\2/" > > > >> > > > #CUT HERE > > > >> ############################################################# > > > >> > > > > > > >> > > > #!/bin/bash > > > >> > > > # > > > >> > > > # External STONITH module for Egenera. > > > >> > > > # > > > >> > > > # Copyright (c) 2004 SUSE LINUX AG - Lars Marowsky-Bree < > > > >> [email protected]> > > > >> > > > # Copyright (c) 2009 Karl W. Lewis <[email protected]> > > > >> > > > # This program is free software; you can redistribute > > > >> > > > #it and/or modify # it under the terms of version 2 of the GNU > > > >> General > > > >> > > > #Public License as # published by the Free Software Foundation. > > > >> > > > # > > > >> > > > # This program is distributed in the hope that it would be > > useful, > > > >> but > > > >> > > > # WITHOUT ANY WARRANTY; without even the implied warranty of > > > >> > > > # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > > >> > > > # > > > >> > > > # Further, this software is distributed without any warranty > > that it > > > >> is > > > >> > > > # free of the rightful claim of any third person regarding > > > >> infringement > > > >> > > > # or the like. Any license provided herein, whether implied or > > > >> > > > # otherwise, applies only to this software file. Patent > > licenses, > > > >> if > > > >> > > > # any, provided herein do not apply to combinations of this > > program > > > >> with > > > >> > > > # other software, or any other product whatsoever. > > > >> > > > # > > > >> > > > # You should have received a copy of the GNU General Public > > License > > > >> > > > # along with this program; if not, write the Free Software > > > >> Foundation, > > > >> > > > # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. > > > >> > > > # > > > >> > > > > > > >> > > > SSH_COMMAND="/usr/bin/ssh" > > > >> > > > > > > >> > > > #LOG=/var/log/egenera-stonith.log > > > >> > > > #echo `date` >> $LOG > > > >> > > > #echo $hostlist >> $LOG > > > >> > > > #echo $1 >> $LOG > > > >> > > > #echo $2 >> $LOG > > > >> > > > #echo "executing--------------" >> $LOG > > > >> > > > > > > >> > > > # Rewrite the hostlist to accept "," as a delimeter for > > hostnames > > > >> too. > > > >> > > > hostlist=`echo $hostlist | tr ',' ' '` > > > >> > > > > > > >> > > > is_host_up() { > > > >> > > > for j in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 > > > >> > > > do > > > >> > > > if > > > >> > > > ping -w1 -c1 "$1" >/dev/null 2>&1 > > > >> > > > then > > > >> > > > sleep 10 > > > >> > > > else > > > >> > > > return 1 > > > >> > > > fi > > > >> > > > done > > > >> > > > return 0 > > > >> > > > } > > > >> > > > > > >> > > This is from external/ssh and used only if the parameter > > > >> > > livedangerously is set to yes. Doesn't strike me as a very robust > > > >> > > way to check if a node is running. Is there another way? Using > > > >> > > the cblades sounds like a most natural way. > > > >> > > > > >> > > > > >> > Yes. You're clearly right. By way of excuse I'd claim I was rather > > > >> > narrowly focused on getting a server fenced and therefore I ignored > > > >> other > > > >> > functions. If you'll tolerate a "version 2, (or, I suppose v0.02), > > I'll > > > >> try > > > >> > again. It'll take me a few hours to fix and test, I'll try again. > > (A > > > >> quick > > > >> > experiment reveals that yes, the master blade does indeed know the > > > >> status of > > > >> > a given processor blade.) > > > >> > > > >> Great. > > > >> > > > >> > > > MASTER_BLADE=`$SSH_COMMAND cblade-1 "bin/get_master.sh" | tail > > -1` > > > >> > > > > > > >> > > > case $1 in > > > >> > > > gethosts) > > > >> > > > for h in $hostlist ; do > > > >> > > > echo $h > > > >> > > > done > > > >> > > > exit 0 > > > >> > > > ;; > > > >> > > > on) > > > >> > > > for h in $hostlist > > > >> > > > do > > > >> > > > if > > > >> > > > [ "$h" != "$2" ] > > > >> > > > then > > > >> > > > continue > > > >> > > > else > > > >> > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE > > > >> > > "bin/get_processor-blade.sh $h"` > > > >> > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' > > > > >> > > /proc/egenera/ipmi/node/$P_BLADE/control/on" > > > >> > > > fi > > > >> > > > sleep 30 > > > >> > > > if > > > >> > > > is_host_up $h > > > >> > > > then > > > >> > > > exit 1 > > > >> > > > else > > > >> > > > exit 0 > > > >> > > > fi > > > >> > > > done > > > >> > > > exit 0 > > > >> > > > ;; > > > >> > > > off) > > > >> > > > for h in $hostlist > > > >> > > > do > > > >> > > > if > > > >> > > > [ "$h" != "$2" ] > > > >> > > > then > > > >> > > > continue > > > >> > > > else > > > >> > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE > > > >> > > "bin/get_processor-blade.sh $h"` > > > >> > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' > > > > >> > > /proc/egenera/ipmi/node/$P_BLADE/control/off" > > > >> > > > fi > > > >> > > > done > > > >> > > > exit 0 > > > >> > > > ;; > > > >> > > > reset) > > > >> > > > for h in $hostlist > > > >> > > > do > > > >> > > > if > > > >> > > > [ "$h" != "$2" ] > > > >> > > > then > > > >> > > > continue > > > >> > > > else > > > >> > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE > > > >> > > "bin/get_processor-blade.sh $h"` > > > >> > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' > > > > >> > > /proc/egenera/ipmi/node/$P_BLADE/control/nmi" > > > >> > > > fi > > > >> > > > sleep 20 > > > >> > > > if > > > >> > > > is_host_up $h > > > >> > > > then > > > >> > > > exit 1 > > > >> > > > else > > > >> > > > exit 0 > > > >> > > > fi > > > >> > > > done > > > >> > > > exit 1 > > > >> > > > ;; > > > >> > > > > > >> > > on), off), and reset) cases are very similar, they should be made > > > >> > > to share some code. Also, the sleep values look too ephemeral for > > > >> > > my taste. There should be a relatively tight loop to check the > > > >> > > host status. > > > >> > > > > > >> > > > > >> > I'm not sure yet what to do about the "ephemeral" sleep values. I > > chose > > > >> > arbitrarily long waits because egenera blades take a surprisingly > > long > > > >> time > > > >> > to boot. Your point is well taken, however, the code can be > > improved. > > > >> > > > >> Looking forward to it :) > > > >> > > > >> > > > status) > > > >> > > > if > > > >> > > > [ -z "$hostlist" ] > > > >> > > > then > > > >> > > > exit 1 > > > >> > > > fi > > > >> > > > for h in $hostlist > > > >> > > > do > > > >> > > > if > > > >> > > > ping -w1 -c1 "$h" 2>&1 | grep "unknown host" > > > >> > > > > > >> > > The status should return the status of the device, not the status > > > >> > > of a node. > > > >> > > > > >> > > > > >> > ?? I must have misunderstood the existing code. Hmmmmm. The > > status of > > > >> > this device is always "0" if there's a hostlist and if $MASTER_BLADE > > is > > > >> set, > > > >> > I think. > > > >> > > > >> And if the c-blade is reachable. Something like > > > >> "ssh $MASTER_BLADE true" should do. > > > >> > > > >> > Thank you for taking the itme to look this over. I'll clean it up > > some > > > >> and > > > >> > try again. > > > >> > > > >> Many thanks. > > > >> > > > >> Cheers, > > > >> > > > >> Dejan > > > >> > > > >> > Be well, > > > >> > > > > >> > Karl > > > >> > > > > >> > > > > > >> > > > > > >> > > Cheers, > > > >> > > > > > >> > > Dejan > > > >> > > > > > >> > > > > > > > > > > > I've cleaned this up some, and reworked much of it. It has been tested > > in > > > production for a couple of weeks now, and it works properly as near as I > > can > > > tell. (It has properly restarted a misbehaving server.) > > > > > > I could not see a way to get the existing IPMI script to work with > > Egenera, > > > and I have abandoned using anything other than native Egenera commands > > to > > > control the blades. These scripts require that ssh keys are set up for > > user > > > root between the cluster servers and the c-blades. > > > > OK. > > > > > Code: > > > > Can you please send the scripts as attachments. The mail agents > > kill the formatting. > > > > > #The two bash scripts here at the top go on each of the Egenera C-Blades. > > > #the STONITH module goes in /usr/lib/stonith/plugins/external/egenera on > > > #each of the servers in the cluster. > > > > These two scripts will go to the doc part of the package. > > > > Cheers, > > > > Dejan > > > > > ############################### cut here > > > ###################################### > > > #!/bin/bash > > > # get_status.sh - returns the status of a pServer, given the LPAN and > > > pServer > > > > > > #called as: > > > # get_status.sh pServer LPAN_Name UP > > > # or > > > # get_status.sh pServer LPAN_Name DOWN > > > #depending on what you want. This only works of the pServer is running > > the > > > PAN Agent > > > #because until the agent comes up, the server does not count as "Up" > > > > > > if [ ${3} = "UP" ] > > > then > > > CONDITION=`/opt/panmgr/bin/esh pserver -l ${2}/${1} | grep ^${1} > > | > > > grep "Booted" | grep "Up"` > > > if [ -z "${CONDITION}" ] > > > then > > > echo "Down" # No Joy > > > else > > > echo "Up" # Joy! > > > fi > > > else # it should be the case that ${3} = "DOWN" > > > CONDITION=`/opt/panmgr/bin/esh pserver -l ${2}/${1} | grep ^${1} > > | > > > grep "Shutdown"` > > > if [ -z "${CONDITION}" ] > > > then > > > echo "Up" # No Joy > > > else > > > echo "Down" # Joy! > > > fi > > > fi > > > > > > ############################### cut here > > > ###################################### > > > > > > #!/bin/bash > > > # get_master.sh - returns the name of the master control blade in the > > PAN. > > > > > > MASTER=`/opt/panmgr/bin/esh pan | grep ^c | grep Master | cut -c 2` > > > echo "cblade-${MASTER}" > > > > > > ############################### cut here > > > ###################################### > > > > > > #!/bin/bash > > > # > > > # External STONITH module for Egenera pServers. > > > # > > > # Copyright (c) 2004 SUSE LINUX AG - Lars Marowsky-Bree <[email protected]> > > > # Copyright (c) 2009 Karl W. Lewis <karl.w.lewis - AT - gmail.com> > > > # > > > # This program is free software; you can redistribute it and/or modify > > > # it under the terms of version 2 of the GNU General Public License as > > > # published by the Free Software Foundation. > > > # > > > # This program is distributed in the hope that it would be useful, but > > > # WITHOUT ANY WARRANTY; without even the implied warranty of > > > # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > > # > > > # Further, this software is distributed without any warranty that it is > > > # free of the rightful claim of any third person regarding infringement > > > # or the like. Any license provided herein, whether implied or > > > # otherwise, applies only to this software file. Patent licenses, if > > > # any, provided herein do not apply to combinations of this program with > > > # other software, or any other product whatsoever. > > > # > > > # You should have received a copy of the GNU General Public License > > > # along with this program; if not, write the Free Software Foundation, > > > # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. > > > # > > > > > > SSH_COMMAND="/usr/bin/ssh -q -x -n -l root" > > > GET_MASTER="/root/bin/get_master.sh" > > > GET_STATUS="/root/bin/get_status.sh" > > > PSERVER_COMMAND="/opt/panmgr/bin/esh pserver" > > > > > > #These TIMEOUT values were obtained from the 'pserver -l pan/pserver' > > > command > > > # on a given Egenera frame under "Timeouts" YMMV. > > > > > > BOOT_TIMEOUT=270 > > > REBOOT_TIMEOUT=450 > > > SHUTDOWN_TIMEOUT=180 > > > > > > TIMEOUT=${REBOOT_TIMEOUT} > > > STATUS="UNKNOWN" > > > > > > #LOG=/var/log/egenera-stonith.log > > > #echo `date` >> ${LOG} > > > #echo ${hostlist} >> ${LOG} > > > #echo ${lpan_name} >> ${LOG} > > > #echo $1 >> ${LOG} > > > #echo $2 >> ${LOG} > > > #echo "executing--------------" >> ${LOG} > > > > > > # Rewrite the hostlist to accept "," as a delimeter for hostnames too. > > > hostlist=`echo ${hostlist} | tr ',' ' '` > > > > > > is_host_up() { > > > TIME=$((${TIMEOUT}/5)) > > > for j in 1 2 3 4 5 6 > > > do > > > STATUS=`${SSH_COMMAND} ${MASTER_BLADE} "${GET_STATUS} ${h} > > > ${lpan_name} UP"` > > > if [ "${STATUS}" = "Down" ] > > > then > > > sleep ${TIME} > > > else > > > SYSTEM_STATUS="On Target" > > > return 1 > > > fi > > > done > > > return 0 > > > } > > > > > > is_host_down() { > > > TIME=$((${TIMEOUT}/5)) > > > for j in 1 2 3 4 5 6 > > > do > > > STATUS=`${SSH_COMMAND} ${MASTER_BLADE} "${GET_STATUS} ${h} > > > ${lpan_name} DOWN"` > > > if [ "${STATUS}" = "Up" ] > > > then > > > sleep ${TIME} > > > else > > > SYSTEM_STATUS="On Target" > > > return 1 > > > fi > > > done > > > return 0 > > > } > > > > > > MASTER_BLADE=`${SSH_COMMAND} cblade-1 "${GET_MASTER}" | tail -1` > > > > > > if [ -z "${MASTER_BLADE}" ] > > > then > > > MASTER_BLADE=`${SSH_COMMAND} cblade-2 "${GET_MASTER}" | tail -1` > > > fi > > > > > > case ${1} in > > > gethosts) > > > for h in ${hostlist} > > > do > > > echo ${h} > > > done > > > exit 0 > > > ;; > > > on|off|reboot) > > > for h in ${hostlist} > > > do > > > if [ "${h}" != "${2}" ] > > > then > > > continue > > > else > > > if [ "${1}" = "on" ] > > > then > > > CMD="--boot" > > > elif [ "${1}" = "off" ] > > > then > > > CMD="--shutdown --force" > > > else #must be "reboot" > > > CMD="--reboot --force" > > > fi > > > ${SSH_COMMAND} ${MASTER_BLADE} "${PSERVER_COMMAND} ${CMD} > > > ${lpan_name}/${h}" > > > if [ "${1}" = "on" ] > > > then > > > TIMEOUT=${BOOT_TIMEOUT} > > > elif [ "${1}" = "reboot" ] > > > then > > > TIMEOUT=${REBOOT_TIMEOUT} > > > else > > > TIMEOUT=${SHUTDOWN_TIMEOUT} > > > fi > > > if [ "${1}" = "off" ] > > > then > > > is_host_down ${h} > > > else > > > is_host_up ${h} > > > fi > > > fi > > > if [ "${SYSTEM_STATUS}" = "On Target" ] > > > then > > > exit 0 > > > else > > > exit 1 > > > fi > > > done > > > exit 1 # We got here only if the host was not in the list.... > > > ;; > > > status) > > > if [ -z "${hostlist}" ] || [ -z "${lpan_name}" ] || [ -z > > > "${MASTER_BLADE}" ] > > > then > > > exit 1 > > > fi > > > exit 0 > > > ;; > > > getconfignames) > > > echo "hostlist lpan_name" > > > exit 0 > > > ;; > > > getinfo-devid) > > > echo "Egenera STONITH device" > > > exit 0 > > > ;; > > > getinfo-devname) > > > echo "Egenera STONITH external device" > > > exit 0 > > > ;; > > > getinfo-devdescr) > > > echo "Egenera-based Linux host reset" > > > exit 0 > > > ;; > > > getinfo-devurl) > > > echo "http://www.egenera.com"; > > > exit 0 > > > ;; > > > getinfo-xml) > > > cat << EGENERAXML > > > <parameters> > > > <parameter name="hostlist" unique="1" required="1"> > > > <content type="string" /> > > > <shortdesc lang="en"> > > > Hostlist > > > </shortdesc> > > > <longdesc lang="en"> > > > The list of hosts that the STONITH device controls > > > </longdesc> > > > </parameter> > > > <parameter name="lpan_name" unique="1" required="1"> > > > <content type="string" /> > > > <shortdesc lang="en"> > > > LPAN_Name > > > </shortdesc> > > > <longdesc lang="en"> > > > The name of the LPAN that the cluster blades, that this STONITH device > > > controls, live in > > > </longdesc> > > > </parameter> > > > </parameters> > > > EGENERAXML > > > exit 0 > > > ;; > > > *) > > > exit 1 > > > ;; > > > esac > > > _______________________________________________ > > > Linux-HA mailing list > > > [email protected] > > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > > See also: http://linux-ha.org/ReportingProblems > > _______________________________________________ > > Linux-HA mailing list > > [email protected] > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > See also: http://linux-ha.org/ReportingProblems > > > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
