On Mon, Sep 7, 2009 at 10:16 AM, Dejan Muhamedagic <[email protected]>wrote:
> Hi, > > On Wed, Aug 26, 2009 at 01:11:24PM -0400, Karl W. Lewis wrote: > > On Wed, Aug 26, 2009 at 10:58 AM, Dejan Muhamedagic <[email protected] > >wrote: > > > > > Hi, > > > > > > On Mon, Aug 24, 2009 at 11:35:15AM -0400, Karl W. Lewis wrote: > > > > I don't know if anyone else is running Heartbeat/Pacemaker on an > Egenera > > > > BladeFrame, but if they are, these scripts might be useful to help > set up > > > > stonith. > > > > > > > > The Egeneras use an IPMI arrangement to control the various processor > > > > blades, and these routines take advantage of that. > > > > > > Did you try ipmilan or external/ipmi too? > > > > > > I did not, but a quick review leads me to believe that the egenera frame > > does not count as an impi "device". One might be able to make that > work, > > but you'd have to modify the ipmi script until it looked much like this > > example, or, you'd have to play very. very hard with the egnera c-blades, > > and I can't imagine anyone would want to do that. > > Well, either they are IPMI compliant or not, i.e. our ipmi > stonith modules should work without modifications. I think it's > worth trying. At any rate, I think that this is a different > solution worth pursuing, since it would reduce the configuration > quite a bit. > I've not forgotten this task, but it's been very busy at work and I'm doing more research. The ipmi commands do not work the way I expected them to, (writing to "off" just resets the blade), so I'm looking at the native Egenera PAN Manager commands. > > > > > This may not be the best method of doing it, but it does appear to > work. > > > > It is not fast. What it lacks in speed, however, it makes up for in > > > > flexibility. It can figure out which c-blade is the master, and then > > > figure > > > > out which p-blade is running the server to be stonithed, and then > > > stonith, > > > > it. The framework for this was an existing external stonith script > that > > > I > > > > simply modified to meet my needs. > > > > > > > > The two short shell scripts at the top need to live in /root/bin/ on > both > > > > > > I guess you wanted to say $HOME/bin. Note that the root user may > > > not always have the same home directory. Perhaps it would be > > > better to go with an absolute path such as /usr/local/bin. > > > > > > I actually put them in root's directory with malice aforethought, but now > > that you've asked the question... I guess I can't defend that decision. > I > > use the user root because that's the only user on the c-blades under > > egenera's setup. One could add another user, but I can't advocate > changing > > the config of the c-blades. > > > > > > > > > > > > > > control blades. The cluster servers need to have ssh keys and any > > > required > > > > firewall rules setup such that root on the cluster servers can ssh to > the > > > > c-blades. > > > > > > There can be more than one? Is that covered in any way by the > > > script, i.e. does it try all available? > > > > > > Yes. Just so. There are two c-baldes; one is master and one is standby. > I > > *think* that the master is the only one that can issue the ipmi commands > and > > get a reaction. I've not experimented with that, however. (I hit a > > solution that worked, and I stopped experimenting.) > > It would be good if you do some :) You don't want any surprises > when it comes to fencing. > Right. Surprises are for Birthday Parties, presents, and such, not so much for servers. Karl > > > The script tracks down > > which of the two is master and then talks to that server. One could > avoid > > that by giving both c-blades the same set of rsa host keys, and then just > > talking to the virtual IP Address that the master blade, which ever of > the > > two it is, is answering on. Again, I am disinclined to modify the > egenera > > setup that way. I tried telling ssh to ignore the key verification, but > it > > still updates the knownhosts file when the script logs in, so it'll throw > > errors the next time it tries if the master has moved. > > Right. Isn't there an option to tell ssh not to update the > knownhosts file. > > > > > If anyone else is already doing this and has a better way, I'd be > happy > > > to > > > > hear about it. > > > > > > > > Karl > > > > > > > ### THESE TWO SHELL SCRIPTS ARE EXPECTED IN /root/bin ON BOTH > c-blades > > > > ### THAT WAY THE SCRPIT THAT FOLLOWS WILL WORK REGARDLESS OF WHICH > > > > ### EGENERA c-blade IS PAN-MASTER > > > > > > > > #!/bin/bash > > > > # get_master.sh - returns the name of the master control blade in > the > > > > PAN. > > > > > > > > MASTER=`/opt/panmgr/bin/esh pan | grep ^c | grep Master | cut -c 2` > > > > echo "cblade-${MASTER}" > > > > #CUT HERE > ############################################################# > > > > > > > > #!/bin/bash > > > > > > > > # get_processor_blade.sh - identifies the blade running a given > pserver. > > > > > > This name doesn't exactly match the references in the script. > > > > > > > > :%s/_blade/-blade/ > > > > I'm a lousy typist.... > > > > > > > > > > > > > > > > BLADE=`/opt/panmgr/bin/esh pan | grep ^${1}` > > > > echo ${BLADE} | sed -e "s/^.*MyBladeFrame\/\(.\)\(.\) .*/a\1-\2/" > > > > #CUT HERE > ############################################################# > > > > > > > > #!/bin/bash > > > > # > > > > # External STONITH module for Egenera. > > > > # > > > > # Copyright (c) 2004 SUSE LINUX AG - Lars Marowsky-Bree <[email protected] > > > > > > # Copyright (c) 2009 Karl W. Lewis <[email protected]> > > > > # This program is free software; you can redistribute > > > > #it and/or modify # it under the terms of version 2 of the GNU > General > > > > #Public License as # published by the Free Software Foundation. > > > > # > > > > # This program is distributed in the hope that it would be useful, > but > > > > # WITHOUT ANY WARRANTY; without even the implied warranty of > > > > # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > > > # > > > > # Further, this software is distributed without any warranty that it > is > > > > # free of the rightful claim of any third person regarding > infringement > > > > # or the like. Any license provided herein, whether implied or > > > > # otherwise, applies only to this software file. Patent licenses, if > > > > # any, provided herein do not apply to combinations of this program > with > > > > # other software, or any other product whatsoever. > > > > # > > > > # You should have received a copy of the GNU General Public License > > > > # along with this program; if not, write the Free Software > Foundation, > > > > # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. > > > > # > > > > > > > > SSH_COMMAND="/usr/bin/ssh" > > > > > > > > #LOG=/var/log/egenera-stonith.log > > > > #echo `date` >> $LOG > > > > #echo $hostlist >> $LOG > > > > #echo $1 >> $LOG > > > > #echo $2 >> $LOG > > > > #echo "executing--------------" >> $LOG > > > > > > > > # Rewrite the hostlist to accept "," as a delimeter for hostnames > too. > > > > hostlist=`echo $hostlist | tr ',' ' '` > > > > > > > > is_host_up() { > > > > for j in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 > > > > do > > > > if > > > > ping -w1 -c1 "$1" >/dev/null 2>&1 > > > > then > > > > sleep 10 > > > > else > > > > return 1 > > > > fi > > > > done > > > > return 0 > > > > } > > > > > > This is from external/ssh and used only if the parameter > > > livedangerously is set to yes. Doesn't strike me as a very robust > > > way to check if a node is running. Is there another way? Using > > > the cblades sounds like a most natural way. > > > > > > Yes. You're clearly right. By way of excuse I'd claim I was rather > > narrowly focused on getting a server fenced and therefore I ignored other > > functions. If you'll tolerate a "version 2, (or, I suppose v0.02), I'll > try > > again. It'll take me a few hours to fix and test, I'll try again. (A > quick > > experiment reveals that yes, the master blade does indeed know the status > of > > a given processor blade.) > > Great. > > > > > MASTER_BLADE=`$SSH_COMMAND cblade-1 "bin/get_master.sh" | tail -1` > > > > > > > > case $1 in > > > > gethosts) > > > > for h in $hostlist ; do > > > > echo $h > > > > done > > > > exit 0 > > > > ;; > > > > on) > > > > for h in $hostlist > > > > do > > > > if > > > > [ "$h" != "$2" ] > > > > then > > > > continue > > > > else > > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE > > > "bin/get_processor-blade.sh $h"` > > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' > > > > /proc/egenera/ipmi/node/$P_BLADE/control/on" > > > > fi > > > > sleep 30 > > > > if > > > > is_host_up $h > > > > then > > > > exit 1 > > > > else > > > > exit 0 > > > > fi > > > > done > > > > exit 0 > > > > ;; > > > > off) > > > > for h in $hostlist > > > > do > > > > if > > > > [ "$h" != "$2" ] > > > > then > > > > continue > > > > else > > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE > > > "bin/get_processor-blade.sh $h"` > > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' > > > > /proc/egenera/ipmi/node/$P_BLADE/control/off" > > > > fi > > > > done > > > > exit 0 > > > > ;; > > > > reset) > > > > for h in $hostlist > > > > do > > > > if > > > > [ "$h" != "$2" ] > > > > then > > > > continue > > > > else > > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE > > > "bin/get_processor-blade.sh $h"` > > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' > > > > /proc/egenera/ipmi/node/$P_BLADE/control/nmi" > > > > fi > > > > sleep 20 > > > > if > > > > is_host_up $h > > > > then > > > > exit 1 > > > > else > > > > exit 0 > > > > fi > > > > done > > > > exit 1 > > > > ;; > > > > > > on), off), and reset) cases are very similar, they should be made > > > to share some code. Also, the sleep values look too ephemeral for > > > my taste. There should be a relatively tight loop to check the > > > host status. > > > > > > > I'm not sure yet what to do about the "ephemeral" sleep values. I chose > > arbitrarily long waits because egenera blades take a surprisingly long > time > > to boot. Your point is well taken, however, the code can be improved. > > Looking forward to it :) > > > > > status) > > > > if > > > > [ -z "$hostlist" ] > > > > then > > > > exit 1 > > > > fi > > > > for h in $hostlist > > > > do > > > > if > > > > ping -w1 -c1 "$h" 2>&1 | grep "unknown host" > > > > > > The status should return the status of the device, not the status > > > of a node. > > > > > > ?? I must have misunderstood the existing code. Hmmmmm. The status of > > this device is always "0" if there's a hostlist and if $MASTER_BLADE is > set, > > I think. > > And if the c-blade is reachable. Something like > "ssh $MASTER_BLADE true" should do. > > > Thank you for taking the itme to look this over. I'll clean it up some > and > > try again. > > Many thanks. > > Cheers, > > Dejan > > > Be well, > > > > Karl > > > > > > > > > > > Cheers, > > > > > > Dejan > > > > > > > then > > > > exit 1 > > > > fi > > > > done > > > > exit 0 > > > > ;; > > > > getconfignames) > > > > echo "hostlist" > > > > exit 0 > > > > ;; > > > > getinfo-devid) > > > > echo "Egenera STONITH device" > > > > exit 0 > > > > ;; > > > > getinfo-devname) > > > > echo "Egenera STONITH external device" > > > > exit 0 > > > > ;; > > > > getinfo-devdescr) > > > > echo "Egenera-based Linux host reset" > > > > exit 0 > > > > ;; > > > > getinfo-devurl) > > > > echo "http://www.egenera.com"; > > > > exit 0 > > > > ;; > > > > getinfo-xml) > > > > cat << EGENERAXML > > > > <parameters> > > > > parameter name="hostlist" unique="1" required="1"> > <contenttype="string" > > > /> > > > > <shortdesc lang="en"> Hostlist </shortdesc> > > > > <longdesc lang="en"> The list of hosts that the STONITH device > controls > > > </longdesc> > > > > </parameter> > > > > </parameters> > > > > EGENERAXML > > > > exit 0 > > > > ;; > > > > *) > > > > exit 1 > > > > ;; > > > > esac > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > Linux-HA mailing list > > > > [email protected] > > > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > > > See also: http://linux-ha.org/ReportingProblems > > > _______________________________________________ > > > Linux-HA mailing list > > > [email protected] > > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > > See also: http://linux-ha.org/ReportingProblems > > > > > _______________________________________________ > > Linux-HA mailing list > > [email protected] > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > See also: http://linux-ha.org/ReportingProblems > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
