Hi,
On Tue, Sep 22, 2009 at 09:39:00AM -0400, Karl W. Lewis wrote:
> On Mon, Sep 7, 2009 at 11:23 AM, Karl W. Lewis <[email protected]>wrote:
>
> >
> >
> > On Mon, Sep 7, 2009 at 10:16 AM, Dejan Muhamedagic
> > <[email protected]>wrote:
> >
> >> Hi,
> >>
> >> On Wed, Aug 26, 2009 at 01:11:24PM -0400, Karl W. Lewis wrote:
> >> > On Wed, Aug 26, 2009 at 10:58 AM, Dejan Muhamedagic <
> >> [email protected]>wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > On Mon, Aug 24, 2009 at 11:35:15AM -0400, Karl W. Lewis wrote:
> >> > > > I don't know if anyone else is running Heartbeat/Pacemaker on an
> >> Egenera
> >> > > > BladeFrame, but if they are, these scripts might be useful to help
> >> set up
> >> > > > stonith.
> >> > > >
> >> > > > The Egeneras use an IPMI arrangement to control the various
> >> processor
> >> > > > blades, and these routines take advantage of that.
> >> > >
> >> > > Did you try ipmilan or external/ipmi too?
> >> >
> >> >
> >> > I did not, but a quick review leads me to believe that the egenera frame
> >> > does not count as an impi "device". One might be able to make that
> >> work,
> >> > but you'd have to modify the ipmi script until it looked much like this
> >> > example, or, you'd have to play very. very hard with the egnera
> >> c-blades,
> >> > and I can't imagine anyone would want to do that.
> >>
> >> Well, either they are IPMI compliant or not, i.e. our ipmi
> >> stonith modules should work without modifications. I think it's
> >> worth trying. At any rate, I think that this is a different
> >> solution worth pursuing, since it would reduce the configuration
> >> quite a bit.
> >>
> >
> > I've not forgotten this task, but it's been very busy at work and I'm doing
> > more research. The ipmi commands do not work the way I expected them to,
> > (writing to "off" just resets the blade), so I'm looking at the native
> > Egenera PAN Manager commands.
> >
> >>
> >> > > > This may not be the best method of doing it, but it does appear to
> >> work.
> >> > > > It is not fast. What it lacks in speed, however, it makes up for in
> >> > > > flexibility. It can figure out which c-blade is the master, and
> >> then
> >> > > figure
> >> > > > out which p-blade is running the server to be stonithed, and then
> >> > > stonith,
> >> > > > it. The framework for this was an existing external stonith script
> >> that
> >> > > I
> >> > > > simply modified to meet my needs.
> >> > > >
> >> > > > The two short shell scripts at the top need to live in /root/bin/ on
> >> both
> >> > >
> >> > > I guess you wanted to say $HOME/bin. Note that the root user may
> >> > > not always have the same home directory. Perhaps it would be
> >> > > better to go with an absolute path such as /usr/local/bin.
> >> >
> >> >
> >> > I actually put them in root's directory with malice aforethought, but
> >> now
> >> > that you've asked the question... I guess I can't defend that decision.
> >> I
> >> > use the user root because that's the only user on the c-blades under
> >> > egenera's setup. One could add another user, but I can't advocate
> >> changing
> >> > the config of the c-blades.
> >> >
> >> >
> >> > >
> >> > >
> >> > > > control blades. The cluster servers need to have ssh keys and any
> >> > > required
> >> > > > firewall rules setup such that root on the cluster servers can ssh
> >> to the
> >> > > > c-blades.
> >> > >
> >> > > There can be more than one? Is that covered in any way by the
> >> > > script, i.e. does it try all available?
> >> >
> >> >
> >> > Yes. Just so. There are two c-baldes; one is master and one is
> >> standby. I
> >> > *think* that the master is the only one that can issue the ipmi commands
> >> and
> >> > get a reaction. I've not experimented with that, however. (I hit a
> >> > solution that worked, and I stopped experimenting.)
> >>
> >> It would be good if you do some :) You don't want any surprises
> >> when it comes to fencing.
> >>
> >
> > Right. Surprises are for Birthday Parties, presents, and such, not so much
> > for servers.
> >
> > Karl
> >
> >
> >
> >>
> >> > The script tracks down
> >> > which of the two is master and then talks to that server. One could
> >> avoid
> >> > that by giving both c-blades the same set of rsa host keys, and then
> >> just
> >> > talking to the virtual IP Address that the master blade, which ever of
> >> the
> >> > two it is, is answering on. Again, I am disinclined to modify the
> >> egenera
> >> > setup that way. I tried telling ssh to ignore the key verification, but
> >> it
> >> > still updates the knownhosts file when the script logs in, so it'll
> >> throw
> >> > errors the next time it tries if the master has moved.
> >>
> >> Right. Isn't there an option to tell ssh not to update the
> >> knownhosts file.
> >>
> >> > > > If anyone else is already doing this and has a better way, I'd be
> >> happy
> >> > > to
> >> > > > hear about it.
> >> > > >
> >> > > > Karl
> >> > >
> >> > > > ### THESE TWO SHELL SCRIPTS ARE EXPECTED IN /root/bin ON BOTH
> >> c-blades
> >> > > > ### THAT WAY THE SCRPIT THAT FOLLOWS WILL WORK REGARDLESS OF WHICH
> >> > > > ### EGENERA c-blade IS PAN-MASTER
> >> > > >
> >> > > > #!/bin/bash
> >> > > > # get_master.sh - returns the name of the master control blade in
> >> the
> >> > > > PAN.
> >> > > >
> >> > > > MASTER=`/opt/panmgr/bin/esh pan | grep ^c | grep Master | cut -c 2`
> >> > > > echo "cblade-${MASTER}"
> >> > > > #CUT HERE
> >> #############################################################
> >> > > >
> >> > > > #!/bin/bash
> >> > > >
> >> > > > # get_processor_blade.sh - identifies the blade running a given
> >> pserver.
> >> > >
> >> > > This name doesn't exactly match the references in the script.
> >> >
> >> >
> >> >
> >> > :%s/_blade/-blade/
> >> >
> >> > I'm a lousy typist....
> >> >
> >> >
> >> >
> >> > >
> >> > >
> >> > > > BLADE=`/opt/panmgr/bin/esh pan | grep ^${1}`
> >> > > > echo ${BLADE} | sed -e "s/^.*MyBladeFrame\/\(.\)\(.\) .*/a\1-\2/"
> >> > > > #CUT HERE
> >> #############################################################
> >> > > >
> >> > > > #!/bin/bash
> >> > > > #
> >> > > > # External STONITH module for Egenera.
> >> > > > #
> >> > > > # Copyright (c) 2004 SUSE LINUX AG - Lars Marowsky-Bree <
> >> [email protected]>
> >> > > > # Copyright (c) 2009 Karl W. Lewis <[email protected]>
> >> > > > # This program is free software; you can redistribute
> >> > > > #it and/or modify # it under the terms of version 2 of the GNU
> >> General
> >> > > > #Public License as # published by the Free Software Foundation.
> >> > > > #
> >> > > > # This program is distributed in the hope that it would be useful,
> >> but
> >> > > > # WITHOUT ANY WARRANTY; without even the implied warranty of
> >> > > > # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> >> > > > #
> >> > > > # Further, this software is distributed without any warranty that it
> >> is
> >> > > > # free of the rightful claim of any third person regarding
> >> infringement
> >> > > > # or the like. Any license provided herein, whether implied or
> >> > > > # otherwise, applies only to this software file. Patent licenses,
> >> if
> >> > > > # any, provided herein do not apply to combinations of this program
> >> with
> >> > > > # other software, or any other product whatsoever.
> >> > > > #
> >> > > > # You should have received a copy of the GNU General Public License
> >> > > > # along with this program; if not, write the Free Software
> >> Foundation,
> >> > > > # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA.
> >> > > > #
> >> > > >
> >> > > > SSH_COMMAND="/usr/bin/ssh"
> >> > > >
> >> > > > #LOG=/var/log/egenera-stonith.log
> >> > > > #echo `date` >> $LOG
> >> > > > #echo $hostlist >> $LOG
> >> > > > #echo $1 >> $LOG
> >> > > > #echo $2 >> $LOG
> >> > > > #echo "executing--------------" >> $LOG
> >> > > >
> >> > > > # Rewrite the hostlist to accept "," as a delimeter for hostnames
> >> too.
> >> > > > hostlist=`echo $hostlist | tr ',' ' '`
> >> > > >
> >> > > > is_host_up() {
> >> > > > for j in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> >> > > > do
> >> > > > if
> >> > > > ping -w1 -c1 "$1" >/dev/null 2>&1
> >> > > > then
> >> > > > sleep 10
> >> > > > else
> >> > > > return 1
> >> > > > fi
> >> > > > done
> >> > > > return 0
> >> > > > }
> >> > >
> >> > > This is from external/ssh and used only if the parameter
> >> > > livedangerously is set to yes. Doesn't strike me as a very robust
> >> > > way to check if a node is running. Is there another way? Using
> >> > > the cblades sounds like a most natural way.
> >> >
> >> >
> >> > Yes. You're clearly right. By way of excuse I'd claim I was rather
> >> > narrowly focused on getting a server fenced and therefore I ignored
> >> other
> >> > functions. If you'll tolerate a "version 2, (or, I suppose v0.02), I'll
> >> try
> >> > again. It'll take me a few hours to fix and test, I'll try again. (A
> >> quick
> >> > experiment reveals that yes, the master blade does indeed know the
> >> status of
> >> > a given processor blade.)
> >>
> >> Great.
> >>
> >> > > > MASTER_BLADE=`$SSH_COMMAND cblade-1 "bin/get_master.sh" | tail -1`
> >> > > >
> >> > > > case $1 in
> >> > > > gethosts)
> >> > > > for h in $hostlist ; do
> >> > > > echo $h
> >> > > > done
> >> > > > exit 0
> >> > > > ;;
> >> > > > on)
> >> > > > for h in $hostlist
> >> > > > do
> >> > > > if
> >> > > > [ "$h" != "$2" ]
> >> > > > then
> >> > > > continue
> >> > > > else
> >> > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE
> >> > > "bin/get_processor-blade.sh $h"`
> >> > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' >
> >> > > /proc/egenera/ipmi/node/$P_BLADE/control/on"
> >> > > > fi
> >> > > > sleep 30
> >> > > > if
> >> > > > is_host_up $h
> >> > > > then
> >> > > > exit 1
> >> > > > else
> >> > > > exit 0
> >> > > > fi
> >> > > > done
> >> > > > exit 0
> >> > > > ;;
> >> > > > off)
> >> > > > for h in $hostlist
> >> > > > do
> >> > > > if
> >> > > > [ "$h" != "$2" ]
> >> > > > then
> >> > > > continue
> >> > > > else
> >> > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE
> >> > > "bin/get_processor-blade.sh $h"`
> >> > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' >
> >> > > /proc/egenera/ipmi/node/$P_BLADE/control/off"
> >> > > > fi
> >> > > > done
> >> > > > exit 0
> >> > > > ;;
> >> > > > reset)
> >> > > > for h in $hostlist
> >> > > > do
> >> > > > if
> >> > > > [ "$h" != "$2" ]
> >> > > > then
> >> > > > continue
> >> > > > else
> >> > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE
> >> > > "bin/get_processor-blade.sh $h"`
> >> > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' >
> >> > > /proc/egenera/ipmi/node/$P_BLADE/control/nmi"
> >> > > > fi
> >> > > > sleep 20
> >> > > > if
> >> > > > is_host_up $h
> >> > > > then
> >> > > > exit 1
> >> > > > else
> >> > > > exit 0
> >> > > > fi
> >> > > > done
> >> > > > exit 1
> >> > > > ;;
> >> > >
> >> > > on), off), and reset) cases are very similar, they should be made
> >> > > to share some code. Also, the sleep values look too ephemeral for
> >> > > my taste. There should be a relatively tight loop to check the
> >> > > host status.
> >> > >
> >> >
> >> > I'm not sure yet what to do about the "ephemeral" sleep values. I chose
> >> > arbitrarily long waits because egenera blades take a surprisingly long
> >> time
> >> > to boot. Your point is well taken, however, the code can be improved.
> >>
> >> Looking forward to it :)
> >>
> >> > > > status)
> >> > > > if
> >> > > > [ -z "$hostlist" ]
> >> > > > then
> >> > > > exit 1
> >> > > > fi
> >> > > > for h in $hostlist
> >> > > > do
> >> > > > if
> >> > > > ping -w1 -c1 "$h" 2>&1 | grep "unknown host"
> >> > >
> >> > > The status should return the status of the device, not the status
> >> > > of a node.
> >> >
> >> >
> >> > ?? I must have misunderstood the existing code. Hmmmmm. The status of
> >> > this device is always "0" if there's a hostlist and if $MASTER_BLADE is
> >> set,
> >> > I think.
> >>
> >> And if the c-blade is reachable. Something like
> >> "ssh $MASTER_BLADE true" should do.
> >>
> >> > Thank you for taking the itme to look this over. I'll clean it up some
> >> and
> >> > try again.
> >>
> >> Many thanks.
> >>
> >> Cheers,
> >>
> >> Dejan
> >>
> >> > Be well,
> >> >
> >> > Karl
> >> >
> >> > >
> >> > >
> >> > > Cheers,
> >> > >
> >> > > Dejan
> >> > >
> >>
> >
> >
> I've cleaned this up some, and reworked much of it. It has been tested in
> production for a couple of weeks now, and it works properly as near as I can
> tell. (It has properly restarted a misbehaving server.)
>
> I could not see a way to get the existing IPMI script to work with Egenera,
> and I have abandoned using anything other than native Egenera commands to
> control the blades. These scripts require that ssh keys are set up for user
> root between the cluster servers and the c-blades.
OK.
> Code:
Can you please send the scripts as attachments. The mail agents
kill the formatting.
> #The two bash scripts here at the top go on each of the Egenera C-Blades.
> #the STONITH module goes in /usr/lib/stonith/plugins/external/egenera on
> #each of the servers in the cluster.
These two scripts will go to the doc part of the package.
Cheers,
Dejan
> ############################### cut here
> ######################################
> #!/bin/bash
> # get_status.sh - returns the status of a pServer, given the LPAN and
> pServer
>
> #called as:
> # get_status.sh pServer LPAN_Name UP
> # or
> # get_status.sh pServer LPAN_Name DOWN
> #depending on what you want. This only works of the pServer is running the
> PAN Agent
> #because until the agent comes up, the server does not count as "Up"
>
> if [ ${3} = "UP" ]
> then
> CONDITION=`/opt/panmgr/bin/esh pserver -l ${2}/${1} | grep ^${1} |
> grep "Booted" | grep "Up"`
> if [ -z "${CONDITION}" ]
> then
> echo "Down" # No Joy
> else
> echo "Up" # Joy!
> fi
> else # it should be the case that ${3} = "DOWN"
> CONDITION=`/opt/panmgr/bin/esh pserver -l ${2}/${1} | grep ^${1} |
> grep "Shutdown"`
> if [ -z "${CONDITION}" ]
> then
> echo "Up" # No Joy
> else
> echo "Down" # Joy!
> fi
> fi
>
> ############################### cut here
> ######################################
>
> #!/bin/bash
> # get_master.sh - returns the name of the master control blade in the PAN.
>
> MASTER=`/opt/panmgr/bin/esh pan | grep ^c | grep Master | cut -c 2`
> echo "cblade-${MASTER}"
>
> ############################### cut here
> ######################################
>
> #!/bin/bash
> #
> # External STONITH module for Egenera pServers.
> #
> # Copyright (c) 2004 SUSE LINUX AG - Lars Marowsky-Bree <[email protected]>
> # Copyright (c) 2009 Karl W. Lewis <karl.w.lewis - AT - gmail.com>
> #
> # This program is free software; you can redistribute it and/or modify
> # it under the terms of version 2 of the GNU General Public License as
> # published by the Free Software Foundation.
> #
> # This program is distributed in the hope that it would be useful, but
> # WITHOUT ANY WARRANTY; without even the implied warranty of
> # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> #
> # Further, this software is distributed without any warranty that it is
> # free of the rightful claim of any third person regarding infringement
> # or the like. Any license provided herein, whether implied or
> # otherwise, applies only to this software file. Patent licenses, if
> # any, provided herein do not apply to combinations of this program with
> # other software, or any other product whatsoever.
> #
> # You should have received a copy of the GNU General Public License
> # along with this program; if not, write the Free Software Foundation,
> # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA.
> #
>
> SSH_COMMAND="/usr/bin/ssh -q -x -n -l root"
> GET_MASTER="/root/bin/get_master.sh"
> GET_STATUS="/root/bin/get_status.sh"
> PSERVER_COMMAND="/opt/panmgr/bin/esh pserver"
>
> #These TIMEOUT values were obtained from the 'pserver -l pan/pserver'
> command
> # on a given Egenera frame under "Timeouts" YMMV.
>
> BOOT_TIMEOUT=270
> REBOOT_TIMEOUT=450
> SHUTDOWN_TIMEOUT=180
>
> TIMEOUT=${REBOOT_TIMEOUT}
> STATUS="UNKNOWN"
>
> #LOG=/var/log/egenera-stonith.log
> #echo `date` >> ${LOG}
> #echo ${hostlist} >> ${LOG}
> #echo ${lpan_name} >> ${LOG}
> #echo $1 >> ${LOG}
> #echo $2 >> ${LOG}
> #echo "executing--------------" >> ${LOG}
>
> # Rewrite the hostlist to accept "," as a delimeter for hostnames too.
> hostlist=`echo ${hostlist} | tr ',' ' '`
>
> is_host_up() {
> TIME=$((${TIMEOUT}/5))
> for j in 1 2 3 4 5 6
> do
> STATUS=`${SSH_COMMAND} ${MASTER_BLADE} "${GET_STATUS} ${h}
> ${lpan_name} UP"`
> if [ "${STATUS}" = "Down" ]
> then
> sleep ${TIME}
> else
> SYSTEM_STATUS="On Target"
> return 1
> fi
> done
> return 0
> }
>
> is_host_down() {
> TIME=$((${TIMEOUT}/5))
> for j in 1 2 3 4 5 6
> do
> STATUS=`${SSH_COMMAND} ${MASTER_BLADE} "${GET_STATUS} ${h}
> ${lpan_name} DOWN"`
> if [ "${STATUS}" = "Up" ]
> then
> sleep ${TIME}
> else
> SYSTEM_STATUS="On Target"
> return 1
> fi
> done
> return 0
> }
>
> MASTER_BLADE=`${SSH_COMMAND} cblade-1 "${GET_MASTER}" | tail -1`
>
> if [ -z "${MASTER_BLADE}" ]
> then
> MASTER_BLADE=`${SSH_COMMAND} cblade-2 "${GET_MASTER}" | tail -1`
> fi
>
> case ${1} in
> gethosts)
> for h in ${hostlist}
> do
> echo ${h}
> done
> exit 0
> ;;
> on|off|reboot)
> for h in ${hostlist}
> do
> if [ "${h}" != "${2}" ]
> then
> continue
> else
> if [ "${1}" = "on" ]
> then
> CMD="--boot"
> elif [ "${1}" = "off" ]
> then
> CMD="--shutdown --force"
> else #must be "reboot"
> CMD="--reboot --force"
> fi
> ${SSH_COMMAND} ${MASTER_BLADE} "${PSERVER_COMMAND} ${CMD}
> ${lpan_name}/${h}"
> if [ "${1}" = "on" ]
> then
> TIMEOUT=${BOOT_TIMEOUT}
> elif [ "${1}" = "reboot" ]
> then
> TIMEOUT=${REBOOT_TIMEOUT}
> else
> TIMEOUT=${SHUTDOWN_TIMEOUT}
> fi
> if [ "${1}" = "off" ]
> then
> is_host_down ${h}
> else
> is_host_up ${h}
> fi
> fi
> if [ "${SYSTEM_STATUS}" = "On Target" ]
> then
> exit 0
> else
> exit 1
> fi
> done
> exit 1 # We got here only if the host was not in the list....
> ;;
> status)
> if [ -z "${hostlist}" ] || [ -z "${lpan_name}" ] || [ -z
> "${MASTER_BLADE}" ]
> then
> exit 1
> fi
> exit 0
> ;;
> getconfignames)
> echo "hostlist lpan_name"
> exit 0
> ;;
> getinfo-devid)
> echo "Egenera STONITH device"
> exit 0
> ;;
> getinfo-devname)
> echo "Egenera STONITH external device"
> exit 0
> ;;
> getinfo-devdescr)
> echo "Egenera-based Linux host reset"
> exit 0
> ;;
> getinfo-devurl)
> echo "http://www.egenera.com";
> exit 0
> ;;
> getinfo-xml)
> cat << EGENERAXML
> <parameters>
> <parameter name="hostlist" unique="1" required="1">
> <content type="string" />
> <shortdesc lang="en">
> Hostlist
> </shortdesc>
> <longdesc lang="en">
> The list of hosts that the STONITH device controls
> </longdesc>
> </parameter>
> <parameter name="lpan_name" unique="1" required="1">
> <content type="string" />
> <shortdesc lang="en">
> LPAN_Name
> </shortdesc>
> <longdesc lang="en">
> The name of the LPAN that the cluster blades, that this STONITH device
> controls, live in
> </longdesc>
> </parameter>
> </parameters>
> EGENERAXML
> exit 0
> ;;
> *)
> exit 1
> ;;
> esac
> _______________________________________________
> Linux-HA mailing list
> [email protected]
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
[email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
