Hi, On Thu, Nov 22, 2007 at 05:05:48PM +0530, [EMAIL PROTECTED] wrote: > > looks like I'm barking up the wrong tree. Stonith isn't what I'm looking for > , or is it? > Pls let me explain my scenario... > > 1. I've a 2 node cluster running HA 2.0.8 . > 2. I've 1 resource group with 2 resources : A. a cluster IP (ocf::Ipaddr) B. > a cluster process (ocf::myOcf) > 3. Preffered node is active but resource stickiness is +INFINITY for both. So > at heartbeat startup active runs the resource and standby is in passive state. > 4. Whenever, active goes down (network down, m/c crash, heartbeat stopped) > cluster fails resource over to standby which runs it even when active comes > backon (failback=off ; though sometimes failback does occur ) > > Split brain problem: > > Once, I'd pulled the network cable on active, resource failed over to > standby, then I put the cable back in active, however accidentally the > standby cable got disconnected. After sometime, with both cables connected, > the resource was running on both machines ( Split-brain I suppose). I now > need to fix this, so that one of the nodes stops running the resource.
Short of having a working stonith device to deal with split-brain there's nothing which can help you in this situation. Andrew: Should you perhaps restart the CRM on split brain? Thanks, Dejan > Now I need to find what HA provides to help me fix this problem (rather > recover from this scenario).I've attached my earliest cib.xml and ha.cf > (before I went down the stonith route and until all was well! ) > > <cib admin_epoch="0" have_quorum="true" ignore_dtd="false" num_peers="1" > cib_feature_revision="1.3" generated="true" ccm_transition="5" > dc_uuid="51dd290b-4285-4d5d-8a86-43927afa0b42" epoch="389" num_updates="4644" > cib-last-written="Thu Jul 19 11:06:04 2007"> > <configuration> > <crm_config> > <cluster_property_set id="cib-bootstrap-options"> > <attributes> > <nvpair id="cib-bootstrap-options-default-resource-stickiness" > name="default-resource-stickiness" value="INFINITY"/> > </attributes> > </cluster_property_set> > </crm_config> > <nodes> > <node id="6f6d5305-43e3-46f9-a8d7-7966218bab0b" uname="ha" > type="normal"/> > <node id="bf83d809-e951-4c04-b605-28b5ead6ccfe" uname="test" > type="normal"/> > </nodes> > <resources> > <group id="proxy_rsc"> > <primitive id="IPaddr_10_114_31_238" class="ocf" type="IPaddr" > provider="heartbeat"> > <instance_attributes id="IPaddr_10_114_31_238_inst_attr"> > <attributes> > <nvpair id="IPaddr_10_114_31_238_attr_0" name="ip" > value="10.114.31.238"/> > </attributes> > </instance_attributes> > </primitive> > <primitive id="Proxy_10_114_31_238" class="ocf" type="myOcf" > provider="heartbeat"> > <operations> > <op id="1" name="start" timeout="5s"/> > <op id="2" name="stop" timeout="5s"/> > <op id="3" name="monitor" timeout="5s" interval="15s"/> > </operations> > </primitive> > </group> > </resources> > <constraints> > <rsc_location id="proxy_rsc_location" rsc="proxy_rsc"> > <rule id="prefered_location_proxy_rsc" score="INFINITY"> > <expression id="prefered_location_proxy_rsc_expr" > attribute="#uname" operation="eq" value="test"/> > </rule> > </rsc_location> > </constraints> > </configuration> > </cib> > > HA.CF > > debugfile /var/log/ha-debug > logfile /var/log/ha-log > keepalive 2 > deadtime 30 > warntime 10 > initdead 60 > udpport 694 > baud 19200 > auto_failback off > node test > node HA > ping 10.114.31.1 > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Andrew Beekhof > Sent: Thursday, November 22, 2007 4:18 PM > To: General Linux-HA mailing list > Subject: Re: [Linux-HA] Re: Linux-HA Digest, Vol 48, Issue 69 > > > On Nov 22, 2007, at 7:05 AM, <[EMAIL PROTECTED]> <[EMAIL PROTECTED] > wrote: > > > > > Hi Andrew, > > Pardon me for my ignorance, but I'm new to stonith. Pls read on... > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] > > ] On Behalf Of Andrew Beekhof > > Sent: Wednesday, November 21, 2007 8:42 PM > > To: General Linux-HA mailing list > > Subject: Re: [Linux-HA] Re: Linux-HA Digest, Vol 48, Issue 69 > > > > > > On Nov 21, 2007, at 3:36 PM, <[EMAIL PROTECTED]> <[EMAIL PROTECTED] > > > wrote: > > > >> > >> Can anyone please help me debug problems with my STONITH RA. I've > >> slightly modified the in-built ssh RA ( changed $ssh_command=rsh and > >> reboot_command to "/etc/init.d/heartbeat restart". > > > > Thats a really _really_ bad idea. > > > > I'm trying to recover from a split brain scenario in 1:1 cluster. > > split brains usually result from comms failure in such cases anything that > uses the network is not going to work and restarting heartbeat is extremely > unlikely to help, rebooting the node is not much better. > > the best thing you can hope for is that the fencing operation fails which > will prevent the cluster from doing anything that would kill your data. > > > Hoping that restarting heartbeat on the other node might sync it up > > with the current state of the cluster and recover from split brain. > > Rebooting is also an option, but don't want to be so excessive for > > testing. Would you have any other ideas for my case ? > > > >> I ran the stonith test on the RA and it seemed to work fine. > >> However , everytime I start heartbeat I get these errors and stonith > >> resource doesn't run. The ha-log is attached.. > >> > >> My questions: > >> 1. why stonith resource doesn't start though it's passed stonith test > >> 2. crm_verify has been run then why the asserts 3. how to know from > >> error code (rc) what is wrong? I've actually removed all exit 1's > >> from the reset operation. > >> 4. this is heartbeat log from standby. Why does standby reset itself? > >> It should reset active. > > > > Presumably because this is how you wrote the RA My RA seems to work > > now, but not very consistently . I modified it a bit to return only > > the node name which it should be used to stonith, standby for active > > and active for standby node. > >> > >> > >> My cib.xml file is identical to the last thread with the changes that > >> frank recommended.. > >> > >> Thanks, > >> Abhi. > >> > >> pengine[1808]: 2007/11/21_19:14:50 ERROR: can_run_resources: No node > >> supplied > >> pengine[1808]: 2007/11/21_19:14:50 ERROR: can_run_resources: No node > >> supplied > >> pengine[1808]: 2007/11/21_19:14:50 WARN: native_color: Resource > >> Stonith:1 cannot run anywhere > > > > This is probably the issue the PE is complaining about. > > I still get this error on standby where the RA failed to start , but > > not on the active node. > > > >> > >> pengine[1808]: 2007/11/21_19:14:50 WARN: stage6: Scheduling Node > >> standby for STONITH > >> pengine[1808]: 2007/11/21_19:14:50 WARN: stage6: Scheduling Node > >> active for STONITH > >> pengine[1808]: 2007/11/21_19:14:51 WARN: stage6: Scheduling Node > >> active for STONITH > >> pengine[1808]: 2007/11/21_19:14:51 WARN: process_pe_message: > >> Transition 0: WARNINGs found during PE processing. PEngine Input > >> stored in: /var/lib/heartbeat/pengine/pe-warn-9620.bz2 > >> pengine[1808]: 2007/11/21_19:14:51 info: process_pe_message: > >> Configuration WARNINGs found during PE processing. Please run > >> "crm_verify -L" to identify issues. > >> crmd[1799]: 2007/11/21_19:14:56 ERROR: get_tag_name: Error parsing > >> token near usage:/usr/lib/: bad character, not in [a-zA-Z_-] > >> crmd[1799]: 2007/11/21_19:14:56 ERROR: crm_abort: parse_xml: > >> Triggered non-fatal assert at xml.c:1365 : len > 0 > >> crmd[1799]: 2007/11/21_19:14:56 ERROR: crm_abort: find_xml_node: > >> Triggered non-fatal assert at xml.c:77 : root != NULL > >> crmd[1799]: 2007/11/21_19:14:56 ERROR: cl_get_value: wrong arugment > >> (__name__) > >> crmd[1799]: 2007/11/21_19:14:56 WARN: find_xml_node: Could not find > >> actions in (null). > > > > The metadata for one of your resources is broken. I'm guessing the > > new one. > > I'm now returning exit 0 for meta-data for my OCF resources.ocf- > > tester helped fix this. > > > >> > >> pengine[1808]: 2007/11/21_19:14:58 notice: cluster_option: Using > >> default value '-1' for cluster option 'pe-error-series-max' > >> pengine[1808]: 2007/11/21_19:14:58 notice: cluster_option: Using > >> default value '-1' for cluster option 'pe-warn-series-max' > >> pengine[1808]: 2007/11/21_19:15:00 ERROR: can_run_resources: No node > >> supplied > >> pengine[1808]: 2007/11/21_19:15:00 ERROR: can_run_resources: No node > >> supplied > >> pengine[1808]: 2007/11/21_19:15:00 WARN: native_color: Resource > >> Stonith:1 cannot run anywhere > >> pengine[1808]: 2007/11/21_19:15:00 WARN: stage6: Scheduling Node > >> standby for STONITH > >> pengine[1808]: 2007/11/21_19:15:00 WARN: stage6: Scheduling Node > >> active for STONITH > >> pengine[1808]: 2007/11/21_19:15:00 WARN: stage6: Scheduling Node > >> active for STONITH > >> pengine[1808]: 2007/11/21_19:15:01 WARN: process_pe_message: > >> Transition 1: WARNINGs found during PE processing. PEngine Input > >> stored in: /var/lib/heartbeat/pengine/pe-warn-9621.bz2 > >> pengine[1808]: 2007/11/21_19:15:01 info: process_pe_message: > >> Configuration WARNINGs found during PE processing. Please run > >> "crm_verify -L" to identify issues. > >> crmd[1799]: 2007/11/21_19:15:03 ERROR: process_lrm_event: LRM > >> operation Stonith:0_start_0 (call=9, rc=1) Error unknown error > >> pengine[1808]: 2007/11/21_19:15:07 WARN: unpack_rsc_op: Processing > >> failed op (Stonith:0_start_0) for Stonith:0 on standby > >> pengine[1808]: 2007/11/21_19:15:07 WARN: unpack_rsc_op: Handling > >> failed start for Stonith:0 on standby > >> pengine[1808]: 2007/11/21_19:15:08 WARN: native_color: Resource > >> Stonith:0 cannot run anywhere > >> pengine[1808]: 2007/11/21_19:15:09 WARN: stage6: Scheduling Node > >> standby for STONITH > >> pengine[1808]: 2007/11/21_19:15:09 WARN: stage6: Scheduling Node > >> active for STONITH > >> pengine[1808]: 2007/11/21_19:15:09 WARN: stage6: Scheduling Node > >> active for STONITH > >> pengine[1808]: 2007/11/21_19:15:09 WARN: process_pe_message: > >> Transition 2: WARNINGs found during PE processing. PEngine Input > >> stored in: /var/lib/heartbeat/pengine/pe-warn-9622.bz2 > >> pengine[1808]: 2007/11/21_19:15:09 info: process_pe_message: > >> Configuration WARNINGs found during PE processing. Please run > >> "crm_verify -L" to identify issues. > >> lrmd[1796]: 2007/11/21_19:15:12 WARN: G_SIG_dispatch: Dispatch > >> function for SIGCHLD took too long to execute: 20 ms (> 10 ms) > >> (GSource: 0x892cd50) > >> crmd[1799]: 2007/11/21_19:15:12 ERROR: process_lrm_event: LRM > >> operation Stonith:1_start_0 (call=12, rc=1) Error unknown error > >> tengine[1807]: 2007/11/21_19:15:12 WARN: status_from_rc: Action start > >> on standby failed (target: (null) vs. rc: 1): Error > > > > Whatever you've done to the RA, it no longer starts. > > I'd suggest OCF tester but the whole idea of a "stonith" device that > > only restarts heartbeat is too horrible to contemplate. > > But, I thought ocf RA and stonith RA are different. Shouldn't stonith > > test be used instead ? > > nod, my mistake > > > > > > >> > >> pengine[1808]: 2007/11/21_19:15:14 notice: cluster_option: Using > >> default value '-1' for cluster option 'pe-error-series-max' > >> pengine[1808]: 2007/11/21_19:15:14 notice: cluster_option: Using > >> default value '-1' for cluster option 'pe-warn-series-max' > >> pengine[1808]: 2007/11/21_19:15:14 WARN: unpack_rsc_op: Processing > >> failed op (Stonith:0_start_0) for Stonith:0 on standby > >> pengine[1808]: 2007/11/21_19:15:14 WARN: unpack_rsc_op: Handling > >> failed start for Stonith:0 on standby > >> pengine[1808]: 2007/11/21_19:15:15 WARN: unpack_rsc_op: Processing > >> failed op (Stonith:1_start_0) for Stonith:1 on standby > >> pengine[1808]: 2007/11/21_19:15:15 WARN: unpack_rsc_op: Handling > >> failed start for Stonith:1 on standby > >> pengine[1808]: 2007/11/21_19:15:16 WARN: native_color: Resource > >> Stonith:1 cannot run anywhere > >> pengine[1808]: 2007/11/21_19:15:16 WARN: native_color: Resource > >> Stonith:0 cannot run anywhere > >> pengine[1808]: 2007/11/21_19:15:16 WARN: stage6: Scheduling Node > >> standby for STONITH > >> pengine[1808]: 2007/11/21_19:15:16 WARN: stage6: Scheduling Node > >> active for STONITH > >> pengine[1808]: 2007/11/21_19:15:16 WARN: stage6: Scheduling Node > >> active for STONITH > >> pengine[1808]: 2007/11/21_19:15:16 WARN: process_pe_message: > >> Transition 3: WARNINGs found during PE processing. PEngine Input > >> stored in: /var/lib/heartbeat/pengine/pe-warn-9623.bz2 > >> pengine[1808]: 2007/11/21_19:15:16 info: process_pe_message: > >> Configuration WARNINGs found during PE processing. Please run > >> "crm_verify -L" to identify issues. > >> tengine[1807]: 2007/11/21_19:15:47 ERROR: tengine_stonith_callback: > >> Stonith of standby failed (2)... aborting transition. > >> > >> -----Original Message----- > >> From: [EMAIL PROTECTED] > >> [mailto:[EMAIL PROTECTED] > >> ] On Behalf Of Frank > >> Sent: Wednesday, November 21, 2007 4:23 PM > >> To: [email protected] > >> Subject: RE: [Linux-HA] Re: Linux-HA Digest, Vol 48, Issue 69 > >> > >>> > >>> Date: Wed, 21 Nov 2007 14:21:30 +0530 > >>> From: <[EMAIL PROTECTED]> > >>> Subject: RE: [Linux-HA] Re: Linux-HA Digest, Vol 48, Issue 69 > >>> To: <[email protected]> > >>> Message-ID: > >>> <[EMAIL PROTECTED]> > >>> Content-Type: text/plain; charset="iso-8859-1" > >>> > >>> > >>> Frank, > >>> Congratulations on getting your stonith config to work. > >>> Unfortunately,I've not been so lucky or I don't understand stonith > >>> very well. I've done the exact same configuration as you've but for > >>> some reason , the stonith RA doesn't start at all. I'm using version > >>> 2.0.8 and the ssh stonith agent to test my configuration. I too have > >>> two nodes : active and standby, where I want active to stonith > >>> standby and vice versa. > >>> > >>> I've attached a snapshot of my stonith configuration for you to look > >>> at . I would really appreciate if you could point out any mistakes I > >>> might have made... > >>> > >>> > >>> <crm_config> > >>> <cluster_property_set id="cib-bootstrap-options"> > >>> <attributes> > >>> <nvpair id="cib-bootstrap-options-default-resource- > >>> stickiness" name="default-resource-stickiness" value="INFINITY"/> > >>> <nvpair id="cib-bootstrap-options-stonith" name="stonith- > >>> enabled" value="TRUE"/> > >>> <nvpair id="cib-bootstrap-options-quorum" name="no-quorum- > >>> policy" value="IGNORE"/> > >>> </attributes> > >>> </cluster_property_set> > >>> </crm_config> > >>> <nodes> > >>> <node id="6f6d5305-43e3-46f9-a8d7-7966218bab0b" > >>> uname="standby" type="normal"/> > >>> <node id="bf83d809-e951-4c04-b605-28b5ead6ccfe" > >>> uname="active" type="normal"/> > >>> </nodes> > >>> <resources> > >>> ... > >>> ... > >>> <clone id="stonith_active"> > >>> <instance_attributes id="stonith_active_attr"> > >>> <attributes> > >>> <nvpair id="stonith_active_clone_max" name="clone_max" > >>> value="1"/> > >>> <nvpair id="stonith_active_clone_node_max" > >>> name="clone_node_max" value="1"/> > >>> </attributes> > >>> </instance_attributes> > >>> <primitive id="Stonith_active" class="stonith" > >>> type="external/rsh" provider="heartbeat"> > >>> <operations> > >>> <op id="4" name="start" timeout="10s" prereq="nothing"/> > >>> <op id="5" name="monitor" timeout="5s" interval="15s"/> > >>> </operations> > >>> <instance_attributes id="rsh_active_instance_attr"> > >>> <attributes> > >>> <nvpair id="rsh_attr_active" name="hostlist" > >>> value="active"/> > >>> </attributes> > >>> </instance_attributes> > >>> </primitive> > >>> </clone> > >>> <clone id="stonith_standby"> > >>> <instance_attributes id="stonith_standby_attr"> > >>> <attributes> > >>> <nvpair id="stonith_standby_clone_max" name="clone_max" > >>> value="1"/> > >>> <nvpair id="stonith_standby_clone_node_max" > >>> name="clone_node_max" value="1"/> > >>> </attributes> > >>> </instance_attributes> > >>> <primitive id="Stonith_standby" class="stonith" > >>> type="external/rsh" provider="heartbeat"> > >>> <operations> > >>> <op id="6" name="start" timeout="10s" prereq="nothing"/> > >>> <op id="7" name="monitor" timeout="5s" interval="15s"/> > >>> </operations> > >>> <instance_attributes id="rsh_standby_instance_attr"> > >>> <attributes> > >>> <nvpair id="rsh_attr_standby" name="hostlist" > >>> value="standby"/> > >>> </attributes> > >>> </instance_attributes> > >>> </primitive> > >>> </clone> > >>> </resources> > >>> <constraints> > >>> <rsc_location id="proxy_stonith_active_rsc_location" > >>> rsc="stonith_active"> > >>> <rule id="prefered_location_proxy_stonith_active_rsc" > >>> score="INFINITY"> > >>> <expression > >>> id="prefered_location_proxy_stonith_active_rsc_expr" > >>> attribute="#uname" operation="eq" value="standby"/> > >>> </rule> > >>> </rsc_location> > >>> <rsc_location id="proxy_stonith_standby_rsc_location" > >>> rsc="stonith_standby"> > >>> <rule id="prefered_location_proxy_stonith_standby_rsc" > >>> score="INFINITY"> > >>> <expression > >>> id="prefered_location_proxy_stonith_standby_rsc_expr" > >>> attribute="#uname" operation="eq" value="active"/> > >>> </rule> > >>> </rsc_location> > >>> </constraints> > >>> > >>> Regards, > >>> Abhi. > >> Hi Abhi, > >> - first I suppose that you have set pulblic keys so that each host > >> can make ssh to the other without password > >> - define only one ssh stonith clone resource with hostlist set to > >> both nodes, with "clone_max" to 2 and "clone_node_max" to 1 > >> - define 2 place constraints with the clone resource, one with "uname > >> eq node1" on the other with "uname eq 2" > >> - start stonith resources > >> > >> I strongly recommend you to read the excellent tutorial > >> "http://linux-ha.org/_cache/HeartbeatTutorials__LCA2007-tutorial.ppt > >> " to clarify concepts. > >> > >> Frank > >> > >> -- > >> Aquest missatge ha estat analitzat per MailScanner a la cerca de > >> virus i d'altres continguts perillosos, i es considera que est? net. > >> For all your IT requirements visit: http://www.transtec.co.uk > >> > >> _______________________________________________ > >> Linux-HA mailing list > >> [email protected] > >> http://lists.linux-ha.org/mailman/listinfo/linux-ha > >> See also: http://linux-ha.org/ReportingProblems > >> > >> > >> The information contained in this electronic message and any > >> attachments to this message are intended for the exclusive use of the > >> addressee(s) and may contain proprietary, confidential or privileged > >> information. If you are not the intended recipient, you should not > >> disseminate, distribute or copy this e-mail. Please notify the sender > >> immediately and destroy all copies of this message and any > >> attachments. > >> > >> WARNING: Computer viruses can be transmitted via email. The recipient > >> should check this email and any attachments for the presence of > >> viruses. The company accepts no liability for any damage caused by > >> any virus transmitted by this email. > >> > >> www.wipro.com > >> _______________________________________________ > >> Linux-HA mailing list > >> [email protected] > >> http://lists.linux-ha.org/mailman/listinfo/linux-ha > >> See also: http://linux-ha.org/ReportingProblems > > > > _______________________________________________ > > Linux-HA mailing list > > [email protected] > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > See also: http://linux-ha.org/ReportingProblems > > > > > > The information contained in this electronic message and any > > attachments to this message are intended for the exclusive use of the > > addressee(s) and may contain proprietary, confidential or privileged > > information. If you are not the intended recipient, you should not > > disseminate, distribute or copy this e-mail. Please notify the sender > > immediately and destroy all copies of this message and any > > attachments. > > > > WARNING: Computer viruses can be transmitted via email. The recipient > > should check this email and any attachments for the presence of > > viruses. The company accepts no liability for any damage caused by any > > virus transmitted by this email. > > > > www.wipro.com > > _______________________________________________ > > Linux-HA mailing list > > [email protected] > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > See also: http://linux-ha.org/ReportingProblems > > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > > > The information contained in this electronic message and any attachments to > this message are intended for the exclusive use of the addressee(s) and may > contain proprietary, confidential or privileged information. If you are not > the intended recipient, you should not disseminate, distribute or copy this > e-mail. Please notify the sender immediately and destroy all copies of this > message and any attachments. > > WARNING: Computer viruses can be transmitted via email. The recipient should > check this email and any attachments for the presence of viruses. The company > accepts no liability for any damage caused by any virus transmitted by this > email. > > www.wipro.com > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems -- Dejan _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
