Hi Andrew, Pardon me for my ignorance, but I'm new to stonith. Pls read on...
-----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Andrew Beekhof Sent: Wednesday, November 21, 2007 8:42 PM To: General Linux-HA mailing list Subject: Re: [Linux-HA] Re: Linux-HA Digest, Vol 48, Issue 69 On Nov 21, 2007, at 3:36 PM, <[EMAIL PROTECTED]> <[EMAIL PROTECTED] > wrote: > > Can anyone please help me debug problems with my STONITH RA. I've > slightly modified the in-built ssh RA ( changed $ssh_command=rsh and > reboot_command to "/etc/init.d/heartbeat restart". Thats a really _really_ bad idea. I'm trying to recover from a split brain scenario in 1:1 cluster. Hoping that restarting heartbeat on the other node might sync it up with the current state of the cluster and recover from split brain. Rebooting is also an option, but don't want to be so excessive for testing. Would you have any other ideas for my case ? > I ran the stonith test on the RA and it seemed to work fine. > However , everytime I start heartbeat I get these errors and stonith > resource doesn't run. The ha-log is attached.. > > My questions: > 1. why stonith resource doesn't start though it's passed stonith test > 2. crm_verify has been run then why the asserts 3. how to know from > error code (rc) what is wrong? I've actually removed all exit 1's from > the reset operation. > 4. this is heartbeat log from standby. Why does standby reset itself? > It should reset active. Presumably because this is how you wrote the RA My RA seems to work now, but not very consistently . I modified it a bit to return only the node name which it should be used to stonith, standby for active and active for standby node. > > > My cib.xml file is identical to the last thread with the changes that > frank recommended.. > > Thanks, > Abhi. > > pengine[1808]: 2007/11/21_19:14:50 ERROR: can_run_resources: No node > supplied > pengine[1808]: 2007/11/21_19:14:50 ERROR: can_run_resources: No node > supplied > pengine[1808]: 2007/11/21_19:14:50 WARN: native_color: Resource > Stonith:1 cannot run anywhere This is probably the issue the PE is complaining about. I still get this error on standby where the RA failed to start , but not on the active node. > > pengine[1808]: 2007/11/21_19:14:50 WARN: stage6: Scheduling Node > standby for STONITH > pengine[1808]: 2007/11/21_19:14:50 WARN: stage6: Scheduling Node > active for STONITH > pengine[1808]: 2007/11/21_19:14:51 WARN: stage6: Scheduling Node > active for STONITH > pengine[1808]: 2007/11/21_19:14:51 WARN: process_pe_message: > Transition 0: WARNINGs found during PE processing. PEngine Input > stored in: /var/lib/heartbeat/pengine/pe-warn-9620.bz2 > pengine[1808]: 2007/11/21_19:14:51 info: process_pe_message: > Configuration WARNINGs found during PE processing. Please run > "crm_verify -L" to identify issues. > crmd[1799]: 2007/11/21_19:14:56 ERROR: get_tag_name: Error parsing > token near usage:/usr/lib/: bad character, not in [a-zA-Z_-] > crmd[1799]: 2007/11/21_19:14:56 ERROR: crm_abort: parse_xml: > Triggered non-fatal assert at xml.c:1365 : len > 0 > crmd[1799]: 2007/11/21_19:14:56 ERROR: crm_abort: find_xml_node: > Triggered non-fatal assert at xml.c:77 : root != NULL > crmd[1799]: 2007/11/21_19:14:56 ERROR: cl_get_value: wrong arugment > (__name__) > crmd[1799]: 2007/11/21_19:14:56 WARN: find_xml_node: Could not find > actions in (null). The metadata for one of your resources is broken. I'm guessing the new one. I'm now returning exit 0 for meta-data for my OCF resources.ocf-tester helped fix this. > > pengine[1808]: 2007/11/21_19:14:58 notice: cluster_option: Using > default value '-1' for cluster option 'pe-error-series-max' > pengine[1808]: 2007/11/21_19:14:58 notice: cluster_option: Using > default value '-1' for cluster option 'pe-warn-series-max' > pengine[1808]: 2007/11/21_19:15:00 ERROR: can_run_resources: No node > supplied > pengine[1808]: 2007/11/21_19:15:00 ERROR: can_run_resources: No node > supplied > pengine[1808]: 2007/11/21_19:15:00 WARN: native_color: Resource > Stonith:1 cannot run anywhere > pengine[1808]: 2007/11/21_19:15:00 WARN: stage6: Scheduling Node > standby for STONITH > pengine[1808]: 2007/11/21_19:15:00 WARN: stage6: Scheduling Node > active for STONITH > pengine[1808]: 2007/11/21_19:15:00 WARN: stage6: Scheduling Node > active for STONITH > pengine[1808]: 2007/11/21_19:15:01 WARN: process_pe_message: > Transition 1: WARNINGs found during PE processing. PEngine Input > stored in: /var/lib/heartbeat/pengine/pe-warn-9621.bz2 > pengine[1808]: 2007/11/21_19:15:01 info: process_pe_message: > Configuration WARNINGs found during PE processing. Please run > "crm_verify -L" to identify issues. > crmd[1799]: 2007/11/21_19:15:03 ERROR: process_lrm_event: LRM > operation Stonith:0_start_0 (call=9, rc=1) Error unknown error > pengine[1808]: 2007/11/21_19:15:07 WARN: unpack_rsc_op: Processing > failed op (Stonith:0_start_0) for Stonith:0 on standby > pengine[1808]: 2007/11/21_19:15:07 WARN: unpack_rsc_op: Handling > failed start for Stonith:0 on standby > pengine[1808]: 2007/11/21_19:15:08 WARN: native_color: Resource > Stonith:0 cannot run anywhere > pengine[1808]: 2007/11/21_19:15:09 WARN: stage6: Scheduling Node > standby for STONITH > pengine[1808]: 2007/11/21_19:15:09 WARN: stage6: Scheduling Node > active for STONITH > pengine[1808]: 2007/11/21_19:15:09 WARN: stage6: Scheduling Node > active for STONITH > pengine[1808]: 2007/11/21_19:15:09 WARN: process_pe_message: > Transition 2: WARNINGs found during PE processing. PEngine Input > stored in: /var/lib/heartbeat/pengine/pe-warn-9622.bz2 > pengine[1808]: 2007/11/21_19:15:09 info: process_pe_message: > Configuration WARNINGs found during PE processing. Please run > "crm_verify -L" to identify issues. > lrmd[1796]: 2007/11/21_19:15:12 WARN: G_SIG_dispatch: Dispatch > function for SIGCHLD took too long to execute: 20 ms (> 10 ms) > (GSource: 0x892cd50) > crmd[1799]: 2007/11/21_19:15:12 ERROR: process_lrm_event: LRM > operation Stonith:1_start_0 (call=12, rc=1) Error unknown error > tengine[1807]: 2007/11/21_19:15:12 WARN: status_from_rc: Action > start on standby failed (target: (null) vs. rc: 1): Error Whatever you've done to the RA, it no longer starts. I'd suggest OCF tester but the whole idea of a "stonith" device that only restarts heartbeat is too horrible to contemplate. But, I thought ocf RA and stonith RA are different. Shouldn't stonith test be used instead ? > > pengine[1808]: 2007/11/21_19:15:14 notice: cluster_option: Using > default value '-1' for cluster option 'pe-error-series-max' > pengine[1808]: 2007/11/21_19:15:14 notice: cluster_option: Using > default value '-1' for cluster option 'pe-warn-series-max' > pengine[1808]: 2007/11/21_19:15:14 WARN: unpack_rsc_op: Processing > failed op (Stonith:0_start_0) for Stonith:0 on standby > pengine[1808]: 2007/11/21_19:15:14 WARN: unpack_rsc_op: Handling > failed start for Stonith:0 on standby > pengine[1808]: 2007/11/21_19:15:15 WARN: unpack_rsc_op: Processing > failed op (Stonith:1_start_0) for Stonith:1 on standby > pengine[1808]: 2007/11/21_19:15:15 WARN: unpack_rsc_op: Handling > failed start for Stonith:1 on standby > pengine[1808]: 2007/11/21_19:15:16 WARN: native_color: Resource > Stonith:1 cannot run anywhere > pengine[1808]: 2007/11/21_19:15:16 WARN: native_color: Resource > Stonith:0 cannot run anywhere > pengine[1808]: 2007/11/21_19:15:16 WARN: stage6: Scheduling Node > standby for STONITH > pengine[1808]: 2007/11/21_19:15:16 WARN: stage6: Scheduling Node > active for STONITH > pengine[1808]: 2007/11/21_19:15:16 WARN: stage6: Scheduling Node > active for STONITH > pengine[1808]: 2007/11/21_19:15:16 WARN: process_pe_message: > Transition 3: WARNINGs found during PE processing. PEngine Input > stored in: /var/lib/heartbeat/pengine/pe-warn-9623.bz2 > pengine[1808]: 2007/11/21_19:15:16 info: process_pe_message: > Configuration WARNINGs found during PE processing. Please run > "crm_verify -L" to identify issues. > tengine[1807]: 2007/11/21_19:15:47 ERROR: tengine_stonith_callback: > Stonith of standby failed (2)... aborting transition. > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > ] On Behalf Of Frank > Sent: Wednesday, November 21, 2007 4:23 PM > To: [email protected] > Subject: RE: [Linux-HA] Re: Linux-HA Digest, Vol 48, Issue 69 > >> >> Date: Wed, 21 Nov 2007 14:21:30 +0530 >> From: <[EMAIL PROTECTED]> >> Subject: RE: [Linux-HA] Re: Linux-HA Digest, Vol 48, Issue 69 >> To: <[email protected]> >> Message-ID: >> <[EMAIL PROTECTED]> >> Content-Type: text/plain; charset="iso-8859-1" >> >> >> Frank, >> Congratulations on getting your stonith config to work. >> Unfortunately,I've not been so lucky or I don't understand stonith >> very well. I've done the exact same configuration as you've but for >> some reason , the stonith RA doesn't start at all. I'm using >> version 2.0.8 and the ssh stonith agent to test my configuration. I >> too have two nodes : active and standby, where I want active to >> stonith standby and vice versa. >> >> I've attached a snapshot of my stonith configuration for you to >> look at . I would really appreciate if you could point out any >> mistakes I might have made... >> >> >> <crm_config> >> <cluster_property_set id="cib-bootstrap-options"> >> <attributes> >> <nvpair id="cib-bootstrap-options-default-resource- >> stickiness" name="default-resource-stickiness" value="INFINITY"/> >> <nvpair id="cib-bootstrap-options-stonith" name="stonith- >> enabled" value="TRUE"/> >> <nvpair id="cib-bootstrap-options-quorum" name="no-quorum- >> policy" value="IGNORE"/> >> </attributes> >> </cluster_property_set> >> </crm_config> >> <nodes> >> <node id="6f6d5305-43e3-46f9-a8d7-7966218bab0b" >> uname="standby" type="normal"/> >> <node id="bf83d809-e951-4c04-b605-28b5ead6ccfe" >> uname="active" type="normal"/> >> </nodes> >> <resources> >> ... >> ... >> <clone id="stonith_active"> >> <instance_attributes id="stonith_active_attr"> >> <attributes> >> <nvpair id="stonith_active_clone_max" name="clone_max" >> value="1"/> >> <nvpair id="stonith_active_clone_node_max" >> name="clone_node_max" value="1"/> >> </attributes> >> </instance_attributes> >> <primitive id="Stonith_active" class="stonith" >> type="external/rsh" provider="heartbeat"> >> <operations> >> <op id="4" name="start" timeout="10s" prereq="nothing"/> >> <op id="5" name="monitor" timeout="5s" interval="15s"/> >> </operations> >> <instance_attributes id="rsh_active_instance_attr"> >> <attributes> >> <nvpair id="rsh_attr_active" name="hostlist" >> value="active"/> >> </attributes> >> </instance_attributes> >> </primitive> >> </clone> >> <clone id="stonith_standby"> >> <instance_attributes id="stonith_standby_attr"> >> <attributes> >> <nvpair id="stonith_standby_clone_max" name="clone_max" >> value="1"/> >> <nvpair id="stonith_standby_clone_node_max" >> name="clone_node_max" value="1"/> >> </attributes> >> </instance_attributes> >> <primitive id="Stonith_standby" class="stonith" >> type="external/rsh" provider="heartbeat"> >> <operations> >> <op id="6" name="start" timeout="10s" prereq="nothing"/> >> <op id="7" name="monitor" timeout="5s" interval="15s"/> >> </operations> >> <instance_attributes id="rsh_standby_instance_attr"> >> <attributes> >> <nvpair id="rsh_attr_standby" name="hostlist" >> value="standby"/> >> </attributes> >> </instance_attributes> >> </primitive> >> </clone> >> </resources> >> <constraints> >> <rsc_location id="proxy_stonith_active_rsc_location" >> rsc="stonith_active"> >> <rule id="prefered_location_proxy_stonith_active_rsc" >> score="INFINITY"> >> <expression >> id="prefered_location_proxy_stonith_active_rsc_expr" >> attribute="#uname" operation="eq" value="standby"/> >> </rule> >> </rsc_location> >> <rsc_location id="proxy_stonith_standby_rsc_location" >> rsc="stonith_standby"> >> <rule id="prefered_location_proxy_stonith_standby_rsc" >> score="INFINITY"> >> <expression >> id="prefered_location_proxy_stonith_standby_rsc_expr" >> attribute="#uname" operation="eq" value="active"/> >> </rule> >> </rsc_location> >> </constraints> >> >> Regards, >> Abhi. > Hi Abhi, > - first I suppose that you have set pulblic keys so that each host > can make ssh to the other without password > - define only one ssh stonith clone resource with hostlist set to > both nodes, with "clone_max" to 2 and "clone_node_max" to 1 > - define 2 place constraints with the clone resource, one with > "uname eq node1" on the other with "uname eq 2" > - start stonith resources > > I strongly recommend you to read the excellent tutorial > "http://linux-ha.org/_cache/HeartbeatTutorials__LCA2007-tutorial.ppt > " to clarify concepts. > > Frank > > -- > Aquest missatge ha estat analitzat per MailScanner a la cerca de > virus i d'altres continguts perillosos, i es considera que està net. > For all your IT requirements visit: http://www.transtec.co.uk > > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > > > The information contained in this electronic message and any > attachments to this message are intended for the exclusive use of > the addressee(s) and may contain proprietary, confidential or > privileged information. If you are not the intended recipient, you > should not disseminate, distribute or copy this e-mail. Please > notify the sender immediately and destroy all copies of this message > and any attachments. > > WARNING: Computer viruses can be transmitted via email. The > recipient should check this email and any attachments for the > presence of viruses. The company accepts no liability for any damage > caused by any virus transmitted by this email. > > www.wipro.com > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
