Hi, On Thu, Dec 20, 2007 at 11:47:55AM +0100, Johan Hoeke wrote: > LS, > > Happy holidays everybody! > > Am please to say I've gotten a cluster up and running, which goes to > show that even regular folks can have a go ;) > > Haven't gotten the hbaping to work on x86_64, but I'll save that for > another day. > > Am testing the failover by disabling a san switch port. (at 10:44 see > attached logs). At 10:44 the cluster failed over from node julia to mode > koch, but did not reboot node julia. A reboot is desired!
You could specify for the monitor action of your resource(s) on_fail=fence. > Can anybody figure out why it didn't reboot the failed node? There was no need. The services failed, but otherwise the node was well behaved. Thanks, Dejan > Any comments about the configs are welcome as well! > > T.I.A. and Happy holidays everybody! > > > system config: > Red Hat Enterprise Linux AS release 4 (Nahant Update 6) > 2.6.9-67.ELsmp > Centos 4.5 RPM's > > > -- > Johan Hoeke <[EMAIL PROTECTED]> ??? UNIX system administration > > Universiteit van Tilburg ??? LIS ??? Postbus 90153, 5000 LE > Bezoekadres : Warandelaan 2 ??? 013 466 3374 ??? http://www.uvt.nl > debugfile /var/log/ha-debug > logfile /var/log/ha-log > logfacility local0 > bcast eth1 # Linux > auto_failback off > node julia.uvt.nl > node koch.uvt.nl > hbaping qlogic-qla2xxx-0 > use_logd yes > crm on > respawn root /usr/lib/heartbeat/pingd -m 100 -d 5s -a pingd > cib[4388]: 2007/12/20_10:27:32 info: cib_stats: Processed 44 operations > (1363.00us average, 0% utilization) in the last 10min > cib[4388]: 2007/12/20_10:43:14 info: cib_stats: Processed 1 operations > (0.00us average, 0% utilization) in the last 10min > attrd[4391]: 2007/12/20_10:45:13 info: attrd_ha_callback: flush message from > julia.uvt.nl > attrd[4391]: 2007/12/20_10:45:13 info: attrd_perform_update: Sent update 6: > pingd=100 > crmd[4392]: 2007/12/20_10:45:14 info: do_lrm_rsc_op: Performing > op=R_137.56.247.151_start_0 key=9:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > IPaddr[4890][4921]: 2007/12/20_10:45:14 INFO: Using calculated nic for > 137.56.247.151: eth0 > IPaddr[4890][4926]: 2007/12/20_10:45:14 INFO: Using calculated netmask for > 137.56.247.151: 255.255.255.0 > IPaddr[4890][4931]: 2007/12/20_10:45:14 DEBUG: Using calculated broadcast for > 137.56.247.151: 137.56.247.255 > IPaddr[4890][4948]: 2007/12/20_10:45:14 INFO: eval ifconfig eth0:0 > 137.56.247.151 netmask 255.255.255.0 broadcast 137.56.247.255 > IPaddr[4890][4953]: 2007/12/20_10:45:14 DEBUG: Sending Gratuitous Arp for > 137.56.247.151 on eth0:0 [eth0] > crmd[4392]: 2007/12/20_10:45:14 info: process_lrm_event: LRM operation > R_137.56.247.151_start_0 (call=11, rc=0) complete > crmd[4392]: 2007/12/20_10:45:14 info: build_operation_update: Digest for > 0:0;9:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9 (R_137.56.247.151_start_0) was > 22e197023bb2e04bb6425008f3b1d0b4 > > crmd[4392]: 2007/12/20_10:45:14 info: log_data_element: > build_operation_update: digest:source <parameters target_role="started" > ip="137.56.247.151"/> > crmd[4392]: 2007/12/20_10:45:15 info: do_lrm_rsc_op: Performing > op=R_san_var_bb_start_0 key=11:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > Filesystem[4978][5007]: 2007/12/20_10:45:15 INFO: Running start for > /dev/mapper/san_var_bb on /var/blackboard > crmd[4392]: 2007/12/20_10:45:15 info: process_lrm_event: LRM operation > R_san_var_bb_start_0 (call=12, rc=0) complete > crmd[4392]: 2007/12/20_10:45:15 info: build_operation_update: Digest for > 0:0;11:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9 (R_san_var_bb_start_0) was > b12952a3cc0f0cfa885d2eee492ee261 > > crmd[4392]: 2007/12/20_10:45:15 info: log_data_element: > build_operation_update: digest:source <parameters target_role="started" > fstype="ext3" directory="/var/blackboard" device="/dev/mapper/san_var_bb"/> > crmd[4392]: 2007/12/20_10:45:16 info: do_lrm_rsc_op: Performing > op=R_san_var_bb_monitor_20000 key=12:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > crmd[4392]: 2007/12/20_10:45:16 info: do_lrm_rsc_op: Performing > op=R_MailTo_start_0 key=14:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > crmd[4392]: 2007/12/20_10:45:16 info: process_lrm_event: LRM operation > R_MailTo_start_0 (call=14, rc=0) complete > crmd[4392]: 2007/12/20_10:45:16 info: build_operation_update: Digest for > 0:0;14:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9 (R_MailTo_start_0) was > 494425cdd1b02f1ad655760f6168534f > > crmd[4392]: 2007/12/20_10:45:16 info: log_data_element: > build_operation_update: digest:source <parameters target_role="started" > email="[EMAIL PROTECTED]" subject="failover occured!"/> > crmd[4392]: 2007/12/20_10:45:17 info: do_lrm_rsc_op: Performing > op=R_MailTo_monitor_10000 key=15:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > crmd[4392]: 2007/12/20_10:45:26 info: process_lrm_event: LRM operation > R_san_var_bb_monitor_20000 (call=13, rc=0) complete > crmd[4392]: 2007/12/20_10:45:27 info: process_lrm_event: LRM operation > R_MailTo_monitor_10000 (call=15, rc=0) complete > debugfile /var/log/ha-debug > logfile /var/log/ha-log > logfacility local0 > bcast eth1 # Linux > auto_failback off > node julia.uvt.nl > node koch.uvt.nl > hbaping qlogic-qla2xxx-0 > use_logd yes > crm on > respawn root /usr/lib/heartbeat/pingd -m 100 -d 5s -a pingd > <cib admin_epoch="0" have_quorum="true" ignore_dtd="false" num_peers="2" > cib_feature_revision="1.3" generated="true" epoch="239" ccm_transition="2" > dc_uuid="ce506f64-da11-4f19-834e-08e52877d1cd" num_updates="63"> > <configuration> > <crm_config> > <cluster_property_set id="cib-bootstrap-options"> > <attributes> > <nvpair name="symmetric-cluster" > id="cib-bootstrap-options-symmetric-cluster" value="True"/> > <nvpair id="cib-bootstrap-options-stonith-enabled" > name="stonith-enabled" value="True"/> > <nvpair id="cib-bootstrap-options-stonith-action" > name="stonith-action" value="reboot"/> > <nvpair > id="cib-bootstrap-options-default-resource-failure-stickiness" > name="default-resource-failure-stickiness" value="0"/> > <nvpair id="cib-bootstrap-options-default-resource-stickiness" > name="default-resource-stickiness" value="0"/> > <nvpair id="cib-bootstrap-options-no-quorum-policy" > name="no-quorum-policy" value="stop"/> > <nvpair id="cib-bootstrap-options-stop-orphan-resources" > name="stop-orphan-resources" value="true"/> > <nvpair id="cib-bootstrap-options-stop-orphan-actions" > name="stop-orphan-actions" value="true"/> > <nvpair id="cib-bootstrap-options-remove-after-stop" > name="remove-after-stop" value="false"/> > <nvpair id="cib-bootstrap-options-short-resource-names" > name="short-resource-names" value="true"/> > <nvpair id="cib-bootstrap-options-transition-idle-timeout" > name="transition-idle-timeout" value="5min"/> > <nvpair id="cib-bootstrap-options-default-action-timeout" > name="default-action-timeout" value="15s"/> > <nvpair id="cib-bootstrap-options-is-managed-default" > name="is-managed-default" value="true"/> > <nvpair name="last-lrm-refresh" > id="cib-bootstrap-options-last-lrm-refresh" value="1197643205"/> > </attributes> > </cluster_property_set> > </crm_config> > <nodes> > <node uname="koch.uvt.nl" type="normal" > id="858b162c-1520-445b-9ab9-e4aeb0d28df3"> > <instance_attributes id="nodes-858b162c-1520-445b-9ab9-e4aeb0d28df3"> > <attributes> > <nvpair name="standby" > id="standby-858b162c-1520-445b-9ab9-e4aeb0d28df3" value="off"/> > </attributes> > </instance_attributes> > </node> > <node uname="julia.uvt.nl" type="normal" > id="ce506f64-da11-4f19-834e-08e52877d1cd"> > <instance_attributes id="nodes-ce506f64-da11-4f19-834e-08e52877d1cd"> > <attributes> > <nvpair name="standby" > id="standby-ce506f64-da11-4f19-834e-08e52877d1cd" value="off"/> > </attributes> > </instance_attributes> > </node> > </nodes> > <resources> > <group ordered="true" collocated="true" id="G_dloprd"> > <primitive id="R_137.56.247.151" class="ocf" type="IPaddr" > provider="heartbeat"> > <instance_attributes id="R_137.56.247.151_instance_attrs"> > <attributes> > <nvpair id="R_137.56.247.151_target_role" name="target_role" > value="started"/> > <nvpair id="600a6be9-16d0-4c5d-afff-9a8d259b6a64" name="ip" > value="137.56.247.151"/> > </attributes> > </instance_attributes> > </primitive> > <primitive id="R_san_var_bb" class="ocf" type="Filesystem" > provider="heartbeat"> > <instance_attributes id="R_san_var_bb_instance_attrs"> > <attributes> > <nvpair id="R_san_var_bb_target_role" name="target_role" > value="started"/> > <nvpair id="09883f7c-08d2-4a9e-ab69-9726842cb86f" > name="fstype" value="ext3"/> > <nvpair id="0ea4b895-f2c7-4a02-b961-58ac309bd38c" > name="device" value="/dev/mapper/san_var_bb"/> > <nvpair id="66169c2f-d4b3-4d62-a811-e19eb84b9773" > name="directory" value="/var/blackboard"/> > </attributes> > </instance_attributes> > <operations> > <op id="b4501ccc-d278-4995-87ca-2a46cc5d4913" name="monitor" > interval="20" timeout="40" start_delay="10" disabled="false" role="Started"/> > <op id="efbefb58-3738-4f36-953f-ac2937f2cbf3" name="notify" > timeout="60" start_delay="0" disabled="false" role="Started"/> > <op id="9d3079e5-a2d4-4254-acba-7522edca83fc" name="start" > timeout="60"/> > <op id="fe02095f-97c3-4805-a07c-a9bc3b322cc9" name="stop" > timeout="60"/> > </operations> > </primitive> > <instance_attributes id="G_dloprd_instance_attrs"> > <attributes> > <nvpair id="G_dloprd_target_role" name="target_role" > value="started"/> > </attributes> > </instance_attributes> > <primitive class="ocf" type="MailTo" provider="heartbeat" > id="R_MailTo"> > <instance_attributes id="R_MailTo_instance_attrs"> > <attributes> > <nvpair name="target_role" id="R_MailTo_target_role" > value="started"/> > <nvpair id="856115e7-e5aa-4f55-9720-4c3d25a0c3ed" name="email" > value="[EMAIL PROTECTED]"/> > <nvpair id="c58c4d84-3781-44eb-875a-d7f9a3e0d500" > name="subject" value="failover occured!"/> > </attributes> > </instance_attributes> > <operations> > <op id="f15ba3a8-96c9-4c04-a06b-4b37ce5b1f1c" name="start" > timeout="10"/> > <op id="49828b21-5fc0-425e-a7c2-1f7660846cc3" name="stop" > timeout="10"/> > <op id="88f9c6b8-0d67-4d53-a9c9-f602c031efa2" name="monitor" > interval="10" timeout="10" start_delay="10"/> > </operations> > </primitive> > </group> > <clone id="CL_ilo_julia"> > <instance_attributes id="CL_ilo_julia_instance_attrs"> > <attributes> > <nvpair id="CL_ilo_julia_clone_max" name="clone_max" value="1"/> > <nvpair id="CL_ilo_julia_clone_node_max" name="clone_node_max" > value="1"/> > </attributes> > </instance_attributes> > <primitive class="stonith" type="external/riloe" > provider="heartbeat" id="R_ilo_julia"> > <instance_attributes id="R_ilo_julia_instance_attrs"> > <attributes> > <nvpair name="target_role" id="R_ilo_julia_target_role" > value="started"/> > <nvpair id="fc761517-9087-4d05-ba10-eb6650ae3b64" > name="hostlist" value="julia.uvt.nl"/> > <nvpair id="39beea78-cf87-486a-abe3-01acff706e68" > name="ilo_hostname" value="julia.ilo.uvt.nl"/> > <nvpair id="7434f914-1d12-4737-92fc-31c31649f676" > name="ilo_user" value="Administrator"/> > <nvpair id="749f74a8-089b-406b-b2be-bdf04406f7ac" > name="ilo_password" value="XXXXXXXX"/> > <nvpair id="11a26c03-e4a0-41e7-bc83-f40cb2fccd6c" > name="ilo_can_reset" value="1"/> > <nvpair id="e5731656-421a-4f6a-b41e-485afa198459" > name="ilo_protocol" value="2.0"/> > <nvpair id="a9792000-93a2-4ea3-802e-7f547f8f3537" > name="ilo_powerdown_method" value="button"/> > </attributes> > </instance_attributes> > <operations> > <op id="0bda4047-9e9f-4ed7-96f2-bd87b0d73a00" name="start" > timeout="15" start_delay="0" disabled="false" role="Started"/> > <op id="17e2ffad-1f7f-43be-ab16-2d4f6de2d34e" name="stop" > timeout="15" start_delay="0" disabled="false" role="Started"/> > <op id="3e30a6b4-2e1c-4c74-adb9-84259249a587" name="monitor" > interval="15" timeout="15" start_delay="15" disabled="false" role="Started"/> > </operations> > </primitive> > </clone> > <clone id="CL_ilo_koch"> > <instance_attributes id="CL_ilo_koch_instance_attrs"> > <attributes> > <nvpair id="CL_ilo_koch_clone_max" name="clone_max" value="1"/> > <nvpair id="CL_ilo_koch_clone_node_max" name="clone_node_max" > value="1"/> > <nvpair id="CL_ilo_koch_target_role" name="target_role" > value="started"/> > </attributes> > </instance_attributes> > <primitive class="stonith" type="external/riloe" > provider="heartbeat" id="R_ilo_koch"> > <instance_attributes id="R_ilo_koch_instance_attrs"> > <attributes> > <nvpair id="R_ilo_koch_target_role" name="target_role" > value="started"/> > <nvpair id="4caf2853-5703-45a5-889a-8e9d9f796be9" > name="hostlist" value="koch.uvt.nl"/> > <nvpair id="451d08c5-1dce-4925-8853-52f7ea792334" > name="ilo_hostname" value="koch.ilo.uvt.nl"/> > <nvpair id="17bfd81d-5b22-44ad-aa9f-1d957ea07aa1" > name="ilo_user" value="Administrator"/> > <nvpair id="c61908dd-b9f6-4a7f-b439-20a97253aca0" > name="ilo_password" value="XXXXXXXX"/> > <nvpair id="aae2f839-51c8-43dc-b90a-aaabcd39eebe" > name="ilo_can_reset" value="1"/> > <nvpair id="ba41204d-6cc6-4c2e-bc1d-ba59c7dbdfeb" > name="ilo_protocol" value="2.0"/> > <nvpair id="582676bb-b522-4ab5-98cf-c4411cafef69" > name="ilo_powerdown_method" value="button"/> > </attributes> > </instance_attributes> > <operations> > <op id="06f9f7a2-bbb0-42b0-9b9d-0610c6147c07" name="start" > timeout="15"/> > <op id="977bfd99-bb7c-4447-8ff6-cb3ad18c53dd" name="stop" > timeout="15"/> > <op id="5d613e86-8954-4a45-8ea8-9d65496d2106" name="monitor" > interval="15" timeout="15" start_delay="15"/> > </operations> > </primitive> > </clone> > </resources> > <constraints> > <rsc_location id="G_dloprd:connected" rsc="G_dloprd"> > <rule id="G_dloprd:connected:rule" score="-INFINITY" boolean_op="or"> > <expression id="G_dloprd:connected:expr:undefined" > attribute="pingd" operation="not_defined"/> > <expression id="G_dloprd:connected:expr:zero" attribute="pingd" > operation="lte" value="0"/> > </rule> > </rsc_location> > <rsc_location id="P_dloprd" rsc="G_dloprd"> > <rule id="prefered_P_dloprd" score="100"> > <expression attribute="#uname" > id="c1abbb35-ec88-41f5-b248-9d8ec11891e6" operation="eq" > value="julia.uvt.nl"/> > </rule> > </rsc_location> > <rsc_location id="P_ilo_julia" rsc="CL_ilo_julia"> > <rule id="prefered_P_ilo_julia" score="INFINITY"> > <expression attribute="#uname" > id="01d3f546-8486-403d-b1af-f1153c636972" operation="eq" > value="julia.uvt.nl"/> > </rule> > </rsc_location> > <rsc_location id="P_ilo_koch" rsc="CL_ilo_koch"> > <rule id="prefered_P_ilo_koch" score="INFINITY"> > <expression attribute="#uname" > id="95c99f07-d9e2-4ab7-bc0c-c90de58301ad" operation="eq" value="koch.uvt.nl"/> > </rule> > </rsc_location> > </constraints> > </configuration> > <status> > <node_state uname="julia.uvt.nl" crmd="online" shutdown="0" > in_ccm="true" ha="active" join="member" expected="member" > id="ce506f64-da11-4f19-834e-08e52877d1cd"> > <transient_attributes id="ce506f64-da11-4f19-834e-08e52877d1cd"> > <instance_attributes > id="status-ce506f64-da11-4f19-834e-08e52877d1cd"> > <attributes> > <nvpair name="pingd" > id="status-ce506f64-da11-4f19-834e-08e52877d1cd-pingd" value="0"/> > <nvpair > id="status-ce506f64-da11-4f19-834e-08e52877d1cd-probe_complete" > name="probe_complete" value="true"/> > <nvpair > id="status-ce506f64-da11-4f19-834e-08e52877d1cd-fail-count-R_ilo_julia:0" > name="fail-count-R_ilo_julia:0" value="1"/> > <nvpair > id="status-ce506f64-da11-4f19-834e-08e52877d1cd-fail-count-R_ilo_koch:0" > name="fail-count-R_ilo_koch:0" value="1"/> > </attributes> > </instance_attributes> > </transient_attributes> > <lrm id="ce506f64-da11-4f19-834e-08e52877d1cd"> > <lrm_resources> > <lrm_resource type="external/riloe" class="stonith" > provider="heartbeat" id="R_ilo_julia:0"> > <lrm_rsc_op operation="monitor" > transition_key="6:0:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="5" > crm_feature_set="1.0.9" rc_code="7" interval="0" > op_digest="1d89283d54188918c08e4deda8bd4e2e" id="R_ilo_julia:0_monitor_0" > transition_magic="4:7;6:0:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > op_status="4"/> > <lrm_rsc_op id="R_ilo_julia:0_start_0" operation="start" > crm-debug-origin="do_update_resource" > transition_key="19:2:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="4:1;19:2:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="19" > crm_feature_set="1.0.9" rc_code="1" op_status="4" interval="0" > op_digest="1d89283d54188918c08e4deda8bd4e2e"/> > <lrm_rsc_op id="R_ilo_julia:0_stop_0" operation="stop" > crm-debug-origin="do_update_resource" > transition_key="1:3:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;1:3:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="21" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="1d89283d54188918c08e4deda8bd4e2e"/> > </lrm_resource> > <lrm_resource type="MailTo" class="ocf" provider="heartbeat" > id="R_MailTo"> > <lrm_rsc_op operation="monitor" > transition_key="5:0:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="4" > crm_feature_set="1.0.9" rc_code="7" interval="0" > op_digest="494425cdd1b02f1ad655760f6168534f" id="R_MailTo_monitor_0" > transition_magic="4:7;5:0:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > op_status="4"/> > <lrm_rsc_op id="R_MailTo_start_0" operation="start" > crm-debug-origin="do_update_resource" > transition_key="7:1:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;7:1:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="15" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="494425cdd1b02f1ad655760f6168534f"/> > <lrm_rsc_op id="R_MailTo_stop_0" operation="stop" > crm-debug-origin="do_update_resource" > transition_key="13:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;13:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="23" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="494425cdd1b02f1ad655760f6168534f"/> > </lrm_resource> > <lrm_resource type="IPaddr" class="ocf" provider="heartbeat" > id="R_137.56.247.151"> > <lrm_rsc_op operation="monitor" > transition_key="3:0:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="2" > crm_feature_set="1.0.9" rc_code="7" interval="0" > op_digest="22e197023bb2e04bb6425008f3b1d0b4" id="R_137.56.247.151_monitor_0" > transition_magic="4:7;3:0:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > op_status="4"/> > <lrm_rsc_op id="R_137.56.247.151_start_0" operation="start" > crm-debug-origin="do_update_resource" > transition_key="4:1:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;4:1:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="10" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="22e197023bb2e04bb6425008f3b1d0b4"/> > <lrm_rsc_op id="R_137.56.247.151_stop_0" operation="stop" > crm-debug-origin="do_update_resource" > transition_key="8:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;8:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="25" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="22e197023bb2e04bb6425008f3b1d0b4"/> > </lrm_resource> > <lrm_resource type="Filesystem" class="ocf" provider="heartbeat" > id="R_san_var_bb"> > <lrm_rsc_op operation="monitor" > transition_key="4:0:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="3" > crm_feature_set="1.0.9" rc_code="7" interval="0" > op_digest="b12952a3cc0f0cfa885d2eee492ee261" id="R_san_var_bb_monitor_0" > transition_magic="4:7;4:0:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > op_status="4"/> > <lrm_rsc_op id="R_san_var_bb_start_0" operation="start" > crm-debug-origin="do_update_resource" > transition_key="5:1:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;5:1:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="13" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="b12952a3cc0f0cfa885d2eee492ee261"/> > <lrm_rsc_op id="R_san_var_bb_stop_0" operation="stop" > crm-debug-origin="do_update_resource" > transition_key="10:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;10:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="24" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="b12952a3cc0f0cfa885d2eee492ee261"/> > </lrm_resource> > <lrm_resource type="external/riloe" class="stonith" > provider="heartbeat" id="R_ilo_koch:0"> > <lrm_rsc_op operation="monitor" > transition_key="7:0:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="6" > crm_feature_set="1.0.9" rc_code="7" interval="0" > op_digest="a0ea723a0a77c5bab4e634f692ef08ab" id="R_ilo_koch:0_monitor_0" > transition_magic="4:7;7:0:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > op_status="4"/> > <lrm_rsc_op id="R_ilo_koch:0_start_0" operation="start" > crm-debug-origin="do_update_resource" > transition_key="24:2:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="4:1;24:2:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="20" > crm_feature_set="1.0.9" rc_code="1" op_status="4" interval="0" > op_digest="a0ea723a0a77c5bab4e634f692ef08ab"/> > <lrm_rsc_op id="R_ilo_koch:0_stop_0" operation="stop" > crm-debug-origin="do_update_resource" > transition_key="4:3:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;4:3:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="22" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="a0ea723a0a77c5bab4e634f692ef08ab"/> > </lrm_resource> > </lrm_resources> > </lrm> > </node_state> > <node_state uname="koch.uvt.nl" ha="active" in_ccm="true" crmd="online" > shutdown="0" id="858b162c-1520-445b-9ab9-e4aeb0d28df3" join="member" > expected="member"> > <lrm id="858b162c-1520-445b-9ab9-e4aeb0d28df3"> > <lrm_resources> > <lrm_resource id="R_ilo_julia:0" type="external/riloe" > class="stonith" provider="heartbeat"> > <lrm_rsc_op id="R_ilo_julia:0_monitor_0" operation="monitor" > crm-debug-origin="do_update_resource" > transition_key="8:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:7;8:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="5" > crm_feature_set="1.0.9" rc_code="7" op_status="0" interval="0" > op_digest="1d89283d54188918c08e4deda8bd4e2e"/> > <lrm_rsc_op id="R_ilo_julia:0_start_0" operation="start" > crm-debug-origin="do_update_resource" > transition_key="21:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;21:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="7" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="1d89283d54188918c08e4deda8bd4e2e"/> > <lrm_rsc_op id="R_ilo_julia:0_monitor_15000" operation="monitor" > crm-debug-origin="do_update_resource" > transition_key="18:5:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;18:5:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="9" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="15000" > op_digest="1d89283d54188918c08e4deda8bd4e2e"/> > </lrm_resource> > <lrm_resource id="R_MailTo" type="MailTo" class="ocf" > provider="heartbeat"> > <lrm_rsc_op id="R_MailTo_monitor_0" operation="monitor" > crm-debug-origin="do_update_resource" > transition_key="7:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:7;7:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="4" > crm_feature_set="1.0.9" rc_code="7" op_status="0" interval="0" > op_digest="494425cdd1b02f1ad655760f6168534f"/> > <lrm_rsc_op id="R_MailTo_start_0" operation="start" > crm-debug-origin="do_update_resource" > transition_key="14:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;14:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="14" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="494425cdd1b02f1ad655760f6168534f"/> > <lrm_rsc_op id="R_MailTo_monitor_10000" operation="monitor" > crm-debug-origin="do_update_resource" > transition_key="15:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;15:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="15" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="10000" > op_digest="494425cdd1b02f1ad655760f6168534f"/> > </lrm_resource> > <lrm_resource id="R_137.56.247.151" type="IPaddr" class="ocf" > provider="heartbeat"> > <lrm_rsc_op id="R_137.56.247.151_monitor_0" operation="monitor" > crm-debug-origin="do_update_resource" > transition_key="5:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:7;5:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="2" > crm_feature_set="1.0.9" rc_code="7" op_status="0" interval="0" > op_digest="22e197023bb2e04bb6425008f3b1d0b4"/> > <lrm_rsc_op id="R_137.56.247.151_start_0" operation="start" > crm-debug-origin="do_update_resource" > transition_key="9:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;9:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="11" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="22e197023bb2e04bb6425008f3b1d0b4"/> > </lrm_resource> > <lrm_resource id="R_san_var_bb" type="Filesystem" class="ocf" > provider="heartbeat"> > <lrm_rsc_op id="R_san_var_bb_monitor_0" operation="monitor" > crm-debug-origin="do_update_resource" > transition_key="6:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:7;6:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="3" > crm_feature_set="1.0.9" rc_code="7" op_status="0" interval="0" > op_digest="b12952a3cc0f0cfa885d2eee492ee261"/> > <lrm_rsc_op id="R_san_var_bb_start_0" operation="start" > crm-debug-origin="do_update_resource" > transition_key="11:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;11:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="12" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="b12952a3cc0f0cfa885d2eee492ee261"/> > <lrm_rsc_op id="R_san_var_bb_monitor_20000" operation="monitor" > crm-debug-origin="do_update_resource" > transition_key="12:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;12:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="13" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="20000" > op_digest="b12952a3cc0f0cfa885d2eee492ee261"/> > </lrm_resource> > <lrm_resource id="R_ilo_koch:0" type="external/riloe" > class="stonith" provider="heartbeat"> > <lrm_rsc_op id="R_ilo_koch:0_monitor_0" operation="monitor" > crm-debug-origin="do_update_resource" > transition_key="9:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:7;9:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="6" > crm_feature_set="1.0.9" rc_code="7" op_status="0" interval="0" > op_digest="a0ea723a0a77c5bab4e634f692ef08ab"/> > <lrm_rsc_op id="R_ilo_koch:0_start_0" operation="start" > crm-debug-origin="do_update_resource" > transition_key="27:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;27:4:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="8" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="0" > op_digest="a0ea723a0a77c5bab4e634f692ef08ab"/> > <lrm_rsc_op id="R_ilo_koch:0_monitor_15000" operation="monitor" > crm-debug-origin="do_update_resource" > transition_key="25:5:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" > transition_magic="0:0;25:5:68904856-5ee2-4bbe-b49d-cd77fa79f7e9" call_id="10" > crm_feature_set="1.0.9" rc_code="0" op_status="0" interval="15000" > op_digest="a0ea723a0a77c5bab4e634f692ef08ab"/> > </lrm_resource> > </lrm_resources> > </lrm> > <transient_attributes id="858b162c-1520-445b-9ab9-e4aeb0d28df3"> > <instance_attributes > id="status-858b162c-1520-445b-9ab9-e4aeb0d28df3"> > <attributes> > <nvpair > id="status-858b162c-1520-445b-9ab9-e4aeb0d28df3-probe_complete" > name="probe_complete" value="true"/> > <nvpair id="status-858b162c-1520-445b-9ab9-e4aeb0d28df3-pingd" > name="pingd" value="100"/> > </attributes> > </instance_attributes> > </transient_attributes> > </node_state> > </status> > </cib> > > cib[4337]: 2007/12/20_10:33:56 info: cib_stats: Processed 40 operations > (4000.00us average, 0% utilization) in the last 10min > heartbeat[3550]: 2007/12/20_10:45:08 WARN: node qlogic-qla2xxx-0: is dead > heartbeat[3550]: 2007/12/20_10:45:08 info: Link > qlogic-qla2xxx-0:qlogic-qla2xxx-0 dead. > pingd[4343]: 2007/12/20_10:45:08 notice: pingd_nstatus_callback: Status > update: Ping node qlogic-qla2xxx-0 now has status [dead] > crmd[4341]: 2007/12/20_10:45:08 notice: crmd_ha_status_callback: Status > update: Node qlogic-qla2xxx-0 now has status [dead] > pingd[4343]: 2007/12/20_10:45:08 info: send_update: 0 active ping nodes > crmd[4341]: 2007/12/20_10:45:08 WARN: get_uuid: Could not calculate UUID for > qlogic-qla2xxx-0 > pingd[4343]: 2007/12/20_10:45:08 notice: pingd_lstatus_callback: Status > update: Ping node qlogic-qla2xxx-0 now has status [dead] > pingd[4343]: 2007/12/20_10:45:08 notice: pingd_nstatus_callback: Status > update: Ping node qlogic-qla2xxx-0 now has status [dead] > pingd[4343]: 2007/12/20_10:45:08 info: send_update: 0 active ping nodes > attrd[4340]: 2007/12/20_10:45:13 info: attrd_trigger_update: Sending flush op > to all hosts for: pingd > attrd[4340]: 2007/12/20_10:45:13 info: attrd_ha_callback: flush message from > julia.uvt.nl > attrd[4340]: 2007/12/20_10:45:13 info: attrd_perform_update: Sent update 11: > pingd=0 > tengine[4346]: 2007/12/20_10:45:13 info: extract_event: Aborting on > transient_attributes changes for ce506f64-da11-4f19-834e-08e52877d1cd > crmd[4341]: 2007/12/20_10:45:13 info: do_state_transition: State transition > S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE > origin=route_message ] > tengine[4346]: 2007/12/20_10:45:13 info: update_abort_priority: Abort > priority upgraded to 1000000 > crmd[4341]: 2007/12/20_10:45:13 info: do_state_transition: All 2 cluster > nodes are eligible to run resources. > tengine[4346]: 2007/12/20_10:45:13 info: te_update_diff: Aborting on > transient_attributes deletions > pengine[4347]: 2007/12/20_10:45:13 notice: cluster_option: Using default > value '60s' for cluster option 'cluster-delay' > pengine[4347]: 2007/12/20_10:45:13 notice: cluster_option: Using default > value '-1' for cluster option 'pe-error-series-max' > pengine[4347]: 2007/12/20_10:45:13 notice: cluster_option: Using default > value '-1' for cluster option 'pe-warn-series-max' > pengine[4347]: 2007/12/20_10:45:13 notice: cluster_option: Using default > value '-1' for cluster option 'pe-input-series-max' > pengine[4347]: 2007/12/20_10:45:13 notice: cluster_option: Using default > value 'true' for cluster option 'startup-fencing' > pengine[4347]: 2007/12/20_10:45:13 info: determine_online_status: Node > julia.uvt.nl is online > pengine[4347]: 2007/12/20_10:45:13 WARN: unpack_rsc_op: Processing failed op > (R_ilo_koch:0_start_0) on julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 WARN: unpack_rsc_op: Handling failed start > for R_ilo_koch:0 on julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 WARN: unpack_rsc_op: Processing failed op > (R_ilo_julia:0_start_0) on julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 WARN: unpack_rsc_op: Handling failed start > for R_ilo_julia:0 on julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 info: determine_online_status: Node > koch.uvt.nl is online > pengine[4347]: 2007/12/20_10:45:13 info: group_print: Resource Group: G_dloprd > pengine[4347]: 2007/12/20_10:45:13 info: native_print: R_137.56.247.151 > (heartbeat::ocf:IPaddr): Started julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 info: native_print: R_san_var_bb > (heartbeat::ocf:Filesystem): Started julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 info: native_print: R_MailTo > (heartbeat::ocf:MailTo): Started julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 info: clone_print: Clone Set: CL_ilo_julia > pengine[4347]: 2007/12/20_10:45:13 info: native_print: R_ilo_julia:0 > (stonith:external/riloe): Started koch.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 info: clone_print: Clone Set: CL_ilo_koch > pengine[4347]: 2007/12/20_10:45:13 info: native_print: R_ilo_koch:0 > (stonith:external/riloe): Started koch.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 notice: NoRoleChange: Move resource > R_137.56.247.151 (julia.uvt.nl -> koch.uvt.nl) > pengine[4347]: 2007/12/20_10:45:13 notice: StopRsc: julia.uvt.nl Stop > R_137.56.247.151 > pengine[4347]: 2007/12/20_10:45:13 notice: StartRsc: koch.uvt.nl Start > R_137.56.247.151 > pengine[4347]: 2007/12/20_10:45:13 notice: NoRoleChange: Move resource > R_san_var_bb (julia.uvt.nl -> koch.uvt.nl) > pengine[4347]: 2007/12/20_10:45:13 notice: StopRsc: julia.uvt.nl Stop > R_san_var_bb > pengine[4347]: 2007/12/20_10:45:13 notice: StartRsc: koch.uvt.nl Start > R_san_var_bb > pengine[4347]: 2007/12/20_10:45:13 notice: RecurringOp: koch.uvt.nl > R_san_var_bb_monitor_20000 > pengine[4347]: 2007/12/20_10:45:13 notice: NoRoleChange: Move resource > R_MailTo (julia.uvt.nl -> koch.uvt.nl) > pengine[4347]: 2007/12/20_10:45:13 notice: StopRsc: julia.uvt.nl Stop > R_MailTo > pengine[4347]: 2007/12/20_10:45:13 notice: StartRsc: koch.uvt.nl Start > R_MailTo > pengine[4347]: 2007/12/20_10:45:13 notice: RecurringOp: koch.uvt.nl > R_MailTo_monitor_10000 > pengine[4347]: 2007/12/20_10:45:13 notice: NoRoleChange: Leave resource > R_ilo_julia:0 (koch.uvt.nl) > pengine[4347]: 2007/12/20_10:45:13 notice: NoRoleChange: Leave resource > R_ilo_koch:0 (koch.uvt.nl) > crmd[4341]: 2007/12/20_10:45:13 info: do_state_transition: State transition > S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS > cause=C_IPC_MESSAGE origin=route_message ] > tengine[4346]: 2007/12/20_10:45:13 info: unpack_graph: Unpacked transition 7: > 12 actions in 12 synapses > tengine[4346]: 2007/12/20_10:45:13 info: te_pseudo_action: Pseudo action 18 > fired and confirmed > tengine[4346]: 2007/12/20_10:45:13 info: send_rsc_command: Initiating action > 13: R_MailTo_stop_0 on julia.uvt.nl > crmd[4341]: 2007/12/20_10:45:13 info: do_lrm_rsc_op: Performing > op=R_MailTo_stop_0 key=13:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > crmd[4341]: 2007/12/20_10:45:13 info: process_lrm_event: LRM operation > R_MailTo_monitor_10000 (call=16, rc=-2) Cancelled > pengine[4347]: 2007/12/20_10:45:13 info: process_pe_message: Transition 7: > PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-94.bz2 > crmd[4341]: 2007/12/20_10:45:13 info: process_lrm_event: LRM operation > R_MailTo_stop_0 (call=23, rc=0) complete > tengine[4346]: 2007/12/20_10:45:13 info: match_graph_event: Action > R_MailTo_stop_0 (13) confirmed on julia.uvt.nl > tengine[4346]: 2007/12/20_10:45:13 info: send_rsc_command: Initiating action > 10: R_san_var_bb_stop_0 on julia.uvt.nl > crmd[4341]: 2007/12/20_10:45:13 info: do_lrm_rsc_op: Performing > op=R_san_var_bb_stop_0 key=10:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > crmd[4341]: 2007/12/20_10:45:13 info: process_lrm_event: LRM operation > R_san_var_bb_monitor_20000 (call=14, rc=-2) Cancelled > Filesystem[12743][12774]: 2007/12/20_10:45:13 INFO: Running stop for > /dev/mapper/san_var_bb on /var/blackboard > Filesystem[12743][12784]: 2007/12/20_10:45:13 INFO: Trying to unmount > /var/blackboard > Filesystem[12743][12786]: 2007/12/20_10:45:13 INFO: unmounted /var/blackboard > successfully > lrmd[4338]: 2007/12/20_10:45:13 info: RA output: (R_san_var_bb:stop:stderr) > BLKFLSBUF: Input/output error > > crmd[4341]: 2007/12/20_10:45:13 info: process_lrm_event: LRM operation > R_san_var_bb_stop_0 (call=24, rc=0) complete > tengine[4346]: 2007/12/20_10:45:13 info: match_graph_event: Action > R_san_var_bb_stop_0 (10) confirmed on julia.uvt.nl > tengine[4346]: 2007/12/20_10:45:13 info: send_rsc_command: Initiating action > 8: R_137.56.247.151_stop_0 on julia.uvt.nl > crmd[4341]: 2007/12/20_10:45:13 info: do_lrm_rsc_op: Performing > op=R_137.56.247.151_stop_0 key=8:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > lrmd[4338]: 2007/12/20_10:45:13 info: RA output: > (R_137.56.247.151:stop:stdout) In IP Stop > > lrmd[4338]: 2007/12/20_10:45:13 info: RA output: > (R_137.56.247.151:stop:stderr) SIOCDELRT: No such process > > IPaddr[12792][12807]: 2007/12/20_10:45:13 INFO: ifconfig eth0:0 down > crmd[4341]: 2007/12/20_10:45:13 info: process_lrm_event: LRM operation > R_137.56.247.151_stop_0 (call=25, rc=0) complete > tengine[4346]: 2007/12/20_10:45:13 info: match_graph_event: Action > R_137.56.247.151_stop_0 (8) confirmed on julia.uvt.nl > tengine[4346]: 2007/12/20_10:45:13 info: te_pseudo_action: Pseudo action 19 > fired and confirmed > tengine[4346]: 2007/12/20_10:45:13 info: te_pseudo_action: Pseudo action 16 > fired and confirmed > tengine[4346]: 2007/12/20_10:45:13 info: send_rsc_command: Initiating action > 9: R_137.56.247.151_start_0 on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:14 info: match_graph_event: Action > R_137.56.247.151_start_0 (9) confirmed on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:14 info: send_rsc_command: Initiating action > 11: R_san_var_bb_start_0 on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:15 info: match_graph_event: Action > R_san_var_bb_start_0 (11) confirmed on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:15 info: send_rsc_command: Initiating action > 12: R_san_var_bb_monitor_20000 on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:15 info: send_rsc_command: Initiating action > 14: R_MailTo_start_0 on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:16 info: match_graph_event: Action > R_MailTo_start_0 (14) confirmed on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:16 info: te_pseudo_action: Pseudo action 17 > fired and confirmed > tengine[4346]: 2007/12/20_10:45:16 info: send_rsc_command: Initiating action > 15: R_MailTo_monitor_10000 on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:26 info: match_graph_event: Action > R_san_var_bb_monitor_20000 (12) confirmed on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:28 info: match_graph_event: Action > R_MailTo_monitor_10000 (15) confirmed on koch.uvt.nl > crmd[4341]: 2007/12/20_10:45:28 info: do_state_transition: State transition > S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE > origin=route_message ] > tengine[4346]: 2007/12/20_10:45:28 info: run_graph: Transition 7: > (Complete=12, Pending=0, Fired=0, Skipped=0, Incomplete=0) > tengine[4346]: 2007/12/20_10:45:28 info: notify_crmd: Transition 7 status: > te_complete - <null> > cib[4337]: 2007/12/20_10:49:26 info: cib_stats: Processed 23 operations > (4347.00us average, 0% utilization) in the last 10min > cib[4337]: 2007/12/20_10:23:56 info: cib_stats: Processed 53 operations > (3773.00us average, 0% utilization) in the last 10min > cib[4337]: 2007/12/20_10:33:56 info: cib_stats: Processed 40 operations > (4000.00us average, 0% utilization) in the last 10min > heartbeat[3550]: 2007/12/20_10:45:08 WARN: node qlogic-qla2xxx-0: is dead > heartbeat[3550]: 2007/12/20_10:45:08 info: Link > qlogic-qla2xxx-0:qlogic-qla2xxx-0 dead. > pingd[4343]: 2007/12/20_10:45:08 notice: pingd_nstatus_callback: Status > update: Ping node qlogic-qla2xxx-0 now has status [dead] > crmd[4341]: 2007/12/20_10:45:08 notice: crmd_ha_status_callback: Status > update: Node qlogic-qla2xxx-0 now has status [dead] > pingd[4343]: 2007/12/20_10:45:08 info: send_update: 0 active ping nodes > crmd[4341]: 2007/12/20_10:45:08 WARN: get_uuid: Could not calculate UUID for > qlogic-qla2xxx-0 > pingd[4343]: 2007/12/20_10:45:08 notice: pingd_lstatus_callback: Status > update: Ping node qlogic-qla2xxx-0 now has status [dead] > pingd[4343]: 2007/12/20_10:45:08 notice: pingd_nstatus_callback: Status > update: Ping node qlogic-qla2xxx-0 now has status [dead] > pingd[4343]: 2007/12/20_10:45:08 info: send_update: 0 active ping nodes > attrd[4340]: 2007/12/20_10:45:13 info: attrd_trigger_update: Sending flush op > to all hosts for: pingd > attrd[4340]: 2007/12/20_10:45:13 info: attrd_ha_callback: flush message from > julia.uvt.nl > attrd[4340]: 2007/12/20_10:45:13 info: attrd_perform_update: Sent update 11: > pingd=0 > tengine[4346]: 2007/12/20_10:45:13 info: extract_event: Aborting on > transient_attributes changes for ce506f64-da11-4f19-834e-08e52877d1cd > crmd[4341]: 2007/12/20_10:45:13 info: do_state_transition: State transition > S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE > origin=route_message ] > tengine[4346]: 2007/12/20_10:45:13 info: update_abort_priority: Abort > priority upgraded to 1000000 > crmd[4341]: 2007/12/20_10:45:13 info: do_state_transition: All 2 cluster > nodes are eligible to run resources. > tengine[4346]: 2007/12/20_10:45:13 info: te_update_diff: Aborting on > transient_attributes deletions > pengine[4347]: 2007/12/20_10:45:13 notice: cluster_option: Using default > value '60s' for cluster option 'cluster-delay' > pengine[4347]: 2007/12/20_10:45:13 notice: cluster_option: Using default > value '-1' for cluster option 'pe-error-series-max' > pengine[4347]: 2007/12/20_10:45:13 notice: cluster_option: Using default > value '-1' for cluster option 'pe-warn-series-max' > pengine[4347]: 2007/12/20_10:45:13 notice: cluster_option: Using default > value '-1' for cluster option 'pe-input-series-max' > pengine[4347]: 2007/12/20_10:45:13 notice: cluster_option: Using default > value 'true' for cluster option 'startup-fencing' > pengine[4347]: 2007/12/20_10:45:13 info: determine_online_status: Node > julia.uvt.nl is online > pengine[4347]: 2007/12/20_10:45:13 WARN: unpack_rsc_op: Processing failed op > (R_ilo_koch:0_start_0) on julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 WARN: unpack_rsc_op: Handling failed start > for R_ilo_koch:0 on julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 WARN: unpack_rsc_op: Processing failed op > (R_ilo_julia:0_start_0) on julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 WARN: unpack_rsc_op: Handling failed start > for R_ilo_julia:0 on julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 info: determine_online_status: Node > koch.uvt.nl is online > pengine[4347]: 2007/12/20_10:45:13 info: group_print: Resource Group: G_dloprd > pengine[4347]: 2007/12/20_10:45:13 info: native_print: R_137.56.247.151 > (heartbeat::ocf:IPaddr): Started julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 info: native_print: R_san_var_bb > (heartbeat::ocf:Filesystem): Started julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 info: native_print: R_MailTo > (heartbeat::ocf:MailTo): Started julia.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 info: clone_print: Clone Set: CL_ilo_julia > pengine[4347]: 2007/12/20_10:45:13 info: native_print: R_ilo_julia:0 > (stonith:external/riloe): Started koch.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 info: clone_print: Clone Set: CL_ilo_koch > pengine[4347]: 2007/12/20_10:45:13 info: native_print: R_ilo_koch:0 > (stonith:external/riloe): Started koch.uvt.nl > pengine[4347]: 2007/12/20_10:45:13 notice: NoRoleChange: Move resource > R_137.56.247.151 (julia.uvt.nl -> koch.uvt.nl) > pengine[4347]: 2007/12/20_10:45:13 notice: StopRsc: julia.uvt.nl Stop > R_137.56.247.151 > pengine[4347]: 2007/12/20_10:45:13 notice: StartRsc: koch.uvt.nl Start > R_137.56.247.151 > pengine[4347]: 2007/12/20_10:45:13 notice: NoRoleChange: Move resource > R_san_var_bb (julia.uvt.nl -> koch.uvt.nl) > pengine[4347]: 2007/12/20_10:45:13 notice: StopRsc: julia.uvt.nl Stop > R_san_var_bb > pengine[4347]: 2007/12/20_10:45:13 notice: StartRsc: koch.uvt.nl Start > R_san_var_bb > pengine[4347]: 2007/12/20_10:45:13 notice: RecurringOp: koch.uvt.nl > R_san_var_bb_monitor_20000 > pengine[4347]: 2007/12/20_10:45:13 notice: NoRoleChange: Move resource > R_MailTo (julia.uvt.nl -> koch.uvt.nl) > pengine[4347]: 2007/12/20_10:45:13 notice: StopRsc: julia.uvt.nl Stop > R_MailTo > pengine[4347]: 2007/12/20_10:45:13 notice: StartRsc: koch.uvt.nl Start > R_MailTo > pengine[4347]: 2007/12/20_10:45:13 notice: RecurringOp: koch.uvt.nl > R_MailTo_monitor_10000 > pengine[4347]: 2007/12/20_10:45:13 notice: NoRoleChange: Leave resource > R_ilo_julia:0 (koch.uvt.nl) > pengine[4347]: 2007/12/20_10:45:13 notice: NoRoleChange: Leave resource > R_ilo_koch:0 (koch.uvt.nl) > crmd[4341]: 2007/12/20_10:45:13 info: do_state_transition: State transition > S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS > cause=C_IPC_MESSAGE origin=route_message ] > tengine[4346]: 2007/12/20_10:45:13 info: unpack_graph: Unpacked transition 7: > 12 actions in 12 synapses > tengine[4346]: 2007/12/20_10:45:13 info: te_pseudo_action: Pseudo action 18 > fired and confirmed > tengine[4346]: 2007/12/20_10:45:13 info: send_rsc_command: Initiating action > 13: R_MailTo_stop_0 on julia.uvt.nl > crmd[4341]: 2007/12/20_10:45:13 info: do_lrm_rsc_op: Performing > op=R_MailTo_stop_0 key=13:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > crmd[4341]: 2007/12/20_10:45:13 info: process_lrm_event: LRM operation > R_MailTo_monitor_10000 (call=16, rc=-2) Cancelled > pengine[4347]: 2007/12/20_10:45:13 info: process_pe_message: Transition 7: > PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-94.bz2 > crmd[4341]: 2007/12/20_10:45:13 info: process_lrm_event: LRM operation > R_MailTo_stop_0 (call=23, rc=0) complete > tengine[4346]: 2007/12/20_10:45:13 info: match_graph_event: Action > R_MailTo_stop_0 (13) confirmed on julia.uvt.nl > tengine[4346]: 2007/12/20_10:45:13 info: send_rsc_command: Initiating action > 10: R_san_var_bb_stop_0 on julia.uvt.nl > crmd[4341]: 2007/12/20_10:45:13 info: do_lrm_rsc_op: Performing > op=R_san_var_bb_stop_0 key=10:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > crmd[4341]: 2007/12/20_10:45:13 info: process_lrm_event: LRM operation > R_san_var_bb_monitor_20000 (call=14, rc=-2) Cancelled > Filesystem[12743][12774]: 2007/12/20_10:45:13 INFO: Running stop for > /dev/mapper/san_var_bb on /var/blackboard > Filesystem[12743][12784]: 2007/12/20_10:45:13 INFO: Trying to unmount > /var/blackboard > Filesystem[12743][12786]: 2007/12/20_10:45:13 INFO: unmounted /var/blackboard > successfully > lrmd[4338]: 2007/12/20_10:45:13 info: RA output: (R_san_var_bb:stop:stderr) > BLKFLSBUF: Input/output error > > crmd[4341]: 2007/12/20_10:45:13 info: process_lrm_event: LRM operation > R_san_var_bb_stop_0 (call=24, rc=0) complete > tengine[4346]: 2007/12/20_10:45:13 info: match_graph_event: Action > R_san_var_bb_stop_0 (10) confirmed on julia.uvt.nl > tengine[4346]: 2007/12/20_10:45:13 info: send_rsc_command: Initiating action > 8: R_137.56.247.151_stop_0 on julia.uvt.nl > crmd[4341]: 2007/12/20_10:45:13 info: do_lrm_rsc_op: Performing > op=R_137.56.247.151_stop_0 key=8:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > lrmd[4338]: 2007/12/20_10:45:13 info: RA output: > (R_137.56.247.151:stop:stdout) In IP Stop > > lrmd[4338]: 2007/12/20_10:45:13 info: RA output: > (R_137.56.247.151:stop:stderr) SIOCDELRT: No such process > > IPaddr[12792][12807]: 2007/12/20_10:45:13 INFO: ifconfig eth0:0 down > crmd[4341]: 2007/12/20_10:45:13 info: process_lrm_event: LRM operation > R_137.56.247.151_stop_0 (call=25, rc=0) complete > tengine[4346]: 2007/12/20_10:45:13 info: match_graph_event: Action > R_137.56.247.151_stop_0 (8) confirmed on julia.uvt.nl > tengine[4346]: 2007/12/20_10:45:13 info: te_pseudo_action: Pseudo action 19 > fired and confirmed > tengine[4346]: 2007/12/20_10:45:13 info: te_pseudo_action: Pseudo action 16 > fired and confirmed > tengine[4346]: 2007/12/20_10:45:13 info: send_rsc_command: Initiating action > 9: R_137.56.247.151_start_0 on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:14 info: match_graph_event: Action > R_137.56.247.151_start_0 (9) confirmed on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:14 info: send_rsc_command: Initiating action > 11: R_san_var_bb_start_0 on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:15 info: match_graph_event: Action > R_san_var_bb_start_0 (11) confirmed on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:15 info: send_rsc_command: Initiating action > 12: R_san_var_bb_monitor_20000 on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:15 info: send_rsc_command: Initiating action > 14: R_MailTo_start_0 on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:16 info: match_graph_event: Action > R_MailTo_start_0 (14) confirmed on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:16 info: te_pseudo_action: Pseudo action 17 > fired and confirmed > tengine[4346]: 2007/12/20_10:45:16 info: send_rsc_command: Initiating action > 15: R_MailTo_monitor_10000 on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:26 info: match_graph_event: Action > R_san_var_bb_monitor_20000 (12) confirmed on koch.uvt.nl > tengine[4346]: 2007/12/20_10:45:28 info: match_graph_event: Action > R_MailTo_monitor_10000 (15) confirmed on koch.uvt.nl > crmd[4341]: 2007/12/20_10:45:28 info: do_state_transition: State transition > S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE > origin=route_message ] > tengine[4346]: 2007/12/20_10:45:28 info: run_graph: Transition 7: > (Complete=12, Pending=0, Fired=0, Skipped=0, Incomplete=0) > tengine[4346]: 2007/12/20_10:45:28 info: notify_crmd: Transition 7 status: > te_complete - <null> > cib[4337]: 2007/12/20_10:49:26 info: cib_stats: Processed 23 operations > (4347.00us average, 0% utilization) in the last 10min > cib[4388]: 2007/12/20_10:27:32 info: cib_stats: Processed 44 operations > (1363.00us average, 0% utilization) in the last 10min > cib[4388]: 2007/12/20_10:43:14 info: cib_stats: Processed 1 operations > (0.00us average, 0% utilization) in the last 10min > attrd[4391]: 2007/12/20_10:45:13 info: attrd_ha_callback: flush message from > julia.uvt.nl > attrd[4391]: 2007/12/20_10:45:13 info: attrd_perform_update: Sent update 6: > pingd=100 > crmd[4392]: 2007/12/20_10:45:14 info: do_lrm_rsc_op: Performing > op=R_137.56.247.151_start_0 key=9:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > IPaddr[4890][4921]: 2007/12/20_10:45:14 INFO: Using calculated nic for > 137.56.247.151: eth0 > IPaddr[4890][4926]: 2007/12/20_10:45:14 INFO: Using calculated netmask for > 137.56.247.151: 255.255.255.0 > IPaddr[4890][4948]: 2007/12/20_10:45:14 INFO: eval ifconfig eth0:0 > 137.56.247.151 netmask 255.255.255.0 broadcast 137.56.247.255 > crmd[4392]: 2007/12/20_10:45:14 info: process_lrm_event: LRM operation > R_137.56.247.151_start_0 (call=11, rc=0) complete > crmd[4392]: 2007/12/20_10:45:14 info: build_operation_update: Digest for > 0:0;9:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9 (R_137.56.247.151_start_0) was > 22e197023bb2e04bb6425008f3b1d0b4 > > crmd[4392]: 2007/12/20_10:45:14 info: log_data_element: > build_operation_update: digest:source <parameters target_role="started" > ip="137.56.247.151"/> > crmd[4392]: 2007/12/20_10:45:15 info: do_lrm_rsc_op: Performing > op=R_san_var_bb_start_0 key=11:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > Filesystem[4978][5007]: 2007/12/20_10:45:15 INFO: Running start for > /dev/mapper/san_var_bb on /var/blackboard > crmd[4392]: 2007/12/20_10:45:15 info: process_lrm_event: LRM operation > R_san_var_bb_start_0 (call=12, rc=0) complete > crmd[4392]: 2007/12/20_10:45:15 info: build_operation_update: Digest for > 0:0;11:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9 (R_san_var_bb_start_0) was > b12952a3cc0f0cfa885d2eee492ee261 > > crmd[4392]: 2007/12/20_10:45:15 info: log_data_element: > build_operation_update: digest:source <parameters target_role="started" > fstype="ext3" directory="/var/blackboard" device="/dev/mapper/san_var_bb"/> > crmd[4392]: 2007/12/20_10:45:16 info: do_lrm_rsc_op: Performing > op=R_san_var_bb_monitor_20000 key=12:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > crmd[4392]: 2007/12/20_10:45:16 info: do_lrm_rsc_op: Performing > op=R_MailTo_start_0 key=14:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > crmd[4392]: 2007/12/20_10:45:16 info: process_lrm_event: LRM operation > R_MailTo_start_0 (call=14, rc=0) complete > crmd[4392]: 2007/12/20_10:45:16 info: build_operation_update: Digest for > 0:0;14:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9 (R_MailTo_start_0) was > 494425cdd1b02f1ad655760f6168534f > > crmd[4392]: 2007/12/20_10:45:16 info: log_data_element: > build_operation_update: digest:source <parameters target_role="started" > email="[EMAIL PROTECTED]" subject="failover occured!"/> > crmd[4392]: 2007/12/20_10:45:17 info: do_lrm_rsc_op: Performing > op=R_MailTo_monitor_10000 key=15:7:68904856-5ee2-4bbe-b49d-cd77fa79f7e9) > crmd[4392]: 2007/12/20_10:45:26 info: process_lrm_event: LRM operation > R_san_var_bb_monitor_20000 (call=13, rc=0) complete > crmd[4392]: 2007/12/20_10:45:27 info: process_lrm_event: LRM operation > R_MailTo_monitor_10000 (call=15, rc=0) complete > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems -- Dejan _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
