Hi. I'm trying to set up a two node cluster with stonith, but I'm
having trouble setting up fencing. I'm trying it out with meatware
stonith, running killall -9 corosync on the other node, but instead of
seeing an "OPERATIOR INTERVENTION REQUIRED" message I get "No match for
shutdown action".
I'm running Debian Squeeze with pacemaker 1.0.9.1+hg15626-1, corosync
1.2.1-4 and cluster-glue 1.0.6-1.
I'm hoping that someone can spot what I'm doing wrong. I've attached
the config and the relevant syslog output. I've set debug output on in
corosync.conf so it's a bit verbose.
<?xml version="1.0" ?>
<cib admin_epoch="0" crm_feature_set="3.0.1" dc-uuid="mgr-testcluster-2" epoch="186" have-quorum="1" num_updates="7" validate-with="pacemaker-1.0">
<configuration>
<crm_config>
<cluster_property_set id="cib-bootstrap-options">
<nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.0.9-74392a28b7f31d7ddc86689598bd23114f58978b"/>
<nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="openais"/>
<nvpair id="cib-bootstrap-options-expected-quorum-votes" name="expected-quorum-votes" value="2"/>
<nvpair id="cib-bootstrap-options-last-lrm-refresh" name="last-lrm-refresh" value="1315986996"/>
</cluster_property_set>
</crm_config>
<rsc_defaults/>
<op_defaults/>
<nodes>
<node id="mgr-testcluster-2" type="normal" uname="mgr-testcluster-2"/>
<node id="mgr-testcluster-1" type="normal" uname="mgr-testcluster-1"/>
</nodes>
<resources>
<clone id="Fencing">
<primitive class="stonith" id="mw-fencing" type="meatware">
<instance_attributes id="mw-fencing-instance_attributes">
<nvpair id="mw-fencing-instance_attributes-hostlist" name="hostlist" value="mgr-testcluster-1 mgr-testcluster-2"/>
</instance_attributes>
<operations>
<op id="mw-fencing-monitor-0" interval="0" name="monitor" timeout="60000"/>
</operations>
</primitive>
</clone>
</resources>
<constraints/>
</configuration>
</cib>
Sep 14 11:17:25 mgr-testcluster-2 corosync[8203]: [TOTEM ] The token was lost in the OPERATIONAL state.
Sep 14 11:17:25 mgr-testcluster-2 corosync[8203]: [TOTEM ] A processor failed, forming new configuration.
Sep 14 11:17:25 mgr-testcluster-2 corosync[8203]: [TOTEM ] Receive multicast socket recv buffer size (262142 bytes).
Sep 14 11:17:25 mgr-testcluster-2 corosync[8203]: [TOTEM ] Transmit multicast socket send buffer size (262142 bytes).
Sep 14 11:17:25 mgr-testcluster-2 corosync[8203]: [TOTEM ] entering GATHER state from 2.
Sep 14 11:17:29 mgr-testcluster-2 cib: [8215]: notice: ais_dispatch: Membership 380: quorum lost
Sep 14 11:17:29 mgr-testcluster-2 cib: [8215]: info: crm_update_peer: Node mgr-testcluster-1: id=230 state=lost (new) addr=r(0) ip(192.168.0.1) votes=1 born=376 seen=376 proc=00000000000000000000000000013312
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] entering GATHER state from 0.
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Creating commit token because I am the rep.
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Saving state aru 57 high seq received 57
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Storing new sequence id for ring 17c
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] entering COMMIT state.
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] got commit token
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] entering RECOVERY state.
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] TRANS [0] member 192.168.0.2:
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] position [0] member 192.168.0.2:
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] previous ring seq 376 rep 192.168.0.1
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: notice: ais_dispatch: Membership 380: quorum lost
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] aru 57 high delivered 57 received flag 1
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: ais_status_callback: status: mgr-testcluster-1 is now lost (was member)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Did not need to originate any messages in recovery.
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: crm_update_peer: Node mgr-testcluster-1: id=230 state=lost (new) addr=r(0) ip(192.168.0.1) votes=1 born=376 seen=376 proc=00000000000000000000000000013312
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] got commit token
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: post_cache_update: Updated cache after membership event 380.
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Sending initial ORF token
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: erase_node_from_join: Removed node mgr-testcluster-1 from join calculations: welcomed=0 itegrated=0 finalized=0 confirmed=1
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: ghash_update_cib_node: Updating mgr-testcluster-1: false (overwrite=false) hash_size=1
Sep 14 11:17:29 mgr-testcluster-2 cib: [8215]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/63, version=0.184.7): ok (rc=0)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: ghash_update_cib_node: Updating mgr-testcluster-2: true (overwrite=false) hash_size=1
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: post_cache_update: post_cache_update added action A_ELECTION_CHECK to the FSA
Sep 14 11:17:29 mgr-testcluster-2 cib: [8215]: debug: activateCibXml: Triggering CIB write for cib_modify op
Sep 14 11:17:29 mgr-testcluster-2 cib: [8215]: info: log_data_element: cib:diff: - <cib have-quorum="1" admin_epoch="0" epoch="184" num_updates="8" />
Sep 14 11:17:29 mgr-testcluster-2 cib: [8215]: info: log_data_element: cib:diff: + <cib have-quorum="0" admin_epoch="0" epoch="185" num_updates="1" />
Sep 14 11:17:29 mgr-testcluster-2 cib: [8215]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/65, version=0.185.1): ok (rc=0)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: crm_update_quorum: Updating quorum status to false (call=65)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] recovery to regular 1-0
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering to app 58 to 57
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 380: memb=1, new=0, lost=1
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [pcmk ] info: pcmk_peer_update: memb: mgr-testcluster-2 231
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [pcmk ] info: pcmk_peer_update: lost: mgr-testcluster-1 230
Sep 14 11:17:29 mgr-testcluster-2 cib: [8215]: debug: cib_process_xpath: Processing cib_query op for //cib/configuration/crm_config//nvpair[@name='expected-quorum-votes'] (/cib/configuration/crm_config/cluster_property_set/nvpair[3])
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 380: memb=1, new=0, lost=0
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [pcmk ] info: pcmk_peer_update: MEMB: mgr-testcluster-2 231
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: log_data_element: find_nvpair_attr: Match <nvpair id="cib-bootstrap-options-expected-quorum-votes" name="expected-quorum-votes" value="2" />
Sep 14 11:17:29 mgr-testcluster-2 cib: [8215]: debug: Forking temp process write_cib_contents
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [pcmk ] info: ais_mark_unseen_peer_dead: Node mgr-testcluster-1 was not seen in the previous transition
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: crm_ais_dispatch: Setting expected votes to 2
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [pcmk ] info: update_member: Node 230/mgr-testcluster-1 is now: lost
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_ELECTION_CHECK
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [pcmk ] debug: pcmk_peer_update: 1 nodes changed
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_election_check: Ignore election check: we not in an election
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [pcmk ] info: send_member_notification: Sending membership update 380 to 2 children
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: te_update_diff: Processing diff (cib_modify): 0.184.7 -> 0.184.8 (S_IDLE)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [pcmk ] debug: send_cluster_id: Local update: id=231, born=368, seq=380
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: get_xpath_object: No match for //cib_update_result//diff-added//crm_config in /notify/cib_update_result/diff
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] This node is within the primary component and will provide service.
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: WARN: match_down_event: No match for shutdown action on mgr-testcluster-1
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] entering OPERATIONAL state.
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: te_update_diff: Stonith/shutdown of mgr-testcluster-1 not matched
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: abort_transition_graph: te_update_diff:191 - Triggered transition abort (complete=1, tag=node_state, id=mgr-testcluster-1, magic=NA, cib=0.184.8) : Node failure
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: log_data_element: abort_transition_graph: Cause <node_state in_ccm="false" id="mgr-testcluster-1" />
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering 0 to 1
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: te_update_diff: Processing diff (cib_modify): 0.184.8 -> 0.185.1 (S_IDLE)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq 1 to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: get_xpath_object: No match for //cib_update_result//diff-added//crm_config in /notify/cib_update_result/diff
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [pcmk ] debug: pcmk_cluster_id_callback: Node update: mgr-testcluster-2 (1.0.9)
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] confchg entries 1
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: need_abort: Aborting on change to have-quorum
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Barrier Start Received From 231
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: s_crmd_fsa: Processing I_PE_CALC: [ state=S_IDLE cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Barrier completion status for nodeid 231 = 1.
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Synchronization barrier completed
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Synchronization actions starting for (dummy CLM service)
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_DC_TIMER_STOP
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_INTEGRATE_TIMER_STOP
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering 1 to 2
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_FINALIZE_TIMER_STOP
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq 2 to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_PE_INVOKE
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: do_pe_invoke: Query 68: Requesting the current CIB: S_POLICY_ENGINE
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including 1
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: s_crmd_fsa: Processing I_PE_CALC: [ state=S_POLICY_ENGINE cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering 2 to 3
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_PE_INVOKE
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq 3 to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: do_pe_invoke: Query 69: Requesting the current CIB: S_POLICY_ENGINE
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] confchg entries 1
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Barrier Start Received From 231
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Barrier completion status for nodeid 231 = 1.
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Synchronization barrier completed
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Committing synchronization for (dummy CLM service)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Synchronization actions starting for (dummy AMF service)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including 2
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including 3
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering 3 to 4
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq 4 to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] confchg entries 1
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Barrier Start Received From 231
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Barrier completion status for nodeid 231 = 1.
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Synchronization barrier completed
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Committing synchronization for (dummy AMF service)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Synchronization actions starting for (dummy CKPT service)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including 4
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering 4 to 5
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq 5 to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] confchg entries 1
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Barrier Start Received From 231
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Barrier completion status for nodeid 231 = 1.
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Synchronization barrier completed
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Committing synchronization for (dummy CKPT service)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Synchronization actions starting for (dummy EVT service)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering 5 to 6
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq 6 to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including 5
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering 6 to 7
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq 7 to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including 6
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including 7
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering 7 to 8
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq 8 to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] confchg entries 1
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Barrier Start Received From 231
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Barrier completion status for nodeid 231 = 1.
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Synchronization barrier completed
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Committing synchronization for (dummy EVT service)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Synchronization actions starting for (corosync cluster closed process group service v1.01)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering 8 to a
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq 9 to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq a to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [CPG ] downlist left_list: 1
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including 8
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering a to b
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq b to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] confchg entries 1
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Barrier Start Received From 231
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Barrier completion status for nodeid 231 = 1.
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Synchronization barrier completed
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [SYNC ] Committing synchronization for (corosync cluster closed process group service v1.01)
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including a
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering b to c
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq c to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including b
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including c
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering c to e
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq d to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq e to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including e
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering e to f
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq f to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including f
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] mcasted message added to pending queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering f to 10
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] Delivering MCAST message with seq 10 to pending delivery queue
Sep 14 11:17:29 mgr-testcluster-2 corosync[8203]: [TOTEM ] releasing messages up to and including 10
Sep 14 11:17:29 mgr-testcluster-2 cib: [8215]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/67, version=0.185.1): ok (rc=0)
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: do_pe_invoke_callback: Invoking the PE: query=69, ref=pe_calc-dc-1315988249-33, seq=380, quorate=0
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'false' for cluster option 'maintenance-mode'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'true' for cluster option 'start-failure-is-fatal'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'true' for cluster option 'stonith-enabled'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value '60s' for cluster option 'stonith-timeout'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value '30' for cluster option 'batch-limit'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'false' for cluster option 'stop-all-resources'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value 'none' for cluster option 'node-health-strategy'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value '0' for cluster option 'node-health-green'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value '0' for cluster option 'node-health-yellow'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: cluster_option: Using default value '-INFINITY' for cluster option 'node-health-red'
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: unpack_config: STONITH timeout: 60000
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: unpack_config: STONITH of failed nodes is enabled
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: unpack_config: Stop all active resources: false
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: unpack_config: Cluster is symmetric - resources can run anywhere by default
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: unpack_config: Default stickiness: 0
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: unpack_config: On loss of CCM Quorum: Stop ALL resources
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: info: determine_online_status: Node mgr-testcluster-2 is online
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: WARN: pe_fence_node: Node mgr-testcluster-1 will be fenced because it is un-expectedly down
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: info: determine_online_status_fencing: #011ha_state=active, ccm_state=false, crm_state=online, join_state=member, expected=member
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: WARN: determine_online_status: Node mgr-testcluster-1 is unclean
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: notice: clone_print: Clone Set: Fencing
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: native_active: Resource mw-fencing:0: node mgr-testcluster-1 is offline
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: native_active: Resource mw-fencing:1 active on mgr-testcluster-2
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: native_active: Resource mw-fencing:1 active on mgr-testcluster-2
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: notice: short_print: Started: [ mgr-testcluster-2 ]
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: notice: short_print: Stopped: [ mw-fencing:0 ]
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: common_apply_stickiness: Resource mw-fencing:0: preferring current location (node=mgr-testcluster-1, weight=1)
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: common_apply_stickiness: Resource mw-fencing:1: preferring current location (node=mgr-testcluster-2, weight=1)
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: native_assign_node: Assigning mgr-testcluster-2 to mw-fencing:1
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: native_assign_node: All nodes for resource mw-fencing:0 are unavailable, unclean or shutting down (mgr-testcluster-1: 0, -1000000)
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: native_assign_node: Could not allocate a node for mw-fencing:0
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: info: native_color: Resource mw-fencing:0 cannot run anywhere
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: debug: clone_color: Allocated 1 Fencing instances of a possible 2
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: WARN: custom_action: Action mw-fencing:0_stop_0 on mgr-testcluster-1 is unrunnable (offline)
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: WARN: custom_action: Marking node mgr-testcluster-1 unclean
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: WARN: stage6: Node mgr-testcluster-1 is unclean!
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore)
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: notice: LogActions: Stop resource mw-fencing:0#011(mgr-testcluster-1)
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: notice: LogActions: Leave resource mw-fencing:1#011(Started mgr-testcluster-2)
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: s_crmd_fsa: Processing I_PE_SUCCESS: [ state=S_POLICY_ENGINE cause=C_IPC_MESSAGE origin=handle_response ]
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_LOG
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_DC_TIMER_STOP
Sep 14 11:17:29 mgr-testcluster-2 cib: [8268]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-61.raw
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_INTEGRATE_TIMER_STOP
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_FINALIZE_TIMER_STOP
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_TE_INVOKE
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: unpack_graph: Unpacked transition 4: 3 actions in 3 synapses
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: do_te_invoke: Processing graph 4 (ref=pe_calc-dc-1315988249-33) derived from /var/lib/pengine/pe-warn-66.bz2
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: te_pseudo_action: Pseudo action 9 fired and confirmed
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: te_pseudo_action: Pseudo action 10 fired and confirmed
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: te_pseudo_action: Pseudo action 1 fired and confirmed
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: run_graph: Transition 4 (Complete=0, Pending=0, Fired=3, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-66.bz2): In-progress
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: run_graph: ====================================================
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: notice: run_graph: Transition 4 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-66.bz2): Complete
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: te_graph_trigger: Transition 4 is now complete
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: notify_crmd: Processing transition completion in state S_TRANSITION_ENGINE
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: notify_crmd: Transition 4 status: done - <null>
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: s_crmd_fsa: Processing I_TE_SUCCESS: [ state=S_TRANSITION_ENGINE cause=C_FSA_INTERNAL origin=notify_crmd ]
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_LOG
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: info: do_state_transition: Starting PEngine Recheck Timer
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: crm_timer_start: Started PEngine Recheck Timer (I_PE_CALC:900000ms), src=73
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_DC_TIMER_STOP
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_INTEGRATE_TIMER_STOP
Sep 14 11:17:29 mgr-testcluster-2 crmd: [8219]: debug: do_fsa_action: actions:trace: #011// A_FINALIZE_TIMER_STOP
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: WARN: process_pe_message: Transition 4: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-66.bz2
Sep 14 11:17:29 mgr-testcluster-2 pengine: [8218]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues.
Sep 14 11:17:29 mgr-testcluster-2 cib: [8268]: info: write_cib_contents: Wrote version 0.185.0 of the CIB to disk (digest: 402f479b97fc81bbd7a5a1669e1b469f)
Sep 14 11:17:29 mgr-testcluster-2 cib: [8268]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.9Qg2gD (digest: /var/lib/heartbeat/crm/cib.AGpuHp)
Sep 14 11:17:29 mgr-testcluster-2 cib: [8215]: info: Managed write_cib_contents process 8268 exited with return code 0.
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
