node node1
primitive FreeSWITCH ocf:fssolutions:FreeSWITCH \
params ips="bond2/212.163.22.155/26:bond2/212.163.22.156/26"
user="freeswitch" group="freeswitch" \
op monitor interval="3s" role="Master" depth="0" \
op monitor interval="10s" role="Slave" depth="0" \
op start interval="0" timeout="65" \
op stop interval="0" timeout="60"
ms FreeSWITCH-MS FreeSWITCH \
meta master-max="1" master-node-max="1" clone-max="1"
clone-node-max="1" notify="false" target-role="Master"
location FreeSWITCH-MS-on-node1 FreeSWITCH-MS 50: node1
property $id="cib-bootstrap-options" \
dc-version="1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14" \
cluster-infrastructure="corosync" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
============
Last updated: Sun Aug 26 16:13:05 2012
Last change: Sun Aug 26 16:11:18 2012 via crm_resource on node1
Stack: corosync
Current DC: node1 - partition WITHOUT quorum
Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14
1 Nodes configured, 2 expected votes
1 Resources configured.
============
Online: [ node1 ]
Master/Slave Set: FreeSWITCH-MS [FreeSWITCH]
Slaves: [ node1 ]
I run this:
crm resource promote FreeSWITCH-MS
Resulting log:
Aug 26 16:19:14 node1 cib[18166]: info: cib_process_request: Operation
complete: op cib_modify for section resources (origin=local/crm_resource/4,
version=0.7.2): ok (rc=0)
That's all it does. So I try:
crm resource demote FreeSWITCH-MS
And I get this:
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <cib
admin_epoch="0" epoch="7" num_updates="2" >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <configuration >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <resources >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <master
id="FreeSWITCH-MS" >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: -
<meta_attributes id="FreeSWITCH-MS-meta_attributes" >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <nvpair
value="Master" id="FreeSWITCH-MS-meta_attributes-target-role" />
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: -
</meta_attributes>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </master>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </resources>
Aug 26 16:20:24 node1 crmd[18171]: info: abort_transition_graph:
te_update_diff:126 - Triggered transition abort (complete=1, tag=diff,
id=(null), magic=NA, cib=0.8.1) : Non-status change
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </configuration>
Aug 26 16:20:24 node1 crmd[18171]: notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
origin=abort_transition_graph ]
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </cib>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <cib epoch="8"
num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2"
crm_feature_set="3.0.6" update-origin="node1" update-client="crm_resource"
cib-last-written="Sun Aug 26 16:11:18 2012" have-quorum="0" dc-uuid="node1"
>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <configuration >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <resources >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <master
id="FreeSWITCH-MS" >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: +
<meta_attributes id="FreeSWITCH-MS-meta_attributes" >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <nvpair
id="FreeSWITCH-MS-meta_attributes-target-role" name="target-role"
value="Slave" />
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: +
</meta_attributes>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </master>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </resources>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </configuration>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </cib>
Aug 26 16:20:24 node1 cib[18166]: info: cib_process_request: Operation
complete: op cib_modify for section resources (origin=local/crm_resource/4,
version=0.8.1): ok (rc=0)
Aug 26 16:20:24 node1 pengine[18170]: notice: unpack_config: On loss of
CCM Quorum: Ignore
Aug 26 16:20:24 node1 crmd[18171]: notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Aug 26 16:20:24 node1 crmd[18171]: info: do_te_invoke: Processing graph
5 (ref=pe_calc-dc-1345994424-17) derived from
/var/lib/pengine/pe-input-5.bz2
Aug 26 16:20:24 node1 crmd[18171]: notice: run_graph: ==== Transition 5
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-5.bz2): Complete
Aug 26 16:20:24 node1 crmd[18171]: notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Aug 26 16:20:24 node1 pengine[18170]: notice: process_pe_message:
Transition 5: PEngine Input stored in: /var/lib/pengine/pe-input-5.bz2
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <cib
admin_epoch="0" epoch="7" num_updates="2" >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <configuration >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <resources >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <master
id="FreeSWITCH-MS" >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: -
<meta_attributes id="FreeSWITCH-MS-meta_attributes" >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <nvpair
value="Master" id="FreeSWITCH-MS-meta_attributes-target-role" />
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: -
</meta_attributes>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </master>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </resources>
Aug 26 16:20:24 node1 crmd[18171]: info: abort_transition_graph:
te_update_diff:126 - Triggered transition abort (complete=1, tag=diff,
id=(null), magic=NA, cib=0.8.1) : Non-status change
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </configuration>
Aug 26 16:20:24 node1 crmd[18171]: notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
origin=abort_transition_graph ]
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </cib>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <cib epoch="8"
num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2"
crm_feature_set="3.0.6" update-origin="node1" update-client="crm_resource"
cib-last-written="Sun Aug 26 16:11:18 2012" have-quorum="0" dc-uuid="node1"
>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <configuration >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <resources >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <master
id="FreeSWITCH-MS" >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: +
<meta_attributes id="FreeSWITCH-MS-meta_attributes" >
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <nvpair
id="FreeSWITCH-MS-meta_attributes-target-role" name="target-role"
value="Slave" />
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: +
</meta_attributes>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </master>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </resources>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </configuration>
Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </cib>
Aug 26 16:20:24 node1 cib[18166]: info: cib_process_request: Operation
complete: op cib_modify for section resources (origin=local/crm_resource/4,
version=0.8.1): ok (rc=0)
Aug 26 16:20:24 node1 pengine[18170]: notice: unpack_config: On loss of
CCM Quorum: Ignore
Aug 26 16:20:24 node1 crmd[18171]: notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Aug 26 16:20:24 node1 crmd[18171]: info: do_te_invoke: Processing graph
5 (ref=pe_calc-dc-1345994424-17) derived from
/var/lib/pengine/pe-input-5.bz2
Aug 26 16:20:24 node1 crmd[18171]: notice: run_graph: ==== Transition 5
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-5.bz2): Complete
Aug 26 16:20:24 node1 crmd[18171]: notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Aug 26 16:20:24 node1 pengine[18170]: notice: process_pe_message:
Transition 5: PEngine Input stored in: /var/lib/pengine/pe-input-5.bz2
Then I try:
crm resource promote FreeSWITCH-MS
This time I get:
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - <cib
admin_epoch="0" epoch="8" num_updates="1" >
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - <configuration >
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - <resources >
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - <master
id="FreeSWITCH-MS" >
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: -
<meta_attributes id="FreeSWITCH-MS-meta_attributes" >
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - <nvpair
value="Slave" id="FreeSWITCH-MS-meta_attributes-target-role" />
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: -
</meta_attributes>
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - </master>
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - </resources>
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - </configuration>
Aug 26 16:21:39 node1 crmd[18171]: info: abort_transition_graph:
te_update_diff:126 - Triggered transition abort (complete=1, tag=diff,
id=(null), magic=NA, cib=0.9.1) : Non-status change
Aug 26 16:21:39 node1 crmd[18171]: notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
origin=abort_transition_graph ]
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - </cib>
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + <cib epoch="9"
num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2"
crm_feature_set="3.0.6" update-origin="node1" update-client="crm_resource"
cib-last-written="Sun Aug 26 16:20:24 2012" have-quorum="0" dc-uuid="node1"
>
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + <configuration >
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + <resources >
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + <master
id="FreeSWITCH-MS" >
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: +
<meta_attributes id="FreeSWITCH-MS-meta_attributes" >
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + <nvpair
id="FreeSWITCH-MS-meta_attributes-target-role" name="target-role"
value="Master" />
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: +
</meta_attributes>
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + </master>
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + </resources>
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + </configuration>
Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + </cib>
Aug 26 16:21:39 node1 cib[18166]: info: cib_process_request: Operation
complete: op cib_modify for section resources (origin=local/crm_resource/4,
version=0.9.1): ok (rc=0)
Aug 26 16:21:39 node1 pengine[18170]: notice: unpack_config: On loss of
CCM Quorum: Ignore
Aug 26 16:21:39 node1 crmd[18171]: notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Aug 26 16:21:39 node1 crmd[18171]: info: do_te_invoke: Processing graph
6 (ref=pe_calc-dc-1345994499-18) derived from
/var/lib/pengine/pe-input-6.bz2
Aug 26 16:21:39 node1 crmd[18171]: notice: run_graph: ==== Transition 6
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-6.bz2): Complete
Aug 26 16:21:39 node1 crmd[18171]: notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Aug 26 16:21:39 node1 pengine[18170]: notice: process_pe_message:
Transition 6: PEngine Input stored in: /var/lib/pengine/pe-input-6.bz2
However, crm status shows:
============
Last updated: Sun Aug 26 16:22:04 2012
Last change: Sun Aug 26 16:21:39 2012 via crm_resource on node1
Stack: corosync
Current DC: node1 - partition WITHOUT quorum
Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14
1 Nodes configured, 2 expected votes
1 Resources configured.
============
Online: [ node1 ]
Master/Slave Set: FreeSWITCH-MS [FreeSWITCH]
Slaves: [ node1 ]
So, I thought maybe my resource agent (which I wrote myself) might be
broken. So I ran ocf-tester on it again, but it passes all tests. Further,
I don't actually get any errors in the log file.
I have also tried doing a resource cleanup on FreeSWITCH-MS, restarting
pacemaker and corosync, putting the node in standby and bringing it back
out, upgrading pacemaker and corosync (to the version you seen in the
output in this E-mail from 1.1.6 previously where it first started
happening) and also completely wiping out everything in the
/var/lib/corosync, /var/lib/heartbeat, etc directories and recreating the
entire config from scratch (which, incidentally, DID clear up a warning I
was getting about a bad UUID), but nothing I have tried has resolved it.
The odd thing is, this single-node setup has been working for nearly a year
with this exact configuration. The box was rebooted last week, and now it
just seems to sit there playing dead.
Thanks in advance for any suggestions on how I might solve this!
i
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
