Thanks, here are the logs, there are infos about how it tried to start
resources on the nodes.
Keep in mind the node1 was already running the resources, and I simulated a
problem by turning down the ha interface.
Gabriele
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
----------------------------------------------------------------------------------
Da: Ulrich Windl <[email protected]>
A: [email protected]
Data: 16 dicembre 2020 15.45.36 CET
Oggetto: [ClusterLabs] Antw: [EXT] delaying start of a resource
>>> Gabriele Bulfon <[email protected]> schrieb am 16.12.2020 um 15:32 in
Nachricht <1523391015.734.1608129155836@www>:
> Hi, I have now a two node cluster using stonith with different
> pcmk_delay_base, so that node 1 has priority to stonith node 2 in case of
> problems.
>
> Though, there is still one problem: once node 2 delays its stonith action
> for 10 seconds, and node 1 just 1, node 2 does not delay start of resources,
> so it happens that while it's not yet powered off by node 1 (and waiting its
> dalay to power off node 1) it actually starts resources, causing a moment of
> few seconds where both NFS IP and ZFS pool (!!!!!) is mounted by both!
AFAIK pacemaker will not start resources on a node that is scheduled for
stonith. Even more: Pacemaker will tra to stop resources on a node scheduled
for stonith to start them elsewhere.
> How can I delay node 2 resource start until the delayed stonith action is
> done? Or how can I just delay the resource start so I can make it larger than
> its pcmk_delay_base?
We probably need to see logs and configs to understand.
>
> Also, I was suggested to set "stonith-enabled=true", but I don't know where
> to set this flag (cib-bootstrap-options is not happy with it...).
I think it's on by default, so you must have set it to false.
In crm shell it is "configure# property stonith-enabled=...".
Regards,
Ulrich
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
Dec 16 15:08:54 [642] xstorage2 corosync notice [TOTEM ] A processor failed,
forming new configuration.
Dec 16 15:08:56 [642] xstorage2 corosync notice [TOTEM ] A new membership
(10.100.100.2:408) was formed. Members left: 1
Dec 16 15:08:56 [642] xstorage2 corosync notice [TOTEM ] Failed to receive the
leave message. failed: 1
Dec 16 15:08:56 [666] attrd: info: pcmk_cpg_membership: Group
attrd event 2: xstha1 (node 1 pid 710) left via cluster exit
Dec 16 15:08:56 [663] cib: info: pcmk_cpg_membership: Group
cib event 2: xstha1 (node 1 pid 707) left via cluster exit
Dec 16 15:08:56 [662] pacemakerd: info: pcmk_cpg_membership: Group
pacemakerd event 2: xstha1 (node 1 pid 687) left via cluster exit
Dec 16 15:08:56 [642] xstorage2 corosync notice [QUORUM] Members[1]: 2
Dec 16 15:08:56 [662] pacemakerd: info: crm_update_peer_proc:
pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [666] attrd: info: crm_update_peer_proc:
pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [662] pacemakerd: info: pcmk_cpg_membership: Group
pacemakerd event 2: xstha2 (node 2 pid 662) is member
Dec 16 15:08:56 [642] xstorage2 corosync notice [MAIN ] Completed service
synchronization, ready to provide service.
Dec 16 15:08:56 [668] crmd: info: pcmk_cpg_membership: Group
crmd event 2: xstha1 (node 1 pid 712) left via cluster exit
Dec 16 15:08:56 [664] stonith-ng: info: pcmk_cpg_membership: Group
stonith-ng event 2: xstha1 (node 1 pid 708) left via cluster exit
Dec 16 15:08:56 [663] cib: info: crm_update_peer_proc:
pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [668] crmd: info: crm_update_peer_proc:
pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [666] attrd: notice: attrd_remove_voter: Lost attribute
writer xstha1
Dec 16 15:08:56 [664] stonith-ng: info: crm_update_peer_proc:
pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [662] pacemakerd: info: pcmk_quorum_notification: Quorum
retained | membership=408 members=1
Dec 16 15:08:56 [663] cib: notice: crm_update_peer_state_iter: Node
xstha1 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Dec 16 15:08:56 [664] stonith-ng: notice: crm_update_peer_state_iter: Node
xstha1 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Dec 16 15:08:56 [662] pacemakerd: notice: crm_update_peer_state_iter: Node
xstha1 state is now lost | nodeid=1 previous=member source=crm_reap_unseen_nodes
Dec 16 15:08:56 [668] crmd: info: peer_update_callback: Client
xstha1/peer now has status [offline] (DC=xstha1, changed=4000000)
Dec 16 15:08:56 [663] cib: info: crm_reap_dead_member:
Removing node with name xstha1 and id 1 from membership cache
Dec 16 15:08:56 [666] attrd: info: attrd_start_election_if_needed:
Starting an election to determine the writer
Dec 16 15:08:56 [663] cib: notice: reap_crm_member: Purged 1 peer
with id=1 and/or uname=xstha1 from the membership cache
Dec 16 15:08:56 [668] crmd: notice: peer_update_callback: Our
peer on the DC (xstha1) is dead
Dec 16 15:08:56 [663] cib: info: pcmk_cpg_membership: Group
cib event 2: xstha2 (node 2 pid 663) is member
Dec 16 15:08:56 [664] stonith-ng: info: crm_reap_dead_member:
Removing node with name xstha1 and id 1 from membership cache
Dec 16 15:08:56 [662] pacemakerd: info: mcp_cpg_deliver: Ignoring
process list sent by peer for local node
Dec 16 15:08:56 [666] attrd: notice: crm_update_peer_state_iter: Node
xstha1 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Dec 16 15:08:56 [664] stonith-ng: notice: reap_crm_member: Purged 1 peer
with id=1 and/or uname=xstha1 from the membership cache
Dec 16 15:08:56 [668] crmd: info: controld_delete_node_state:
Deleting transient attributes for node xstha1 (via CIB call 18) |
xpath=//node_state[@uname='xstha1']/transient_attributes
Dec 16 15:08:56 [664] stonith-ng: info: pcmk_cpg_membership: Group
stonith-ng event 2: xstha2 (node 2 pid 664) is member
Dec 16 15:08:56 [666] attrd: notice: attrd_peer_remove: Removing all
xstha1 attributes for peer loss
Dec 16 15:08:56 [668] crmd: info: pcmk_cpg_membership: Group
crmd event 2: xstha2 (node 2 pid 668) is member
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_delete operation for section
//node_state[@uname='xstha1']/transient_attributes to all (origin=local/crmd/18)
Dec 16 15:08:56 [666] attrd: info: crm_reap_dead_member:
Removing node with name xstha1 and id 1 from membership cache
Dec 16 15:08:56 [668] crmd: notice: do_state_transition: State
transition S_NOT_DC -> S_ELECTION | input=I_ELECTION
cause=C_CRMD_STATUS_CALLBACK origin=peer_update_callback
Dec 16 15:08:56 [666] attrd: notice: reap_crm_member: Purged 1 peer
with id=1 and/or uname=xstha1 from the membership cache
Dec 16 15:08:56 [668] crmd: info: update_dc: Unset DC. Was xstha1
Dec 16 15:08:56 [666] attrd: info: pcmk_cpg_membership: Group
attrd event 2: xstha2 (node 2 pid 666) is member
Dec 16 15:08:56 [666] attrd: info: election_check: election-attrd
won by local node
Dec 16 15:08:56 [668] crmd: info: pcmk_quorum_notification: Quorum
retained | membership=408 members=1
Dec 16 15:08:56 [666] attrd: notice: attrd_declare_winner:
Recorded local node as attribute writer (was unset)
Dec 16 15:08:56 [668] crmd: notice: crm_update_peer_state_iter: Node
xstha1 state is now lost | nodeid=1 previous=member source=crm_reap_unseen_nodes
Dec 16 15:08:56 [668] crmd: info: peer_update_callback: Cluster
node xstha1 is now lost (was member)
Dec 16 15:08:56 [666] attrd: info: write_attribute: Processed 1
private change for #attrd-protocol, id=n/a, set=n/a
Dec 16 15:08:56 [668] crmd: info: election_check: election-DC won
by local node
Dec 16 15:08:56 [668] crmd: info: do_log: Input I_ELECTION_DC
received in state S_ELECTION from election_win_cb
Dec 16 15:08:56 [668] crmd: notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=election_win_cb
Dec 16 15:08:56 [668] crmd: info: do_te_control: Registering TE
UUID: f340fcfc-17fa-ebf0-c5bf-8299546d41b6
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_delete operation for section
//node_state[@uname='xstha1']/transient_attributes: OK (rc=0,
origin=xstha2/crmd/18, version=0.46.19)
Dec 16 15:08:56 [668] crmd: info: set_graph_functions: Setting
custom graph functions
Dec 16 15:08:56 [668] crmd: info: do_dc_takeover: Taking over DC
status for this partition
Dec 16 15:08:56 [663] cib: info: cib_process_readwrite: We are
now in R/W mode
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_master operation for section 'all': OK (rc=0,
origin=local/crmd/19, version=0.46.19)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section cib to all (origin=local/crmd/20)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section cib: OK (rc=0,
origin=xstha2/crmd/20, version=0.46.19)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section crm_config to all
(origin=local/crmd/22)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section crm_config: OK (rc=0,
origin=xstha2/crmd/22, version=0.46.19)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section crm_config to all
(origin=local/crmd/24)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section crm_config: OK (rc=0,
origin=xstha2/crmd/24, version=0.46.19)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section crm_config to all
(origin=local/crmd/26)
Dec 16 15:08:56 [668] crmd: info: corosync_cluster_name: Cannot
get totem.cluster_name: CS_ERR_NOT_EXIST (12)
Dec 16 15:08:56 [668] crmd: info: join_make_offer: Making join-1
offers based on membership event 408
Dec 16 15:08:56 [668] crmd: info: join_make_offer: Sending join-1
offer to xstha2
Dec 16 15:08:56 [668] crmd: info: join_make_offer: Not making
join-1 offer to inactive node xstha1
Dec 16 15:08:56 [668] crmd: info: do_dc_join_offer_all: Waiting
on join-1 requests from 1 outstanding node
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section crm_config: OK (rc=0,
origin=xstha2/crmd/26, version=0.46.19)
Dec 16 15:08:56 [668] crmd: info: update_dc: Set DC to xstha2
(3.0.14)
Dec 16 15:08:56 [668] crmd: info: crm_update_peer_expected:
update_dc: Node xstha2[2] - expected state is now member (was (null))
Dec 16 15:08:56 [668] crmd: info: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state
Dec 16 15:08:56 [663] cib: info: cib_process_replace: Digest
matched on replace from xstha2: 4835352cb7b4920917d8beee219bc962
Dec 16 15:08:56 [663] cib: info: cib_process_replace:
Replaced 0.46.19 with 0.46.19 from xstha2
Dec 16 15:08:56 [668] crmd: info: controld_delete_node_state:
Deleting resource history for node xstha2 (via CIB call 31) |
xpath=//node_state[@uname='xstha2']/lrm
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_replace operation for section 'all': OK (rc=0,
origin=xstha2/crmd/29, version=0.46.19)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section nodes to all (origin=local/crmd/30)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_delete operation for section //node_state[@uname='xstha2']/lrm
to all (origin=local/crmd/31)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all (origin=local/crmd/32)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section nodes: OK (rc=0,
origin=xstha2/crmd/30, version=0.46.19)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: ---
0.46.19 2
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: +++
0.46.20 (null)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: --
/cib/status/node_state[@id='2']/lrm[@id='2']
Dec 16 15:08:56 [663] cib: info: cib_perform_op: + /cib:
@num_updates=20
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_delete operation for section //node_state[@uname='xstha2']/lrm:
OK (rc=0, origin=xstha2/crmd/31, version=0.46.20)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: ---
0.46.20 2
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: +++
0.46.21 (null)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: + /cib:
@num_updates=21
Dec 16 15:08:56 [663] cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']: <lrm id="2"/>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_resources>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_resource id="zpool_data" type="ZFS" class="ocf"
provider="heartbeat">
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_rsc_op id="zpool_data_last_0"
operation_key="zpool_data_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14"
transition-key="3:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76"
transition-magic="0:7;3:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76"
exit-reason="" on_node="xstha2" call-id="13" rc-code="7" op-status="0"
interval="0" last-run="1608127496" last-rc-change="1608127
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm_resource>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_resource id="xstha1-stonith" type="external/ipmi"
class="stonith">
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_rsc_op id="xstha1-stonith_last_0"
operation_key="xstha1-stonith_start_0" operation="start"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14"
transition-key="9:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76"
transition-magic="0:0;9:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76"
exit-reason="" on_node="xstha2" call-id="22" rc-code="0" op-status="0"
interval="0" last-run="1608127496" last-rc-change="160
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_rsc_op id="xstha1-stonith_monitor_25000"
operation_key="xstha1-stonith_monitor_25000" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14"
transition-key="10:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76"
transition-magic="0:0;10:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76"
exit-reason="" on_node="xstha2" call-id="24" rc-code="0" op-status="0"
interval="25000" last-rc-change="1608
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm_resource>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_resource id="xstha2-stonith" type="external/ipmi"
class="stonith">
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_rsc_op id="xstha2-stonith_last_0"
operation_key="xstha2-stonith_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14"
transition-key="5:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76"
transition-magic="0:7;5:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76"
exit-reason="" on_node="xstha2" call-id="21" rc-code="7" op-status="0"
interval="0" last-run="1608127496" last-rc-change=
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm_resource>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_resource id="xstha1_san0_IP" type="IPaddr"
class="ocf" provider="heartbeat">
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_rsc_op id="xstha1_san0_IP_last_0"
operation_key="xstha1_san0_IP_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14"
transition-key="1:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76"
transition-magic="0:7;1:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76"
exit-reason="" on_node="xstha2" call-id="5" rc-code="7" op-status="0"
interval="0" last-run="1608127496" last-rc-change="
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm_resource>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_resource id="xstha2_san0_IP" type="IPaddr"
class="ocf" provider="heartbeat">
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_rsc_op id="xstha2_san0_IP_last_0"
operation_key="xstha2_san0_IP_start_0" operation="start"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14"
transition-key="7:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76"
transition-magic="0:0;7:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76"
exit-reason="" on_node="xstha2" call-id="23" rc-code="0" op-status="0"
interval="0" last-run="1608127497" last-rc-change="160
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm_resource>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm_resources>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm>
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=xstha2/crmd/32, version=0.46.21)
Dec 16 15:08:56 [668] crmd: info: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state
Dec 16 15:08:56 [668] crmd: info: abort_transition_graph:
Transition aborted: Peer Cancelled | source=do_te_invoke:143 complete=true
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section nodes to all (origin=local/crmd/35)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all (origin=local/crmd/36)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section cib to all (origin=local/crmd/37)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section nodes: OK (rc=0,
origin=xstha2/crmd/35, version=0.46.21)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: ---
0.46.21 2
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: +++
0.46.22 (null)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: + /cib:
@num_updates=22
Dec 16 15:08:56 [663] cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']: @in_ccm=false, @crmd=offline,
@crm-debug-origin=do_state_transition, @join=down
Dec 16 15:08:56 [663] cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=xstha2/crmd/36, version=0.46.22)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: ---
0.46.22 2
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: +++
0.46.23 (null)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: + /cib:
@num_updates=23, @dc-uuid=2
Dec 16 15:08:56 [663] cib: info: cib_file_backup: Archived
previous version as /sonicle/var/cluster/lib/pacemaker/cib/cib-8.raw
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section cib: OK (rc=0,
origin=xstha2/crmd/37, version=0.46.23)
Dec 16 15:08:56 [663] cib: info: cib_file_write_with_digest: Wrote
version 0.46.0 of the CIB to disk (digest: 1ea3e3ee6c388f74623494869acf32d0)
Dec 16 15:08:56 [663] cib: info: cib_file_write_with_digest: Reading
cluster configuration file /sonicle/var/cluster/lib/pacemaker/cib/cib.4LaWbc
(digest: /sonicle/var/cluster/lib/pacemaker/cib/cib.5LaWbc)
Dec 16 15:08:56 [667] pengine: warning: unpack_config: Support for
stonith-action of 'poweroff' is deprecated and will be removed in a future
release (use 'off' instead)
Dec 16 15:08:56 [667] pengine: warning: pe_fence_node: Cluster node
xstha1 will be fenced: peer is no longer part of the cluster
Dec 16 15:08:56 [667] pengine: warning: determine_online_status: Node
xstha1 is unclean
Dec 16 15:08:56 [667] pengine: info: determine_online_status_fencing:
Node xstha2 is active
Dec 16 15:08:56 [667] pengine: info: determine_online_status: Node
xstha2 is online
Dec 16 15:08:56 [667] pengine: info: unpack_node_loop: Node 1 is
already processed
Dec 16 15:08:56 [667] pengine: info: unpack_node_loop: Node 2 is
already processed
Dec 16 15:08:56 [667] pengine: info: unpack_node_loop: Node 1 is
already processed
Dec 16 15:08:56 [667] pengine: info: unpack_node_loop: Node 2 is
already processed
Dec 16 15:08:56 [667] pengine: info: common_print: xstha1_san0_IP
(ocf::heartbeat:IPaddr): Started xstha1 (UNCLEAN)
Dec 16 15:08:56 [667] pengine: info: common_print: xstha2_san0_IP
(ocf::heartbeat:IPaddr): Started xstha2
Dec 16 15:08:56 [667] pengine: info: common_print: zpool_data
(ocf::heartbeat:ZFS): Started xstha1 (UNCLEAN)
Dec 16 15:08:56 [667] pengine: info: common_print: xstha1-stonith
(stonith:external/ipmi): Started xstha2
Dec 16 15:08:56 [667] pengine: info: common_print: xstha2-stonith
(stonith:external/ipmi): Started xstha1 (UNCLEAN)
Dec 16 15:08:56 [667] pengine: info: pcmk__native_allocate:
Resource xstha2-stonith cannot run anywhere
Dec 16 15:08:56 [667] pengine: warning: custom_action: Action
xstha1_san0_IP_stop_0 on xstha1 is unrunnable (offline)
Dec 16 15:08:56 [667] pengine: warning: custom_action: Action
zpool_data_stop_0 on xstha1 is unrunnable (offline)
Dec 16 15:08:56 [667] pengine: warning: custom_action: Action
xstha2-stonith_stop_0 on xstha1 is unrunnable (offline)
Dec 16 15:08:56 [667] pengine: warning: custom_action: Action
xstha2-stonith_stop_0 on xstha1 is unrunnable (offline)
Dec 16 15:08:56 [667] pengine: warning: stage6: Scheduling Node xstha1
for STONITH
Dec 16 15:08:56 [667] pengine: info: native_stop_constraints:
xstha1_san0_IP_stop_0 is implicit after xstha1 is fenced
Dec 16 15:08:56 [667] pengine: info: native_stop_constraints:
zpool_data_stop_0 is implicit after xstha1 is fenced
Dec 16 15:08:56 [667] pengine: info: native_stop_constraints:
xstha2-stonith_stop_0 is implicit after xstha1 is fenced
Dec 16 15:08:56 [667] pengine: notice: LogNodeActions: * Fence (off)
xstha1 'peer is no longer part of the cluster'
Dec 16 15:08:56 [667] pengine: notice: LogAction: * Move
xstha1_san0_IP ( xstha1 -> xstha2 )
Dec 16 15:08:56 [667] pengine: info: LogActions: Leave xstha2_san0_IP
(Started xstha2)
Dec 16 15:08:56 [667] pengine: notice: LogAction: * Move
zpool_data ( xstha1 -> xstha2 )
Dec 16 15:08:56 [667] pengine: info: LogActions: Leave xstha1-stonith
(Started xstha2)
Dec 16 15:08:56 [667] pengine: notice: LogAction: * Stop
xstha2-stonith ( xstha1 ) due to node availability
Dec 16 15:08:56 [667] pengine: warning: process_pe_message: Calculated
transition 0 (with warnings), saving inputs in
/sonicle/var/cluster/lib/pacemaker/pengine/pe-warn-6.bz2
Dec 16 15:08:56 [668] crmd: info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response
Dec 16 15:08:56 [668] crmd: info: do_te_invoke: Processing
graph 0 (ref=pe_calc-dc-1608127736-14) derived from
/sonicle/var/cluster/lib/pacemaker/pengine/pe-warn-6.bz2
Dec 16 15:08:56 [668] crmd: notice: te_fence_node: Requesting
fencing (off) of node xstha1 | action=1 timeout=60000
Dec 16 15:08:56 [664] stonith-ng: notice: handle_request: Client
crmd.668.c46cefe4 wants to fence (off) 'xstha1' with device '(any)'
Dec 16 15:08:56 [664] stonith-ng: notice: initiate_remote_stonith_op:
Requesting peer fencing (off) targeting xstha1 |
id=3cdbf44e-e860-c100-95e0-db72cc63ae16 state=0
Dec 16 15:08:56 [664] stonith-ng: info: dynamic_list_search_cb:
Refreshing port list for xstha1-stonith
Dec 16 15:08:56 [664] stonith-ng: info: process_remote_stonith_query:
Query result 1 of 1 from xstha2 for xstha1/off (1 devices)
3cdbf44e-e860-c100-95e0-db72cc63ae16
Dec 16 15:08:56 [664] stonith-ng: info: call_remote_stonith: Total
timeout set to 60 for peer's fencing targeting xstha1 for
crmd.668|id=3cdbf44e-e860-c100-95e0-db72cc63ae16
Dec 16 15:08:56 [664] stonith-ng: notice: call_remote_stonith:
Requesting that xstha2 perform 'off' action targeting xstha1 | for client
crmd.668 (72s, 0s)
Dec 16 15:08:56 [664] stonith-ng: notice: can_fence_host_with_device:
xstha1-stonith can fence (off) xstha1: dynamic-list
Dec 16 15:08:56 [664] stonith-ng: info: stonith_fence_get_devices_cb:
Found 1 matching devices for 'xstha1'
Dec 16 15:08:56 [664] stonith-ng: notice: schedule_stonith_command:
Delaying 'off' action targeting xstha1 on xstha1-stonith for 10s (timeout=60s,
requested_delay=0s, base=10s, max=10s)
Dec 16 15:08:54 [642] xstorage2 corosync notice [TOTEM ] A processor failed,
forming new configuration.
Dec 16 15:08:56 [642] xstorage2 corosync notice [TOTEM ] A new membership
(10.100.100.2:408) was formed. Members left: 1
Dec 16 15:08:56 [642] xstorage2 corosync notice [TOTEM ] Failed to receive the
leave message. failed: 1
Dec 16 15:08:56 [666] attrd: info: pcmk_cpg_membership: Group
attrd event 2: xstha1 (node 1 pid 710) left via cluster exit
Dec 16 15:08:56 [663] cib: info: pcmk_cpg_membership: Group
cib event 2: xstha1 (node 1 pid 707) left via cluster exit
Dec 16 15:08:56 [662] pacemakerd: info: pcmk_cpg_membership: Group
pacemakerd event 2: xstha1 (node 1 pid 687) left via cluster exit
Dec 16 15:08:56 [642] xstorage2 corosync notice [QUORUM] Members[1]: 2
Dec 16 15:08:56 [662] pacemakerd: info: crm_update_peer_proc:
pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [666] attrd: info: crm_update_peer_proc:
pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [662] pacemakerd: info: pcmk_cpg_membership: Group
pacemakerd event 2: xstha2 (node 2 pid 662) is member
Dec 16 15:08:56 [642] xstorage2 corosync notice [MAIN ] Completed service
synchronization, ready to provide service.
Dec 16 15:08:56 [668] crmd: info: pcmk_cpg_membership: Group
crmd event 2: xstha1 (node 1 pid 712) left via cluster exit
Dec 16 15:08:56 [664] stonith-ng: info: pcmk_cpg_membership: Group
stonith-ng event 2: xstha1 (node 1 pid 708) left via cluster exit
Dec 16 15:08:56 [663] cib: info: crm_update_peer_proc:
pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [668] crmd: info: crm_update_peer_proc:
pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [666] attrd: notice: attrd_remove_voter: Lost attribute
writer xstha1
Dec 16 15:08:56 [664] stonith-ng: info: crm_update_peer_proc:
pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [662] pacemakerd: info: pcmk_quorum_notification: Quorum
retained | membership=408 members=1
Dec 16 15:08:56 [663] cib: notice: crm_update_peer_state_iter: Node
xstha1 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Dec 16 15:08:56 [664] stonith-ng: notice: crm_update_peer_state_iter: Node
xstha1 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Dec 16 15:08:56 [662] pacemakerd: notice: crm_update_peer_state_iter: Node
xstha1 state is now lost | nodeid=1 previous=member source=crm_reap_unseen_nodes
Dec 16 15:08:56 [668] crmd: info: peer_update_callback: Client
xstha1/peer now has status [offline] (DC=xstha1, changed=4000000)
Dec 16 15:08:56 [663] cib: info: crm_reap_dead_member:
Removing node with name xstha1 and id 1 from membership cache
Dec 16 15:08:56 [666] attrd: info: attrd_start_election_if_needed:
Starting an election to determine the writer
Dec 16 15:08:56 [663] cib: notice: reap_crm_member: Purged 1 peer
with id=1 and/or uname=xstha1 from the membership cache
Dec 16 15:08:56 [668] crmd: notice: peer_update_callback: Our
peer on the DC (xstha1) is dead
Dec 16 15:08:56 [663] cib: info: pcmk_cpg_membership: Group
cib event 2: xstha2 (node 2 pid 663) is member
Dec 16 15:08:56 [664] stonith-ng: info: crm_reap_dead_member:
Removing node with name xstha1 and id 1 from membership cache
Dec 16 15:08:56 [662] pacemakerd: info: mcp_cpg_deliver: Ignoring
process list sent by peer for local node
Dec 16 15:08:56 [666] attrd: notice: crm_update_peer_state_iter: Node
xstha1 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Dec 16 15:08:56 [664] stonith-ng: notice: reap_crm_member: Purged 1 peer
with id=1 and/or uname=xstha1 from the membership cache
Dec 16 15:08:56 [668] crmd: info: controld_delete_node_state:
Deleting transient attributes for node xstha1 (via CIB call 18) |
xpath=//node_state[@uname='xstha1']/transient_attributes
Dec 16 15:08:56 [664] stonith-ng: info: pcmk_cpg_membership: Group
stonith-ng event 2: xstha2 (node 2 pid 664) is member
Dec 16 15:08:56 [666] attrd: notice: attrd_peer_remove: Removing all
xstha1 attributes for peer loss
Dec 16 15:08:56 [668] crmd: info: pcmk_cpg_membership: Group
crmd event 2: xstha2 (node 2 pid 668) is member
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_delete operation for section
//node_state[@uname='xstha1']/transient_attributes to all (origin=local/crmd/18)
Dec 16 15:08:56 [666] attrd: info: crm_reap_dead_member:
Removing node with name xstha1 and id 1 from membership cache
Dec 16 15:08:56 [668] crmd: notice: do_state_transition: State
transition S_NOT_DC -> S_ELECTION | input=I_ELECTION
cause=C_CRMD_STATUS_CALLBACK origin=peer_update_callback
Dec 16 15:08:56 [666] attrd: notice: reap_crm_member: Purged 1 peer
with id=1 and/or uname=xstha1 from the membership cache
Dec 16 15:08:56 [668] crmd: info: update_dc: Unset DC. Was xstha1
Dec 16 15:08:56 [666] attrd: info: pcmk_cpg_membership: Group
attrd event 2: xstha2 (node 2 pid 666) is member
Dec 16 15:08:56 [666] attrd: info: election_check: election-attrd
won by local node
Dec 16 15:08:56 [668] crmd: info: pcmk_quorum_notification: Quorum
retained | membership=408 members=1
Dec 16 15:08:56 [666] attrd: notice: attrd_declare_winner:
Recorded local node as attribute writer (was unset)
Dec 16 15:08:56 [668] crmd: notice: crm_update_peer_state_iter: Node
xstha1 state is now lost | nodeid=1 previous=member source=crm_reap_unseen_nodes
Dec 16 15:08:56 [668] crmd: info: peer_update_callback: Cluster
node xstha1 is now lost (was member)
Dec 16 15:08:56 [666] attrd: info: write_attribute: Processed 1
private change for #attrd-protocol, id=n/a, set=n/a
Dec 16 15:08:56 [668] crmd: info: election_check: election-DC won
by local node
Dec 16 15:08:56 [668] crmd: info: do_log: Input I_ELECTION_DC
received in state S_ELECTION from election_win_cb
Dec 16 15:08:56 [668] crmd: notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=election_win_cb
Dec 16 15:08:56 [668] crmd: info: do_te_control: Registering TE
UUID: f340fcfc-17fa-ebf0-c5bf-8299546d41b6
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_delete operation for section
//node_state[@uname='xstha1']/transient_attributes: OK (rc=0,
origin=xstha2/crmd/18, version=0.46.19)
Dec 16 15:08:56 [668] crmd: info: set_graph_functions: Setting
custom graph functions
Dec 16 15:08:56 [668] crmd: info: do_dc_takeover: Taking over DC
status for this partition
Dec 16 15:08:56 [663] cib: info: cib_process_readwrite: We are
now in R/W mode
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_master operation for section 'all': OK (rc=0,
origin=local/crmd/19, version=0.46.19)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section cib to all (origin=local/crmd/20)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section cib: OK (rc=0,
origin=xstha2/crmd/20, version=0.46.19)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section crm_config to all
(origin=local/crmd/22)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section crm_config: OK (rc=0,
origin=xstha2/crmd/22, version=0.46.19)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section crm_config to all
(origin=local/crmd/24)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section crm_config: OK (rc=0,
origin=xstha2/crmd/24, version=0.46.19)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section crm_config to all
(origin=local/crmd/26)
Dec 16 15:08:56 [668] crmd: info: corosync_cluster_name: Cannot
get totem.cluster_name: CS_ERR_NOT_EXIST (12)
Dec 16 15:08:56 [668] crmd: info: join_make_offer: Making join-1
offers based on membership event 408
Dec 16 15:08:56 [668] crmd: info: join_make_offer: Sending join-1
offer to xstha2
Dec 16 15:08:56 [668] crmd: info: join_make_offer: Not making
join-1 offer to inactive node xstha1
Dec 16 15:08:56 [668] crmd: info: do_dc_join_offer_all: Waiting
on join-1 requests from 1 outstanding node
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section crm_config: OK (rc=0,
origin=xstha2/crmd/26, version=0.46.19)
Dec 16 15:08:56 [668] crmd: info: update_dc: Set DC to xstha2
(3.0.14)
Dec 16 15:08:56 [668] crmd: info: crm_update_peer_expected:
update_dc: Node xstha2[2] - expected state is now member (was (null))
Dec 16 15:08:56 [668] crmd: info: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state
Dec 16 15:08:56 [663] cib: info: cib_process_replace: Digest
matched on replace from xstha2: 4835352cb7b4920917d8beee219bc962
Dec 16 15:08:56 [663] cib: info: cib_process_replace:
Replaced 0.46.19 with 0.46.19 from xstha2
Dec 16 15:08:56 [668] crmd: info: controld_delete_node_state:
Deleting resource history for node xstha2 (via CIB call 31) |
xpath=//node_state[@uname='xstha2']/lrm
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_replace operation for section 'all': OK (rc=0,
origin=xstha2/crmd/29, version=0.46.19)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section nodes to all (origin=local/crmd/30)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_delete operation for section //node_state[@uname='xstha2']/lrm
to all (origin=local/crmd/31)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all (origin=local/crmd/32)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section nodes: OK (rc=0,
origin=xstha2/crmd/30, version=0.46.19)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: ---
0.46.19 2
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: +++
0.46.20 (null)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: --
/cib/status/node_state[@id='2']/lrm[@id='2']
Dec 16 15:08:56 [663] cib: info: cib_perform_op: + /cib:
@num_updates=20
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_delete operation for section //node_state[@uname='xstha2']/lrm:
OK (rc=0, origin=xstha2/crmd/31, version=0.46.20)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: ---
0.46.20 2
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: +++
0.46.21 (null)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: + /cib:
@num_updates=21
Dec 16 15:08:56 [663] cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']: <lrm id="2"/>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_resources>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_resource id="zpool_data" type="ZFS" class="ocf"
provider="heartbeat">
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_rsc_op id="zpool_data_last_0"
operation_key="zpool_data_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14"
transition-key="3:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76"
transition-magic="0:7;3:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76"
exit-reason="" on_node="xstha2" call-id="13" rc-code="7" op-status="0"
interval="0" last-run="1608127496" last-rc-change="1608127
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm_resource>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_resource id="xstha1-stonith" type="external/ipmi"
class="stonith">
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_rsc_op id="xstha1-stonith_last_0"
operation_key="xstha1-stonith_start_0" operation="start"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14"
transition-key="9:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76"
transition-magic="0:0;9:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76"
exit-reason="" on_node="xstha2" call-id="22" rc-code="0" op-status="0"
interval="0" last-run="1608127496" last-rc-change="160
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_rsc_op id="xstha1-stonith_monitor_25000"
operation_key="xstha1-stonith_monitor_25000" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14"
transition-key="10:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76"
transition-magic="0:0;10:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76"
exit-reason="" on_node="xstha2" call-id="24" rc-code="0" op-status="0"
interval="25000" last-rc-change="1608
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm_resource>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_resource id="xstha2-stonith" type="external/ipmi"
class="stonith">
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_rsc_op id="xstha2-stonith_last_0"
operation_key="xstha2-stonith_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14"
transition-key="5:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76"
transition-magic="0:7;5:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76"
exit-reason="" on_node="xstha2" call-id="21" rc-code="7" op-status="0"
interval="0" last-run="1608127496" last-rc-change=
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm_resource>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_resource id="xstha1_san0_IP" type="IPaddr"
class="ocf" provider="heartbeat">
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_rsc_op id="xstha1_san0_IP_last_0"
operation_key="xstha1_san0_IP_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14"
transition-key="1:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76"
transition-magic="0:7;1:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76"
exit-reason="" on_node="xstha2" call-id="5" rc-code="7" op-status="0"
interval="0" last-run="1608127496" last-rc-change="
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm_resource>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_resource id="xstha2_san0_IP" type="IPaddr"
class="ocf" provider="heartbeat">
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
<lrm_rsc_op id="xstha2_san0_IP_last_0"
operation_key="xstha2_san0_IP_start_0" operation="start"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14"
transition-key="7:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76"
transition-magic="0:0;7:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76"
exit-reason="" on_node="xstha2" call-id="23" rc-code="0" op-status="0"
interval="0" last-run="1608127497" last-rc-change="160
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm_resource>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm_resources>
Dec 16 15:08:56 [663] cib: info: cib_perform_op: ++
</lrm>
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=xstha2/crmd/32, version=0.46.21)
Dec 16 15:08:56 [668] crmd: info: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state
Dec 16 15:08:56 [668] crmd: info: abort_transition_graph:
Transition aborted: Peer Cancelled | source=do_te_invoke:143 complete=true
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section nodes to all (origin=local/crmd/35)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all (origin=local/crmd/36)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Forwarding cib_modify operation for section cib to all (origin=local/crmd/37)
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section nodes: OK (rc=0,
origin=xstha2/crmd/35, version=0.46.21)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: ---
0.46.21 2
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: +++
0.46.22 (null)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: + /cib:
@num_updates=22
Dec 16 15:08:56 [663] cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']: @in_ccm=false, @crmd=offline,
@crm-debug-origin=do_state_transition, @join=down
Dec 16 15:08:56 [663] cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=xstha2/crmd/36, version=0.46.22)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: ---
0.46.22 2
Dec 16 15:08:56 [663] cib: info: cib_perform_op: Diff: +++
0.46.23 (null)
Dec 16 15:08:56 [663] cib: info: cib_perform_op: + /cib:
@num_updates=23, @dc-uuid=2
Dec 16 15:08:56 [663] cib: info: cib_file_backup: Archived
previous version as /sonicle/var/cluster/lib/pacemaker/cib/cib-8.raw
Dec 16 15:08:56 [663] cib: info: cib_process_request:
Completed cib_modify operation for section cib: OK (rc=0,
origin=xstha2/crmd/37, version=0.46.23)
Dec 16 15:08:56 [663] cib: info: cib_file_write_with_digest: Wrote
version 0.46.0 of the CIB to disk (digest: 1ea3e3ee6c388f74623494869acf32d0)
Dec 16 15:08:56 [663] cib: info: cib_file_write_with_digest: Reading
cluster configuration file /sonicle/var/cluster/lib/pacemaker/cib/cib.4LaWbc
(digest: /sonicle/var/cluster/lib/pacemaker/cib/cib.5LaWbc)
Dec 16 15:08:56 [667] pengine: warning: unpack_config: Support for
stonith-action of 'poweroff' is deprecated and will be removed in a future
release (use 'off' instead)
Dec 16 15:08:56 [667] pengine: warning: pe_fence_node: Cluster node
xstha1 will be fenced: peer is no longer part of the cluster
Dec 16 15:08:56 [667] pengine: warning: determine_online_status: Node
xstha1 is unclean
Dec 16 15:08:56 [667] pengine: info: determine_online_status_fencing:
Node xstha2 is active
Dec 16 15:08:56 [667] pengine: info: determine_online_status: Node
xstha2 is online
Dec 16 15:08:56 [667] pengine: info: unpack_node_loop: Node 1 is
already processed
Dec 16 15:08:56 [667] pengine: info: unpack_node_loop: Node 2 is
already processed
Dec 16 15:08:56 [667] pengine: info: unpack_node_loop: Node 1 is
already processed
Dec 16 15:08:56 [667] pengine: info: unpack_node_loop: Node 2 is
already processed
Dec 16 15:08:56 [667] pengine: info: common_print: xstha1_san0_IP
(ocf::heartbeat:IPaddr): Started xstha1 (UNCLEAN)
Dec 16 15:08:56 [667] pengine: info: common_print: xstha2_san0_IP
(ocf::heartbeat:IPaddr): Started xstha2
Dec 16 15:08:56 [667] pengine: info: common_print: zpool_data
(ocf::heartbeat:ZFS): Started xstha1 (UNCLEAN)
Dec 16 15:08:56 [667] pengine: info: common_print: xstha1-stonith
(stonith:external/ipmi): Started xstha2
Dec 16 15:08:56 [667] pengine: info: common_print: xstha2-stonith
(stonith:external/ipmi): Started xstha1 (UNCLEAN)
Dec 16 15:08:56 [667] pengine: info: pcmk__native_allocate:
Resource xstha2-stonith cannot run anywhere
Dec 16 15:08:56 [667] pengine: warning: custom_action: Action
xstha1_san0_IP_stop_0 on xstha1 is unrunnable (offline)
Dec 16 15:08:56 [667] pengine: warning: custom_action: Action
zpool_data_stop_0 on xstha1 is unrunnable (offline)
Dec 16 15:08:56 [667] pengine: warning: custom_action: Action
xstha2-stonith_stop_0 on xstha1 is unrunnable (offline)
Dec 16 15:08:56 [667] pengine: warning: custom_action: Action
xstha2-stonith_stop_0 on xstha1 is unrunnable (offline)
Dec 16 15:08:56 [667] pengine: warning: stage6: Scheduling Node xstha1
for STONITH
Dec 16 15:08:56 [667] pengine: info: native_stop_constraints:
xstha1_san0_IP_stop_0 is implicit after xstha1 is fenced
Dec 16 15:08:56 [667] pengine: info: native_stop_constraints:
zpool_data_stop_0 is implicit after xstha1 is fenced
Dec 16 15:08:56 [667] pengine: info: native_stop_constraints:
xstha2-stonith_stop_0 is implicit after xstha1 is fenced
Dec 16 15:08:56 [667] pengine: notice: LogNodeActions: * Fence (off)
xstha1 'peer is no longer part of the cluster'
Dec 16 15:08:56 [667] pengine: notice: LogAction: * Move
xstha1_san0_IP ( xstha1 -> xstha2 )
Dec 16 15:08:56 [667] pengine: info: LogActions: Leave xstha2_san0_IP
(Started xstha2)
Dec 16 15:08:56 [667] pengine: notice: LogAction: * Move
zpool_data ( xstha1 -> xstha2 )
Dec 16 15:08:56 [667] pengine: info: LogActions: Leave xstha1-stonith
(Started xstha2)
Dec 16 15:08:56 [667] pengine: notice: LogAction: * Stop
xstha2-stonith ( xstha1 ) due to node availability
Dec 16 15:08:56 [667] pengine: warning: process_pe_message: Calculated
transition 0 (with warnings), saving inputs in
/sonicle/var/cluster/lib/pacemaker/pengine/pe-warn-6.bz2
Dec 16 15:08:56 [668] crmd: info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response
Dec 16 15:08:56 [668] crmd: info: do_te_invoke: Processing
graph 0 (ref=pe_calc-dc-1608127736-14) derived from
/sonicle/var/cluster/lib/pacemaker/pengine/pe-warn-6.bz2
Dec 16 15:08:56 [668] crmd: notice: te_fence_node: Requesting
fencing (off) of node xstha1 | action=1 timeout=60000
Dec 16 15:08:56 [664] stonith-ng: notice: handle_request: Client
crmd.668.c46cefe4 wants to fence (off) 'xstha1' with device '(any)'
Dec 16 15:08:56 [664] stonith-ng: notice: initiate_remote_stonith_op:
Requesting peer fencing (off) targeting xstha1 |
id=3cdbf44e-e860-c100-95e0-db72cc63ae16 state=0
Dec 16 15:08:56 [664] stonith-ng: info: dynamic_list_search_cb:
Refreshing port list for xstha1-stonith
Dec 16 15:08:56 [664] stonith-ng: info: process_remote_stonith_query:
Query result 1 of 1 from xstha2 for xstha1/off (1 devices)
3cdbf44e-e860-c100-95e0-db72cc63ae16
Dec 16 15:08:56 [664] stonith-ng: info: call_remote_stonith: Total
timeout set to 60 for peer's fencing targeting xstha1 for
crmd.668|id=3cdbf44e-e860-c100-95e0-db72cc63ae16
Dec 16 15:08:56 [664] stonith-ng: notice: call_remote_stonith:
Requesting that xstha2 perform 'off' action targeting xstha1 | for client
crmd.668 (72s, 0s)
Dec 16 15:08:56 [664] stonith-ng: notice: can_fence_host_with_device:
xstha1-stonith can fence (off) xstha1: dynamic-list
Dec 16 15:08:56 [664] stonith-ng: info: stonith_fence_get_devices_cb:
Found 1 matching devices for 'xstha1'
Dec 16 15:08:56 [664] stonith-ng: notice: schedule_stonith_command:
Delaying 'off' action targeting xstha1 on xstha1-stonith for 10s (timeout=60s,
requested_delay=0s, base=10s, max=10s)
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
