Brian,
I've followed several tutorials about setting up a simple three-node
cluster, with no resources (yet), under CentOS 7.
I've discovered the cluster won't restart upon rebooting a node.
The other two nodes, however, do claim the cluster is up, as shown
with 'pcs status cluster'.
I tracked down that on the rebooted node, corosync exited with a
'0' status. Nothing outright seems to be what I would call an error
message, but this was recorded:
[MAIN ] Corosync main process was not scheduled for 2145.7053
ms (threshold is 1320.0000 ms). Consider token timeout increase.
This seems related:
https://access.redhat.com/solutions/1217663
High Availability cluster node logs the message "Corosync main
process was not scheduled for X ms (threshold is Y ms). Consider
token timeout increase."
I've confirmed that corosync is running with the maximum realtime
scheduling priority:
[root@node1 ~]# ps -eo cmd,rtprio | grep -e [c]orosync -e RTPRIO
CMD RTPRIO
corosync 99
I am doing my testing in an admittedly underprovisioned VM environment.
I've used this same environment for CentOS 6 / heartbeat-based
solutions, and they were nowhere near as sensitive to these timing
issues.
Manually running 'pcs cluster start' does indeed fire everything
up without a hitch, and remains running for days at a crack.
The 'consider token timeout increase' message has me looking at this:
https://access.redhat.com/solutions/221263
Which makes this assertion:
RHEL 7 or 8
If no token value is specified in the corosync configuration, the
default is 1000 ms, or 1 second for a 2 node cluster, increasing
by 650ms for each additional member.
I have a three-node cluster, and the arithmetic for totem.token
seems to hold:
[root@node3 ~]# corosync-cmapctl | grep totem.token
runtime.config.totem.token (u32) = 1650
runtime.config.totem.token_retransmit (u32) = 392
runtime.config.totem.token_retransmits_before_loss_const (u32) = 4
I'm confused on a number of issues:
- The 'totem.token' value of 1650 doesn't seem to related to the
threshold number in the diagnostic message the corosync service
logged:
threshold is 1320.0000 ms
Can someone explain the relationship between these values?
Yes. Threshold is 80% of used token timeout.
- If I manually set 'totem.token' to a higher value, am I responsible
for tracking the number of nodes in the cluster, to keep in
alignment with what Red Hat's page says?
Nope. I've tried to explain what is really happening in the manpage
corosync.conf(5). totem.token and totem.token_coefficient are used in
the following formula:
runtime.config.token = totem.token + (number_of_nodes - 2) *
totem.token_coefficient
Corosync used runtime.config.token.
- Under these conditions, when corosync exits, why does it do so
with a zero status? It seems to me that if it exited at all,
That's a good question. How reproducible is the issue? Corosync
shouldn't "exit" with zero status.
without someone controllably stopping the service, it warrants a
non-zero status.
- Is there a recommended way to alter either pacemaker/corosync or
systemd configuration of these services to harden against resource
issues?
Enlarging timeout seems like a right way to go.
I don't know if corosync's startup can be deferred until the CPU
load settles, or if the some automatic retry can be set up...
This seems more like a init system question.
Regards,
Honza
Details of my environment; I'm happy to provide others, if anyone
has any specific questions:
[root@node1 ~]# cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)
[root@node1 ~]# rpm -qa | egrep 'pacemaker|corosync'
corosynclib-2.4.3-4.el7.x86_64
pacemaker-cluster-libs-1.1.19-8.el7_6.4.x86_64
corosync-2.4.3-4.el7.x86_64
pacemaker-cli-1.1.19-8.el7_6.4.x86_64
pacemaker-1.1.19-8.el7_6.4.x86_64
pacemaker-libs-1.1.19-8.el7_6.4.x86_64
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
