Thanks for the quick response and for offering to dig into this. Answers to
your points below.
The OCF agent and its configuration:
The failing resource is a lightly modified copy of the standard upstream
"Dummy" OCF agent (resource-agents Dummy). For reproduction, the key point is
that the start action is forced to fail. Below is how start/stop and the action
dispatcher look in our agent.
start - deliberately returns a generic error after a fixed sleep (the real
systemctl-based start logic is present but sits after the exit, so it is not
reached while we are reproducing the failure):
start() {
ocf_log info "Initiating ESAFMA component start"
sleep 20
ocf_log warn "Unable to start ESAFMA component"
exit $OCF_ERR_GENERIC # start always returns 1
# --- normal start logic (not reached in this failure scenario) ---
systemctl start esafma.service
sleep 10
output=$(systemctl is-active esafma.service)
if [ "$output" = "active" ]; then
PID_COUNT=$(systemctl show --property MainPID --value
esafma.service)
echo "$PID_COUNT" > /opt/occ/var/pid/esafma.pid
ocf_log info "ESAFMA component started successfully"
return $OCF_SUCCESS
else
ocf_log warn "Unable to start ESAFMA component"
exit $OCF_ERR_GENERIC
fi
}
stop - normal systemctl-based stop:
stop() {
ocf_log info "Initiating ESAFMA component stop"
systemctl stop esafma.service
sleep 10
output=$(systemctl is-active esafma.service)
if [ "$output" != "active" ]; then
rm -f /opt/occ/var/pid/esafma.pid
ocf_log info "ESAFMA component stopped successfully"
return $OCF_SUCCESS
else
ocf_log warn "Unable to stop ESAFMA component"
exit $OCF_ERR_GENERIC
fi
}
monitor - normal pidfile/systemctl-based check (reports running / not-running /
error). It is not really relevant to this reproduction because start never
succeeds, so the resource is never up:
monitor() {
PIDFILE="/opt/occ/var/pid/esafma.pid"
if [ -n "$PIDFILE" -a -f "$PIDFILE" ]; then
PID=$(cat "$PIDFILE")
if [ -n "$PID" ]; then
timeout 90s ps -f --pid $PID
rc=$?
case $rc in
0) ocf_log info "ESAFMA component is running with $PID
number"
return $OCF_SUCCESS ;;
1) ocf_log err "ESAFMA component is not running"
return $OCF_NOT_RUNNING ;;
*) ocf_log warn "System command failed with $rc return code"
return $OCF_ERR_GENERIC ;;
esac
fi
fi
}
action dispatcher:
case $__OCF_ACTION in
meta-data) meta_data; exit $OCF_SUCCESS ;;
start) start ;;
stop) stop ;;
monitor) monitor ;;
validate-all) validate ;;
*) usage; exit $OCF_ERR_UNIMPLEMENTED ;;
esac
The metadata is the stock metadata (start/stop/monitor timeout=20, monitor
interval=10).
To reproduce on a stock system you only need to take the upstream agent, make
its start action "sleep 20; exit $OCF_ERR_GENERIC" (OCF_ERR_GENERIC=1), install
it as an OCF agent (e.g /usr/lib/ocf/resource.d/provider/ESAFMA_RA, class=ocf,
provider=provider, type=ESAFMA_RA), and configure the primitive with the
operation timeouts/thresholds already shared in the original report. Pacemaker
then loops stop -> start -> stop roughly every 30s.
on-fail / failure-handling configuration:
On the unaffected node (Pacemaker 2.1.6) we run exactly the same configuration
- on-fail=restart, same timeouts and thresholds - and it works correctly: the
resource is marked FAILED and the cluster stays up. Since only the Pacemaker
version differs between the two nodes. If there is no fix or alternative
available, we will try the on-fail=stop workaround.
Other 3.0.x releases:
We haven't tried other 3.0.x releases yet. We'll try to reproduce on 3.0.1 and
let you know.
For reference:
- affected node: Pacemaker 3.0.3-1.el8, corosync 3.1.10, pcs 0.12.3.1
- unaffected node: Pacemaker 2.1.6-1.el8, corosync 3.1.7, pcs 0.10.16
Thanks and Regards,
S Sathish S
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
