Hi Team,
We found one observation where a single OCF resource that keeps failing to
start eventually crashes pacemaker-controld (SIGSEGV) and then brings the
entire cluster down(pacemaker service went down) with a fatal error
(CRM_EX_FATAL / exit 100) on Pacemaker 3.0.3. The exact same condition on
Pacemaker 2.1.6 does NOT crash controld - the resource is simply marked FAILED
and the cluster stays up. This looks like a regression in the 3.0.x controller
/ IPC event-handling path.
To compare behavior, we intentionally kept one resource (an OCF agent whose
start consistently fails) in a permanent start-failure state.
Environment:
Affected (crashes):
Pacemaker 3.0.3-1.el8 (pacemakerd --version: "Pacemaker 3.0.3-1.el8")
pcs 0.12.3.1-1.el8, corosync 3.1.10-1.el8
RHEL 8.10 (Ootpa), kernel 4.18.0-553.174.1.el8_10.x86_64
Single-node cluster (node is DC), stonith configured.
Not affected (survives identical trigger):
Pacemaker 2.1.6-1.el8, pcs 0.10.16-1.el8, corosync 3.1.7-1.el8
Same OS, same resource, same resource configuration.
Symptom / impact:
* A single resource that repeatedly fails 'start' drives Pacemaker into a
continuous stop -> start -> stop recovery loop (roughly every 30s in our setup).
* After ~2 days / ~2300 failed start attempts, pacemaker-controld crashed
with signal 11 (SIGSEGV), and because the controld respawn budget was
exhausted, pacemakerd declared a fatal failure and shut the whole stack down,
and it stays down.
* Result: total loss of the cluster manager on the node; all managed
processes left running with no cluster manager controlling them; manual
pacemaker restart was required.
Timeline (affected node):
* 2026-10-28 14:00 CET (approx) Target OCF resource intentionally driven
into a permanent start-failure state; recovery loop begins (every 'start'
fails, status 1, ~20s timeout)
* 2026-10-30 14:53:48 CET pacemaker-controld fatal failure -> CRM_EX_FATAL
cluster shutdown. ExecMainStatus=100, ActiveState=inactive/dead.
[root@testnode-1 testuser]# systemctl status pacemaker
● pacemaker.service - Pacemaker High Availability Cluster Manager
Loaded: loaded (/usr/lib/systemd/system/pacemaker.service; enabled; vendor
preset: disabled)
Active: inactive (dead) since Fri 2026-10-30 14:53:48 CET; 4 days ago
Docs: man:pacemakerd
https://clusterlabs.org/pacemaker/doc/
Process: 62993 ExecStart=/usr/sbin/pacemakerd (code=exited, status=100)
Main PID: 62993 (code=exited, status=100)
Tasks: 121
Memory: 9.4G
CGroup: /system.slice/pacemaker.service
├─79496 /opt/java//jre-17-openjdk/bin/java -server -XX:+UseG1GC
-XX:G1ReservePercent=30 -XX:+AlwaysPreTouch -XX:+UseTransparentHugePages
-XX:ConcGCThreads=2 -XX:ParallelGCT…
├─79894 /opt/java//jre-17-openjdk/bin/java
-Djava.util.logging.config.file=/opt/webstart//conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLog…
└─96139 /usr/sbin/crm_mon -d -i 15 -E
/opt/occ/CXP_902_0588_R20CT801/tools/PCSESA.sh --output-as=html
--output-to=/tmp/ClusterMon_SNMP_testuser-1.html
Oct 30 14:53:48 testnode-1 systemd[1]: pacemaker.service: Succeeded.
Logs:
The crash is consistently preceded by pacemaker-execd failing to deliver a
resource action event to controld over IPC. Full shutdown sequence (2026-10-30
14:53:48, from pacemaker.log):
Oct 30 14:53:48 testnode-1 pacemakerd [62993]
(pcmk__ipc_is_authentic_process_active) info: Could not connect to attrd IPC:
Connection reset by peer
Oct 30 14:53:48 testnode-1 pacemakerd [62993] (check_next_subdaemon)
notice: Subdaemon pacemaker-attrd[62998] is unresponsive to IPC after 1
attempt (will recheck later)
Oct 30 14:53:48 testnode-1 crm_mon [96138] (crm_ipc_read)
error: Connection to cib_rw IPC failed
Oct 30 14:53:48 testnode-1 crm_mon [96138]
(mainloop_gio_callback) error: Connection to cib_rw closed |
client=0x55b2ed79dd80 condition=17
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997]
(cancel_recurring_action) info: Cancelling ocf operation
ESAPMA_OCC_monitor_10000
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (qb_ipcs_event_sendv)
warning: new_event_notification (/dev/shm/qb-62997-63000-6-7AAcJe/qb): Bad
file descriptor (9)
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (pcmk__ipc_send_xml)
notice: IPC message to pid 63000 failed: Bad file descriptor | rc=9
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (send_client_notify)
warning: Could not notify client crmd: Bad file descriptor | rc=9
Oct 30 14:53:48 testnode-1 pacemakerd [62993] (pcmk_child_exit)
warning: Shutting cluster down because pacemaker-controld[63000] had fatal
failure
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997]
(cancel_recurring_action) info: Cancelling ocf operation
ESASMA_testnode-1_monitor_10000
Oct 30 14:53:48 testnode-1 pacemakerd [62993] (pcmk_shutdown_worker)
notice: Shutting down Pacemaker
Oct 30 14:53:48 testnode-1 pacemakerd [62993] (stop_child)
notice: Stopping subdaemon pacemaker-schedulerd | via signal 15 to process 62999
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997]
(cancel_recurring_action) info: Cancelling ocf operation
MANAGER_OCC_monitor_10000
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997]
(cancel_recurring_action) info: Cancelling ocf operation
ESASMS_testnode-1_monitor_10000
Oct 30 14:53:48 testnode-1 pacemakerd [62993] (pcmk_child_exit)
error: pacemaker-attrd[62998] exited with status 102 (Not connected)
The IPC failure pattern that immediately precedes the crash is: execd tries to
notify controld of a resource action result (the repeated start/stop churn),
controld's IPC endpoint is already gone (Bad file descriptor), and controld
then crashes with SIGSEGV. Since the respawn budget was exhausted, pacemakerd
treated this as fatal and shut the whole stack down (attrd exiting with status
102 "Not connected" is a side effect of controld's IPC teardown, not an
independent fault).
Expected Behavior:
A resource that repeatedly fails to start should be handled via
migration-threshold / on-fail (marked FAILED / stopped) without crashing
pacemaker-controld or forcing a whole-cluster fatal shutdown - i.e. the
Pacemaker 2.1.6 behavior we observe on the comparison node.
Resource configuration (same on both nodes)
--------------------------------------------------------------------------------
migration-threshold=5 failure-timeout=120s priority=70
start: timeout=120s on-fail=restart
monitor: interval=10s timeout=120s on-fail=restart
stop: timeout=120s on-fail=block
Question:
1. Is this a known issue / already fixed after 3.0.3 in the controller or
IPC (libcrmcommon mainloop_gio / event-notification) code?
2. Is there a recommended configuration workaround to avoid the fatal path
while a fix is pending (e.g. tuning around resource failure handling)?
Attachments we provided for working and non-working node for pacemaker logs.
Pacemaker:3.0.3 --> testnode-1_pacemaker_not-working.log
Pacemaker:2.1.6 --> testnode-2_pacemaker-working.log and
testnode-2_pacemaker_rotated-working.log
Thanks and Regards,
S Sathish S
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
