I guess a core dump with analyzed backtrace would be helpful. Kind regards, Ulrich Windl
From: Users <[email protected]> On Behalf Of S Sathish S via Users Sent: Wednesday, November 4, 2026 5:40 AM To: Cluster Labs - All topics related to open-source clustering welcomed <[email protected]> Cc: S Sathish S <[email protected]>; Devakumar K <[email protected]> Subject: [EXT] [EXT] [ClusterLabs] Pacemaker 3.0.3 controld SIGSEGV on repeated resource start-failure → cluster shutdown (not seen on 2.1.6) Hi Team, We found one observation where a single OCF resource that keeps failing to start eventually crashes pacemaker-controld (SIGSEGV) and then brings the entire cluster down(pacemaker service went down) with a fatal error (CRM_EX_FATAL / exit 100) on Pacemaker 3.0.3. The exact same condition on Pacemaker 2.1.6 does NOT crash controld — the resource is simply marked FAILED and the cluster stays up. This looks like a regression in the 3.0.x controller / IPC event-handling path. To compare behavior, we intentionally kept one resource (an OCF agent whose start consistently fails) in a permanent start-failure state. Environment: Affected (crashes): Pacemaker 3.0.3-1.el8 (pacemakerd --version: "Pacemaker 3.0.3-1.el8") pcs 0.12.3.1-1.el8, corosync 3.1.10-1.el8 RHEL 8.10 (Ootpa), kernel 4.18.0-553.174.1.el8_10.x86_64 Single-node cluster (node is DC), stonith configured. Not affected (survives identical trigger): Pacemaker 2.1.6-1.el8, pcs 0.10.16-1.el8, corosync 3.1.7-1.el8 Same OS, same resource, same resource configuration. Symptom / impact: * A single resource that repeatedly fails 'start' drives Pacemaker into a continuous stop -> start -> stop recovery loop (roughly every 30s in our setup). * After ~2 days / ~2300 failed start attempts, pacemaker-controld crashed with signal 11 (SIGSEGV), and because the controld respawn budget was exhausted, pacemakerd declared a fatal failure and shut the whole stack down, and it stays down. * Result: total loss of the cluster manager on the node; all managed processes left running with no cluster manager controlling them; manual pacemaker restart was required. Timeline (affected node): * 2026-10-28 14:00 CET (approx) Target OCF resource intentionally driven into a permanent start-failure state; recovery loop begins (every 'start' fails, status 1, ~20s timeout) * 2026-10-30 14:53:48 CET pacemaker-controld fatal failure -> CRM_EX_FATAL cluster shutdown. ExecMainStatus=100, ActiveState=inactive/dead. [root@testnode-1 testuser]# systemctl status pacemaker ● pacemaker.service - Pacemaker High Availability Cluster Manager Loaded: loaded (/usr/lib/systemd/system/pacemaker.service; enabled; vendor preset: disabled) Active: inactive (dead) since Fri 2026-10-30 14:53:48 CET; 4 days ago Docs: man:pacemakerd https://clusterlabs.org/pacemaker/doc/ Process: 62993 ExecStart=/usr/sbin/pacemakerd (code=exited, status=100) Main PID: 62993 (code=exited, status=100) Tasks: 121 Memory: 9.4G CGroup: /system.slice/pacemaker.service ├─79496 /opt/java//jre-17-openjdk/bin/java -server -XX:+UseG1GC -XX:G1ReservePercent=30 -XX:+AlwaysPreTouch -XX:+UseTransparentHugePages -XX:ConcGCThreads=2 -XX:ParallelGCT… ├─79894 /opt/java//jre-17-openjdk/bin/java -Djava.util.logging.config.file=/opt/webstart//conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLog… └─96139 /usr/sbin/crm_mon -d -i 15 -E /opt/occ/CXP_902_0588_R20CT801/tools/PCSESA.sh --output-as=html --output-to=/tmp/ClusterMon_SNMP_testuser-1.html Oct 30 14:53:48 testnode-1 systemd[1]: pacemaker.service: Succeeded. Logs: The crash is consistently preceded by pacemaker-execd failing to deliver a resource action event to controld over IPC. Full shutdown sequence (2026-10-30 14:53:48, from pacemaker.log): Oct 30 14:53:48 testnode-1 pacemakerd [62993] (pcmk__ipc_is_authentic_process_active) info: Could not connect to attrd IPC: Connection reset by peer Oct 30 14:53:48 testnode-1 pacemakerd [62993] (check_next_subdaemon) notice: Subdaemon pacemaker-attrd[62998] is unresponsive to IPC after 1 attempt (will recheck later) Oct 30 14:53:48 testnode-1 crm_mon [96138] (crm_ipc_read) error: Connection to cib_rw IPC failed Oct 30 14:53:48 testnode-1 crm_mon [96138] (mainloop_gio_callback) error: Connection to cib_rw closed | client=0x55b2ed79dd80 condition=17 Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (cancel_recurring_action) info: Cancelling ocf operation ESAPMA_OCC_monitor_10000 Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (qb_ipcs_event_sendv) warning: new_event_notification (/dev/shm/qb-62997-63000-6-7AAcJe/qb): Bad file descriptor (9) Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (pcmk__ipc_send_xml) notice: IPC message to pid 63000 failed: Bad file descriptor | rc=9 Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (send_client_notify) warning: Could not notify client crmd: Bad file descriptor | rc=9 Oct 30 14:53:48 testnode-1 pacemakerd [62993] (pcmk_child_exit) warning: Shutting cluster down because pacemaker-controld[63000] had fatal failure Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (cancel_recurring_action) info: Cancelling ocf operation ESASMA_testnode-1_monitor_10000 Oct 30 14:53:48 testnode-1 pacemakerd [62993] (pcmk_shutdown_worker) notice: Shutting down Pacemaker Oct 30 14:53:48 testnode-1 pacemakerd [62993] (stop_child) notice: Stopping subdaemon pacemaker-schedulerd | via signal 15 to process 62999 Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (cancel_recurring_action) info: Cancelling ocf operation MANAGER_OCC_monitor_10000 Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (cancel_recurring_action) info: Cancelling ocf operation ESASMS_testnode-1_monitor_10000 Oct 30 14:53:48 testnode-1 pacemakerd [62993] (pcmk_child_exit) error: pacemaker-attrd[62998] exited with status 102 (Not connected) The IPC failure pattern that immediately precedes the crash is: execd tries to notify controld of a resource action result (the repeated start/stop churn), controld's IPC endpoint is already gone (Bad file descriptor), and controld then crashes with SIGSEGV. Since the respawn budget was exhausted, pacemakerd treated this as fatal and shut the whole stack down (attrd exiting with status 102 "Not connected" is a side effect of controld's IPC teardown, not an independent fault). Expected Behavior: A resource that repeatedly fails to start should be handled via migration-threshold / on-fail (marked FAILED / stopped) without crashing pacemaker-controld or forcing a whole-cluster fatal shutdown — i.e. the Pacemaker 2.1.6 behavior we observe on the comparison node. Resource configuration (same on both nodes) -------------------------------------------------------------------------------- migration-threshold=5 failure-timeout=120s priority=70 start: timeout=120s on-fail=restart monitor: interval=10s timeout=120s on-fail=restart stop: timeout=120s on-fail=block Question: 1. Is this a known issue / already fixed after 3.0.3 in the controller or IPC (libcrmcommon mainloop_gio / event-notification) code? 2. Is there a recommended configuration workaround to avoid the fatal path while a fix is pending (e.g. tuning around resource failure handling)? Attachments we provided for working and non-working node for pacemaker logs. Pacemaker:3.0.3 --> testnode-1_pacemaker_not-working.log Pacemaker:2.1.6 --> testnode-2_pacemaker-working.log and testnode-2_pacemaker_rotated-working.log Thanks and Regards, S Sathish S
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
