> The special part is the timing: > the network came back while the fence was in flight, …
It seems the timeouts are inconsistent: The fencing timeout should be shorter than the time the fenced node takes to reboot (and re-join), so that the new DC is elected before the old one comes back. Kind regards, Ulrich Windl From: Michael Riexinger <[email protected]> Sent: Friday, November 6, 2026 9:57 AM To: Windl, Ulrich <[email protected]>; Cluster Labs - All topics related to open-source clustering welcomed <[email protected]> Cc: Michael Riexinger <[email protected]> Subject: [EXT] Re: [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected Hi Ulrich, thanks for the reply. Let me clarify the order, as it is the other way round: > Why would all nodes lose the network when one node is fenced? They don't. The network outage comes first and the fencing is its consequence. In our test environment the virtual switch was restarted on all hypervisors at once, which cut every node off for about 7 s. We reproduce that by dropping all traffic on both nodes and the qdevice host for 7 s. Node 1 keeps quorum through the qdevice and fences node 2. > Things may be different if the node isn't actually fenced, however. Node 2 was really fenced: the fence agent powers the VM off and confirms the power state, and the result was OK at 15:47:46.8. The special part is the timing: the network came back while the fence was in flight, so node 2 rejoined the corosync membership at 15:47:34 and stayed a member until the power-off took effect at 15:47:43. > Usually formation of a new cluster (DC selection) is quite fast and it should > be finished before a fenced node is restarted. The DC election is not the slow part, and the restart is not involved in the delay. Node 1 stays DC throughout. What takes 30 s is the join finalization that was pending when node 2 rejoined: the DC waits for the fence to complete and then syncs the CIB from node 2, which is already powered off: 15:47:46.782 Finalizing join-2 for 1 node (sync'ing CIB from gw-2) 15:48:16.735 Could not sync CIB from gw-2 in join-2: Timer expired Only after that timeout are the resources started, 30 s after the fence completed and while node 2 is still off. The restart of node 2 only matters for the second symptom: its next join (15:51:44) is rejected because of the failed sync recorded in join-2, and it is fenced a second time. So the question remains whether the DC should check again that the sync source is still an active member when finalization resumes after fencing. Best regards Michael Riexinger Head of Cloud Infrastructure Engineering -- [email protected]<mailto:[email protected]> Tel: +49 69 408018-261 Claranet GmbH Hanauer Landstraße 196 60314 Frankfurt am Main https://www.claranet.com/de<https://www.claranet.com/de/> [https://claranet.email/signatur/icon/xing.png]<https://www.xing.com/pages/claranetgmbh/> [https://claranet.email/signatur/icon/linkedin.png] <https://www.linkedin.com/company/claranet-deutschland/> [https://claranet.email/signatur/icon/youtube.png] <https://www.youtube.com/claranetDE/> [https://claranet.email/signatur/logo/claranet.png]<https://www.claranet.com/de/> Sitz der Gesellschaft: Frankfurt am Main, Deutschland Geschäftsführer Sebastian Vögel, Olaf Fischer HRB 50381 AG Frankfurt am Main Vat-ID DE812918694 Claranet nimmt den Schutz Ihrer personenbezogenen Daten sehr ernst und hält sich strikt an die Regeln der Datenschutzgesetze. Näheres entnehmen Sie bitte unserer Datenschutzerklärung: https://www.claranet.com/de/datenschutz [https://claranet.email/signatur/banner/sovereignty.png]<https://www.claranet.com/de/challenges/digitale-souveraenitaet/?utm_source=claranet&utm_medium=signatur&utm_campaign=mc_digsov> From: Windl, Ulrich <[email protected]> Date: Friday, 6. November 2026 at 08:26 To: Cluster Labs - All topics related to open-source clustering welcomed <[email protected]> Cc: Michael Riexinger <[email protected]> Subject: RE: [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected Usually formation of a new cluster (DC selection) is quite fast and it should be finished before a fenced node is restarted. Thinks may be different if the node isn’t actually fenced, however. Why would all nodes loose the network when one node is fenced? Kind regards, Ulrich Windl From: Users <[email protected]> On Behalf Of Michael Riexinger via Users Sent: Tuesday, November 3, 2026 10:16 AM To: [email protected] Cc: Michael Riexinger <[email protected]> Subject: [EXT] [EXT] [ClusterLabs] [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected Hi, I'd like to report what looks like a race in the controller's join handling. Bugzilla is under maintenance right now, so I'm sending it here; I'm happy to file it there once it's back up. Summary ------- The DC finalizes a join by syncing the CIB from a node it has just fenced. Recovery is delayed by 30 s, and the fenced node's next join is rejected ("failed to sync its CIB"), so it gets fenced a second time. Versions -------- Pacemaker 3.0.0 (Debian 13 package 3.0.0-2), corosync 3.1.9, corosync-qdevice 3.0.3 (net, ffsplit), sbd 1.5.2 (watchdog only). The code involved looks unchanged in 2.1.6, 3.0.1 and current main. Setup ----- Two nodes plus a quorum device. Fencing level 1 powers the node off through the cloud API (pcmk_reboot_action=off); level 2 is watchdog SBD. What happens ------------ 1. All nodes lose the network for about 7 s. 2. Node 1 (DC) keeps quorum through the quorum device and fences node 2. 3. While the fence is in flight, the network returns and node 2 rejoins the corosync membership. 4. The DC delays the join until the fence completes. It then finalizes the join by syncing the CIB *from node 2*, which is powered off by then. 5. The sync times out after 30 s. Only then does the DC hold a new election and start the resources. 6. When node 2 is started again later, the DC rejects its join because the earlier sync failed. Node 2 leaves and is fenced a second time. The start after that joins normally. Reproduced in 1 of 3 runs with a 7 s outage of all nodes. It needs the network to return while the fence is in flight. DC journal (UTC, node 2 = gw-2) ------------------------------- 15:47:33.525 pacemaker-controld: notice: Requesting fencing (reboot) targeting node gw-2 15:47:34.342 corosync [TOTEM ] A new membership (1.4f) was formed. Members joined: 2 15:47:38.870 pacemaker-controld: warning: Delaying join-2 finalization while transition in progress 15:47:43.204 pacemaker-based: notice: Node gw-2 state is now lost 15:47:46.781 pacemaker-fenced: notice: Operation 'reboot' targeting gw-2 by gw-1 for pacemaker-controld@gw-1: OK (Done) 15:47:46.782 pacemaker-controld: notice: Finalizing join-2 for 1 node (sync'ing CIB from gw-2) 15:48:16.735 pacemaker-controld: error: Could not sync CIB from gw-2 in join-2: Timer expired 15:48:16.735 pacemaker-controld: warning: Input I_ELECTION_DC received in state S_FINALIZE_JOIN from finalize_sync_callback 15:48:16.760 pacemaker-schedulerd: notice: Actions: Start ... (resources start only now) 15:51:44.756 pacemaker-controld: error: Rejecting join-3 request from node gw-2 because we failed to sync its CIB in join-2 15:51:46.800 pacemaker-controld: notice: Requesting fencing (reboot) targeting node gw-2 Code pointers (main at d981f262f81644d9faad4fe6be158401fdb53bf8) ----------------------------------------------------------------- 1. The sync source is chosen at offer time and never revalidated. do_dc_join_filter_offer() stores the node with the newest CIB in max_generation_from (daemons/controld/controld_join_dc.c L542, L568). In our case that was gw-2, whose join offer arrived right after it rejoined. https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_join_dc.c#L542 2. Finalization resumes after the fencing transition with the stale source. do_dc_join_finalize() stalls while the transition (the fence of gw-2) runs (L634-L639). When it resumes, it only checks whether max_generation_from is the local node (L628-L632), not whether that node is still an active member, and calls sync_from() on a node that is powered off (L668-L671). https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_join_dc.c#L628-L671 3. The 30 s come from the CIB call timeout. cib_op_timeout() is max(10 s x (nodes + 1), MIN_CIB_OP_TIMEOUT = 30) (daemons/controld/controld_cib.c L203, L214-L220), so 30 s for two nodes. 4. The failure is recorded after the node has already left. finalize_sync_callback() calls record_failed_sync_node() for gw-2 (controld_join_dc.c L694). The only removals are in peer_update_callback(), when a node changes away from member (L175) or loses its controller process (L214) (daemons/controld/controld_callbacks.c). Both had already happened for gw-2 at 15:47:43, before the entry was recorded at 15:48:16. https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_callbacks.c#L175 5. The stale entry rejects the next join. do_dc_join_filter_offer() rejects gw-2 on its next join (controld_join_dc.c L486-L490). The rejection makes gw-2 leave, which removes the entry, so the join after that succeeds. Expected -------- 1. A node that is being fenced, or has just been fenced, in the current transition is never chosen as the CIB sync source. 2. A failed sync from a node that has already left does not cause its next join to be rejected. Possible directions ------------------- - When finalization resumes, check that max_generation_from is still an active member. If it is not, restart the join or fall back to the local CIB. - Clear max_generation_from (or restart the join) when that node is lost or fenced during the join. - Do not record a failed sync for a node that is no longer active, or clear its entry when its fencing succeeds. Impact ------ Resources start 30 s later than necessary. After a restart, the fenced node is fenced once more, so automated recovery needs two attempts. No data was at risk in our tests. A note on a fix --------------- I would have liked to send a patch for this myself. However, the code analysis above was done with the help of an AI assistant, and under the ClusterLabs AI policy (https://github.com/ClusterLabs/ai-policies) AI-assisted code is not accepted. So I'm leaving the fix to you and providing the analysis instead. The test setup and the logs are ours, and I checked the findings against the source. I can rerun the scenario or collect more logs (crm_report, full journals) if that helps. Best regards Michael Riexinger Head of Cloud Infrastructure Engineering — Claranet GmbH Hanauer Landstraße 196 60314 Frankfurt am Main https://www.claranet.com/de<https://www.claranet.com/de/> Sitz der Gesellschaft: Frankfurt am Main, Deutschland Geschäftsführer Sebastian Vögel, Olaf Fischer HRB 50381 AG Frankfurt am Main Vat-ID DE812918694 Claranet nimmt den Schutz Ihrer personenbezogenen Daten sehr ernst und hält sich strikt an die Regeln der Datenschutzgesetze. Näheres entnehmen Sie bitte unserer Datenschutzerklärung: https://www.claranet.com/de/datenschutz
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
