- **status**: review --> fixed
- **Comment**:
commit 9518b6eb21e32b3486d9fcbd63cf0f9f0bdb5a39 (HEAD -> develop,
origin/develop, ticket-3407)
Author: Thang Duc Nguyen <[email protected]>
Date: Fri Oct 16 01:17:11 2026 +0000
amf: reboot node when CLM initialization hangs indefinitely [#3407]
avnd_clm_init() retries saClmInitialize_4() indefinitely on
SA_AIS_ERR_TRY_AGAIN, SA_AIS_ERR_TIMEOUT, and SA_AIS_ERR_UNAVAILABLE
with only a 100ms sleep between retries and no timeout. This causes
osafamfnd to hang permanently, leaving the node disabled in AMF with
no automatic recovery.
Fix by adding a 60-second wall-clock deadline to the retry loop. If
saClmInitialize_4() has not succeeded within 60 seconds, call
opensaf_reboot() with reason "CLM init timeout" so the node re-joins
the cluster cleanly rather than staying permanently disabled.
The retry interval remains 100ms; the wall-clock deadline is the
primary bound on total wait time. A pure retry count is insufficient
because each call can block up to ~10 seconds when CLMS is registered
but unresponsive, making elapsed time per retry unpredictable. The
wall-clock deadline bounds the total wait regardless of how long each
individual call takes.
---
**[tickets:#3407] amf: amfnd hangs indefinitely when CLMS is unresponsive
during initialization**
**Status:** fixed
**Milestone:** 5.27.01
**Created:** Thu Oct 15, 2026 10:40 AM UTC by Thang Duc Nguyen
**Last Updated:** Fri Oct 16, 2026 04:09 AM UTC
**Owner:** Thang Duc Nguyen
avnd_clm_init() retries saClmInitialize_4() indefinitely on
SA_AIS_ERR_TRY_AGAIN, SA_AIS_ERR_TIMEOUT, and SA_AIS_ERR_UNAVAILABLE with only
a 100ms sleep between retries, but no timeout or reboot trigger. This causes
osafamfnd to stay permanently stuck, leaving the node disabled in AMF with no
automatic recovery.
Root Cause:
saClmInitialize_4() can fail persistently in two distinct ways:
1. CLMS not yet registered in MDS (clms_up == false): returns
SA_AIS_ERR_UNAVAILABLE immediately. This occurs when CLMD has not yet started
or has not yet registered with MDS. Each retry returns instantly, so the 100ms
sleep paces the loop at ~10 retries/second — but since there is no retry limit,
the loop runs forever if CLMS never comes up.
2. CLMS registered but unresponsive (clms_up == true): the agent proceeds to
send an MDS SNDRSP message to CLMS and waits for a response. If CLMS does not
respond (e.g. CLMD has started and registered in MDS but has not yet been
elected active coordinator by AMF), the MDS sync send blocks for the full
CLMS_WAIT_TIME timeout (1000 × 10ms = 10 seconds) before returning
SA_AIS_ERR_TRY_AGAIN. In this case each retry costs ~10 seconds, making the
100ms sleep irrelevant and the stuck state effectively permanent.
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list._______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets