Hi Nagendra Kumar,
The issue only reproduces on the application.
They are use openstack tool to split/merge network.
Cluster size 7 SCs.
This is my analysis get from IMM trace.
~~~
$ grep -a "@Gateway-SC-1" ./*/opensaf/osaf.log | sort -k 2,2
./SC-5/opensaf/osaf.log:<141>1 2025-10-08T19:05:42.909897+03:00 SC-5 osafimmnd
4367 osaf.log [meta sequenceId="1536"] 4367:src/imm/immnd/ImmModel.cc:14113 NO
Implementer disconnected 495 <0, 2470b(down)> (@Gateway-SC-1)
./SC-7/opensaf/osaf.log:<141>1 2025-10-08T19:05:43.048365+03:00 SC-7 osafimmnd
4378 osaf.log [meta sequenceId="659"] 4378:src/imm/immnd/ImmModel.cc:14113 NO
Implementer disconnected 495 <0, 2470b(down)> (@Gateway-SC-1)
./SC-2/opensaf/osaf.log:<141>1 2025-10-08T19:05:43.369992+03:00 SC-2 osafimmnd
4366 osaf.log [meta sequenceId="661"] 4366:src/imm/immnd/ImmModel.cc:14113 NO
Implementer disconnected 495 <0, 2470b(down)> (@Gateway-SC-1)
./SC-3/opensaf/osaf.log:<141>1 2025-10-08T19:05:43.402132+03:00 SC-3 osafimmnd
4356 osaf.log [meta sequenceId="1727"] 4356:src/imm/immnd/ImmModel.cc:14113 NO
Implementer disconnected 495 <0, 2470b(down)> (@Gateway-SC-1)
./SC-1/opensaf/osaf.log:<141>1 2025-10-08T19:05:43.500636+03:00 SC-1 osafimmnd
4376 osaf.log [meta sequenceId="510"] 4376:src/imm/immnd/ImmModel.cc:14228 NO
Implementer disconnected 495 <417, 2470b> (@Gateway-SC-1)
./SC-4/opensaf/osaf.log:<141>1 2025-10-08T19:05:43.605153+03:00 SC-4 osafimmnd
4372 osaf.log [meta sequenceId="634"] 4372:src/imm/immnd/ImmModel.cc:14113 NO
Implementer disconnected 495 <0, 2470b(down)> (@Gateway-SC-1)
./SC-6/opensaf/osaf.log:<141>1 2025-10-08T19:06:12.989761+03:00 SC-6 osafimmnd
4370 osaf.log [meta sequenceId="3958"] 4370:src/imm/immnd/ImmModel.cc:14113 NO
Implementer disconnected 495 <0, 2470b(down)> (@Gateway-SC-1)
./SC-5/opensaf/osaf.log:<141>1 2025-10-08T19:06:26.030299+03:00 SC-5 osafimmnd
4367 osaf.log [meta sequenceId="1675"] 4367:src/imm/immnd/ImmModel.cc:15144 NO
Implementer (applier) connected: 530 (@Gateway-SC-1) <0, 2470b>
./SC-1/opensaf/osaf.log:<141>1 2025-10-08T19:06:26.032695+03:00 SC-1 osafimmnd
4376 osaf.log [meta sequenceId="670"] 4376:src/imm/immnd/ImmModel.cc:15144 NO
Implementer (applier) connected: 530 (@Gateway-SC-1) <1753, 2470b>
./SC-7/opensaf/osaf.log:<141>1 2025-10-08T19:06:26.032737+03:00 SC-7 osafimmnd
4378 osaf.log [meta sequenceId="765"] 4378:src/imm/immnd/ImmModel.cc:15144 NO
Implementer (applier) connected: 530 (@Gateway-SC-1) <0, 2470b>
./SC-4/opensaf/osaf.log:<141>1 2025-10-08T19:06:26.032858+03:00 SC-4 osafimmnd
4372 osaf.log [meta sequenceId="757"] 4372:src/imm/immnd/ImmModel.cc:15144 NO
Implementer (applier) connected: 530 (@Gateway-SC-1) <0, 2470b>
./SC-2/opensaf/osaf.log:<141>1 2025-10-08T19:06:26.033066+03:00 SC-2 osafimmnd
4366 osaf.log [meta sequenceId="768"] 4366:src/imm/immnd/ImmModel.cc:15144 NO
Implementer (applier) connected: 530 (@Gateway-SC-1) <0, 2470b>
./SC-5/opensaf/osaf.log:<141>1 2025-10-08T19:07:19.704591+03:00 SC-5 osafimmnd
4367 osaf.log [meta sequenceId="2343"] 4367:src/imm/immnd/ImmModel.cc:14113 NO
Implementer disconnected 530 <0, 2470b(down)> (@Gateway-SC-1)
./SC-1/opensaf/osaf.log:<141>1 2025-10-08T19:07:20.077658+03:00 SC-1 osafimmnd
4376 osaf.log [meta sequenceId="1298"] 4376:src/imm/immnd/ImmModel.cc:14228 NO
Implementer disconnected 530 <1753, 2470b> (@Gateway-SC-1)
./SC-5/opensaf/osaf.log:<141>1 2025-10-08T19:08:51.21794+03:00 SC-5 osafimmnd
4367 osaf.log [meta sequenceId="2674"] 4367:src/imm/immnd/ImmModel.cc:15144 NO
Implementer (applier) connected: 582 (@Gateway-SC-1) <0, 2470b>
$ grep -a "Implementer name:@Gateway-SC-1" ./*/opensaf/osafimmd*
./SC-3/opensaf/osafimmd.2:<143>1 2025-10-08T19:08:51.207617+03:00 SC-3 osafimmd
4342 osafimmd [meta sequenceId="3421"] 4342:src/imm/immd/immd_evt.c:2387 T5
Implementer name:@Gateway-SC-1
./SC-4/opensaf/osafimmd:<143>1 2025-10-08T19:06:26.031376+03:00 SC-4 osafimmd
4353 osafimmd [meta sequenceId="39378"] 4353:src/imm/immd/immd_evt.c:2387 T5
Implementer name:@Gateway-SC-1
// Pending request for set implementer
<143>1 2025-10-08T19:07:27.17458+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="283723"] 4376:src/imm/immnd/immnd_evt.c:1031 T2 Added client with
id: 7010002470b <node:2470b, count:1793>
<143>1 2025-10-08T19:08:27.179062+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="284690"] 4376:src/imm/common/immsv_evt.c:6566 T8 Received:
IMMND_EVT_A2ND_OI_IMPL_SET (40) from 2470b
<143>1 2025-10-08T19:08:27.179096+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="284691"] 4376:src/imm/immnd/immnd_evt.c:2977 TR Messages pending:2
<143>1 2025-10-08T19:08:27.179107+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="284692"] 4376:src/imm/common/immsv_evt.c:6545 T8 Sending:
IMMD_EVT_ND2D_IMPLSET_REQ to 0
<143>1 2025-10-08T19:08:27.180827+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="284693"] 4376:src/imm/common/immsv_evt.c:6566 T8 Received:
IMMND_EVT_A2ND_OI_IMPL_SET (40) from 2470b
<143>1 2025-10-08T19:08:27.180854+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="284694"] 4376:src/imm/immnd/immnd_evt.c:2977 TR Messages pending:3
<143>1 2025-10-08T19:08:27.180864+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="284695"] 4376:src/imm/common/immsv_evt.c:6545 T8 Sending:
IMMD_EVT_ND2D_IMPLSET_REQ to 0
// IMM agent request reach limit fevs
// discard client. but the set implementer request still in pending request at
IMMD.
<143>1 2025-10-08T19:08:37.227825+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="285004"] 4376:src/imm/common/immsv_evt.c:6566 T8 Received:
IMMND_EVT_A2ND_OI_IMPL_SET (40) from 2470b
<140>1 2025-10-08T19:08:37.227863+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="285005"] 4376:src/imm/immnd/immnd_mds.c:79 WA ERR_BAD_HANDLE:
Handle use is blocked by pending reply on syncronous call
<143>1 2025-10-08T19:08:37.227868+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="285006"] 4376:src/imm/immnd/immnd_proc.c:92 >>
immnd_proc_imma_discard_connection
<143>1 2025-10-08T19:08:37.227873+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="285007"] 4376:src/imm/immnd/immnd_proc.c:96 T5 Attempting discard
connection id:7010002470b <n:2470b, c:1793>
<143>1 2025-10-08T19:08:37.22788+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="285008"] 4376:src/imm/immnd/ImmModel.cc:14312 >>
discardContinuations
<143>1 2025-10-08T19:08:37.227897+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="285009"] 4376:src/imm/immnd/ImmModel.cc:14365 <<
discardContinuations
<143>1 2025-10-08T19:08:37.227908+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="285010"] 4376:src/imm/immnd/immnd_proc.c:273 T5 Discard connection
id:7010002470b succeeded
<143>1 2025-10-08T19:08:37.227919+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="285011"] 4376:src/imm/immnd/immnd_proc.c:277 <<
immnd_proc_imma_discard_connection
<143>1 2025-10-08T19:08:37.22793+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="285012"] 4376:src/imm/immnd/immnd_evt.c:3025 T2 SENDRSP FAIL 9
// Client already discard.
<143>1 2025-10-08T19:08:37.228604+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="285013"] 4376:src/imm/common/immsv_evt.c:6566 T8 Received:
IMMND_EVT_A2ND_IMM_OI_FINALIZE (6) from 2470b
<143>1 2025-10-08T19:08:37.228617+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="285014"] 4376:src/imm/immnd/immnd_evt.c:2438 T2 finalize for
handle: 7010002470b
<140>1 2025-10-08T19:08:37.228641+03:00 SC-1 osafimmnd 4376 osafimmnd [meta
sequenceId="285015"] 4376:src/imm/immnd/immnd_evt.c:2443 WA IMMND - Client Node
Get Failed for cli_hdl 7700876510987
// the implementer name @Gateway-SC-1 was set after time client discard.
<143>1 2025-10-08T19:08:51.207555+03:00 SC-3 osafimmd 4342 osafimmd [meta
sequenceId="3419"] 4342:src/imm/immd/immd_evt.c:125 >> immd_process_evt
<143>1 2025-10-08T19:08:51.207586+03:00 SC-3 osafimmd 4342 osafimmd [meta
sequenceId="3420"] 4342:src/imm/common/immsv_evt.c:6560 T8 Received:
IMMD_EVT_ND2D_IMPLSET_REQ from 2470b
<143>1 2025-10-08T19:08:51.207617+03:00 SC-3 osafimmd 4342 osafimmd [meta
sequenceId="3421"] 4342:src/imm/immd/immd_evt.c:2387 T5 Implementer
name:@Gateway-SC-1
<143>1 2025-10-08T19:08:51.207647+03:00 SC-3 osafimmd 4342 osafimmd [meta
sequenceId="3422"] 4342:src/imm/immd/immd_mbcsv.c:82 >> immd_mbcsv_async_update
// SC-5 received the IMMND_EVT_D2ND_IMPLSET_RSP over FEVS
<143>1 2025-10-08T19:08:51.217783+03:00 SC-5 osafimmnd 4367 osafimmnd [meta
sequenceId="658803"] 4367:src/imm/common/immsv_evt.c:6566 T8 Received:
IMMND_EVT_D2ND_IMPLSET_RSP (60) from 0
<143>1 2025-10-08T19:08:51.217823+03:00 SC-5 osafimmnd 4367 osafimmnd [meta
sequenceId="658804"] 4367:src/imm/immnd/immnd_evt.c:11686 T2 originated here?:0
nodeId:2470b conn: 1794
<143>1 2025-10-08T19:08:51.21786+03:00 SC-5 osafimmnd 4367 osafimmnd [meta
sequenceId="658805"] 4367:src/imm/immnd/ImmModel.cc:14991 >> implementerSet
<143>1 2025-10-08T19:08:51.217899+03:00 SC-5 osafimmnd 4367 osafimmnd [meta
sequenceId="658806"] 4367:src/imm/immnd/ImmModel.cc:15051 T7 Re-using
implementer for @Gateway-SC-1
<141>1 2025-10-08T19:08:51.218016+03:00 SC-5 osafimmnd 4367 osafimmnd [meta
sequenceId="658807"] 4367:src/imm/immnd/ImmModel.cc:15144 NO Implementer
(applier) connected: 582 (@Gateway-SC-1) <0, 2470b>
<143>1 2025-10-08T19:08:51.218069+03:00 SC-5 osafimmnd 4367 osafimmnd [meta
sequenceId="658808"] 4367:src/imm/immnd/ImmModel.cc:15219 << implementerSet
// around time 19:08:51 SC-2 sync with coord node (SC-5)
// The Applier @Gateway-SC-1 already synced on the other node
./SC-5/opensaf/osaf.log:<142>1 2025-10-08T19:08:51.063495+03:00 SC-5
osafimmpbed 29215 osaf.log [meta sequenceId="1"]
29215:src/imm/immpbed/immpbe.cc:140 IN arg[0] ==
'/usr/lib64/opensaf/osafimmpbed'
./SC-5/opensaf/osaf.log:<142>1 2025-10-08T19:08:51.063565+03:00 SC-5
osafimmpbed 29215 osaf.log [meta sequenceId="2"]
29215:src/imm/immpbed/immpbe.cc:140 IN arg[1] == '--pbe'
./SC-5/opensaf/osaf.log:<142>1 2025-10-08T19:08:51.063593+03:00 SC-5
osafimmpbed 29215 osaf.log [meta sequenceId="3"]
29215:src/imm/immpbed/immpbe.cc:140 IN arg[2] == '/cluster/storage/imm.db'
<140>1 2025-10-08T19:08:51.066855+03:00 SC-1 osafimmnd 4376 osaf.log [meta
sequenceId="1431"] 4376:src/imm/immnd/immnd_evt.c:10581 WA Restart to sync with
Coord! Exit
<141>1 2025-10-08T19:08:51.199228+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="27"] 24276:src/imm/immnd/immnd_proc.c:2056 NO SERVER STATE:
IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING
<141>1 2025-10-08T19:08:51.300017+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="31"] 24276:src/imm/immnd/immnd_proc.c:2099 NO SERVER STATE:
IMM_SERVER_CLUSTER_WAITING --> IMM_SERVER_LOADING_PENDING
<141>1 2025-10-08T19:08:51.401082+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="33"] 24276:src/imm/immnd/immnd_proc.c:2191 NO SERVER STATE:
IMM_SERVER_LOADING_PENDING --> IMM_SERVER_SYNC_PENDING
<141>1 2025-10-08T19:08:51.40289+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="34"] 24276:src/imm/immnd/ImmModel.cc:2515 NO NODE STATE->
IMM_NODE_ISOLATED
<141>1 2025-10-08T19:08:57.735223+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="88"] 24276:src/imm/immnd/ImmModel.cc:2541 NO NODE STATE->
IMM_NODE_W_AVAILABLE
<141>1 2025-10-08T19:08:57.828827+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="111"] 24276:src/imm/immnd/immnd_proc.c:2232 NO SERVER STATE:
IMM_SERVER_SYNC_PENDING --> IMM_SERVER_SYNC_CLIENT
<141>1 2025-10-08T19:09:02.10838+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="1400"] 24276:src/imm/immnd/ImmModel.cc:2848 NO NODE STATE->
IMM_NODE_FULLY_AVAILABLE 2848
<141>1 2025-10-08T19:09:02.119959+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="1405"] 24276:src/imm/immnd/immnd_proc.c:2372 NO SERVER STATE:
IMM_SERVER_SYNC_CLIENT --> IMM_SERVER_READY
<141>1 2025-10-08T19:09:52.682147+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="2098"] 24276:src/imm/immnd/ImmModel.cc:2547 NO NODE STATE->
IMM_NODE_R_AVAILABLE
<141>1 2025-10-08T19:10:02.39325+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="2109"] 24276:src/imm/immnd/ImmModel.cc:19999 NO NODE STATE->
IMM_NODE_FULLY_AVAILABLE 19999
<141>1 2025-10-08T19:10:06.006342+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="2213"] 24276:src/imm/immnd/ImmModel.cc:2547 NO NODE STATE->
IMM_NODE_R_AVAILABLE
<141>1 2025-10-08T19:10:13.469864+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="2217"] 24276:src/imm/immnd/ImmModel.cc:19999 NO NODE STATE->
IMM_NODE_FULLY_AVAILABLE 19999
<141>1 2025-10-08T19:10:17.42411+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="2344"] 24276:src/imm/immnd/ImmModel.cc:2547 NO NODE STATE->
IMM_NODE_R_AVAILABLE
<141>1 2025-10-08T19:10:25.633441+03:00 SC-1 osafimmnd 24276 osaf.log [meta
sequenceId="2353"] 24276:src/imm/immnd/ImmModel.cc:19999 NO NODE STATE->
IMM_NODE_FULLY_AVAILABLE 19999
~~~
Best Regards,
Thien
---
**[tickets:#3390] imm: cannot set implementer after merge network**
**Status:** review
**Milestone:** 5.25.09
**Created:** Mon Nov 17, 2025 07:17 AM UTC by Thien Minh Huynh
**Last Updated:** Mon Nov 17, 2025 12:02 PM UTC
**Owner:** Thien Minh Huynh
The issue observes when testing roaming SC with split brain recovery.
~~~
Oct 08 19:07:20 SC-1 immagent[4920]: m_immHandle != 0 - re-init. rc: SA_AIS_OK
Oct 08 19:07:27 SC-1 immagent[4920]: initialize: saImmOiInitialize_2 - SA_AIS_OK
Oct 08 19:07:27 SC-1 immagent[4920]: initialize: saImmOiImplementerSet -
SA_AIS_ERR_TRY_AGAIN
Oct 08 19:07:31 SC-1 immagent[4920]: initialize: saImmOiImplementerSet -
SA_AIS_ERR_TRY_AGAIN
Oct 08 19:07:39 SC-1 immagent[4920]: initialize: saImmOiImplementerSet -
SA_AIS_ERR_TRY_AGAIN
Oct 08 19:07:51 SC-1 immagent[4920]: initialize: saImmOiImplementerSet -
SA_AIS_ERR_TRY_AGAIN
Oct 08 19:08:07 SC-1 immagent[4920]: initialize: saImmOiImplementerSet -
SA_AIS_ERR_TRY_AGAIN
Oct 08 19:08:37 SC-1 immagent[4920]: initialize: saImmOiImplementerSet -
SA_AIS_ERR_TIMEOUT
Oct 08 19:09:01 SC-1 immagent[4920]: initialize: saImmOiImplementerSet -
SA_AIS_ERR_BAD_HANDLE
Oct 08 19:09:01 SC-1 immagent[4920]: initialize: init failed, finalize -
SA_AIS_OK
Oct 08 19:09:01 SC-1 immagent[4920]: initialize: saImmOiImplementerSet -
killing application - SA_AIS_ERR_BAD_HANDLE
Oct 08 19:09:11 SC-1 immagent[25217]: initialize: saImmOiInitialize_2 -
SA_AIS_OK
Oct 08 19:09:11 SC-1 immagent[25217]: initialize: saImmOiImplementerSet -
SA_AIS_ERR_EXIST
~~~
cannot set implementer even immagent process already restart. It's always
return SA_AIS_ERR_EXIST
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
