Yes BAD_HANDLE should be returned in all cases like this one. A bad handle is a
bad handle...
As can be seen in the syslog snippet, after killing immnd amfd gets BAD_HANDLE
and the problem disappears.
---
** [tickets:#637] amfd cannot update state in IMM**
**Status:** accepted
**Created:** Fri Nov 22, 2013 02:34 PM UTC by Hans Feldt
**Last Updated:** Mon Nov 25, 2013 07:39 AM UTC
**Owner:** Anders Bjornerstedt
OpenSAF 4.2.2 changeset 3796, 79 extra patches
System: RHEL based, 2 node cluster, MDS/TIPC
After system restore (and start) active amfd process cannot use IMM.
Continuously this is logged:
Nov 21 17:42:38 warning atrcxb3166 osafimmnd[5779]: IMMND - Client went down so
no response
Nov 21 17:42:48 warning atrcxb3166 osafimmnd[5779]: IMMND - Client went down so
no response
the 10sec delay between I interpret as the default IMMA timeout.
amfd trace:
Nov 21 16:44:15.043767 osafamfd [6712:avd_imm.c:1603] >> avd_job_fifo_execute
Nov 21 16:44:15.043775 osafamfd [6712:avd_imm.c:1522] >>
job_exec_imm_objupdate: safComp=AMFWDOG,safSu=SC-1,safSg=NoRed,safApp=OpenSAF
saAmfCompPresenceState
Nov 21 16:44:15.043782 osafamfd [6712:imma_oi_api.c:2235] >>
saImmOiRtObjectUpdate_2
Nov 21 16:44:25.054146 osafamfd [6712:imma_oi_api.c:2513] <<
saImmOiRtObjectUpdate_2
Nov 21 16:44:25.054177 osafamfd [6712:avd_imm.c:1541] TR TIMEOUT
Nov 21 16:44:25.054189 osafamfd [6712:avd_imm.c:1620] << avd_job_fifo_execute: 2
immnd trace:
Nov 21 16:53:51.605545 osafimmnd [5779:immnd_proc.c:1463] T5 tmout:1000 ste:10
ME:1015 RE:1015 crd:1 rim:KEEP_REPO 4.1A:1
Nov 21 16:53:55.665763 osafimmnd [5779:immsv_evt.c:4958] T8 Received:
IMMND_EVT_A2ND_CL_TIMEOUT (93) from 2010f
Nov 21 16:53:55.665816 osafimmnd [5779:immnd_evt.c:1649] >>
immnd_evt_proc_cl_imma_timeout
Nov 21 16:53:55.665831 osafimmnd [5779:immnd_evt.c:1651] T2 timeout in imma
library for handle: 130002010f
Nov 21 16:53:55.665846 osafimmnd [5779:immnd_evt.c:1679] <<
immnd_evt_proc_cl_imma_timeout
Nov 21 16:53:55.666543 osafimmnd [5779:immsv_evt.c:4958] T8 Received:
IMMND_EVT_A2ND_OI_OBJ_MODIFY (38) from 2010f
Nov 21 16:53:55.666589 osafimmnd [5779:immnd_evt.c:2253] >>
immnd_evt_proc_rt_update
Nov 21 16:53:55.667745 osafimmnd [5779:immnd_evt.c:2259] WA IMMND - Client went
down so no response
Nov 21 16:53:55.667769 osafimmnd [5779:immnd_evt.c:2260] <<
immnd_evt_proc_rt_update
Nov 21 16:53:56.630075 osafimmnd [5779:immnd_proc.c:1463] T5 tmout:1000 ste:10
ME:1015 RE:1015 crd:1 rim:KEEP_REPO 4.1A:1
this goes on forever, at least for 2 days...
After I killed immnd things recovered:
Nov 22 11:44:37 notice atrcxb3166 osafamfnd[6965]:
'safComp=IMMND,safSu=SC-1,safSg=NoRed,safApp=OpenSAF' faulted due to
'avaDown(8)' : Recovery is 'componentRestart(2)'
Nov 22 11:44:37 warning atrcxb3166 osafimmd[5766]: IMMND coordinator at 2010f
apparently crashed => electing new coord
Nov 22 11:44:37 notice atrcxb3166 osafimmd[5766]: New coord elected, resides at
2020f
Nov 22 11:44:37 notice atrcxb3166 osafimmnd[7595]: Started
Nov 22 11:44:37 notice atrcxb3166 osafimmnd[7595]: Persistent Back-End
capability configured, Pbe file:imm.db
Nov 22 11:44:37 notice atrcxb3166 osafimmnd[7595]: Director Service is up
Nov 22 11:44:37 notice atrcxb3166 osafimmnd[7595]: SERVER STATE:
IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING
Nov 22 11:44:37 notice atrcxb3166 osafimmd[5766]: New IMMND process is on
ACTIVE Controller at 2010f
Nov 22 11:44:37 notice atrcxb3166 osafimmnd[7595]: SERVER STATE:
IMM_SERVER_CLUSTER_WAITING --> IMM_SERVER_LOADING_PENDING
Nov 22 11:44:37 notice atrcxb3166 osafimmnd[7595]: SERVER STATE:
IMM_SERVER_LOADING_PENDING --> IMM_SERVER_SYNC_PENDING
Nov 22 11:44:37 warning atrcxb3166 osafimmd[5766]: IMMND on controller (not
currently coord) requests sync
Nov 22 11:44:37 notice atrcxb3166 osafimmd[5766]: Node 2010f request sync
sync-pid:7595 epoch:0
Nov 22 11:44:37 notice atrcxb3166 osafimmnd[7595]: NODE STATE->
IMM_NODE_ISOLATED
Nov 22 11:44:37 notice atrcxb3166 osafamfd[6712]: Re-initializing with IMM
Nov 22 11:44:38 notice atrcxb3166 osafimmd[5766]: Successfully announced sync.
New ruling epoch:1043
Nov 22 11:44:38 notice atrcxb3166 osafimmnd[7595]: NODE STATE->
IMM_NODE_W_AVAILABLE
Nov 22 11:44:38 notice atrcxb3166 osafimmnd[7595]: SERVER STATE:
IMM_SERVER_SYNC_PENDING --> IMM_SERVER_SYNC_CLIENT
Nov 22 11:44:39 notice atrcxb3166 osafimmd[5766]: ACT: New Epoch for IMMND
process at node 2020f old epoch: 1042 new epoch:1043
Nov 22 11:44:39 notice atrcxb3166 osafimmnd[7595]: NODE STATE->
IMM_NODE_FULLY_AVAILABLE 1977
Nov 22 11:44:39 notice atrcxb3166 osafimmnd[7595]: RepositoryInitModeT is
SA_IMM_KEEP_REPOSITORY
Nov 22 11:44:39 notice atrcxb3166 osafimmnd[7595]: Epoch set to 1043 in ImmModel
Nov 22 11:44:39 notice atrcxb3166 osafimmd[5766]: ACT: New Epoch for IMMND
process at node 2010f old epoch: 0 new epoch:1043
Nov 22 11:44:39 notice atrcxb3166 osafimmnd[7595]: Implementer connected: 26
(safSmfService) <965, 2010f>
Nov 22 11:44:39 notice atrcxb3166 osafimmnd[7595]: Implementer connected: 28
(safClmService) <14, 2010f>
Nov 22 11:44:39 notice atrcxb3166 osafimmnd[7595]: Implementer connected: 30
(safLogService) <4, 2010f>
Nov 22 11:44:39 notice atrcxb3166 osafimmnd[7595]: Implementer connected: 31
(safCheckPointService) <1021, 2010f>
Nov 22 11:44:39 notice atrcxb3166 osafimmnd[7595]: SERVER STATE:
IMM_SERVER_SYNC_CLIENT --> IMM SERVER READY
Nov 22 11:44:39 notice atrcxb3166 osafimmnd[7595]: Implementer connected: 33
(safAmfService) <1064, 2010f>
Nov 22 11:44:39 notice atrcxb3166 osafamfd[6712]: Finished re-initializing with
IMM
Nov 22 11:44:41 notice atrcxb3166 osafimmnd[7595]: PBE-OI established on other
SC. Dumping incrementally to file imm.db
Unfortunately no syslogs from the opensaf start on this node exist. Meaning the
point in time where the problem is introduced does not have any logs.
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
------------------------------------------------------------------------------
Shape the Mobile Experience: Free Subscription
Software experts and developers: Be at the forefront of tech innovation.
Intel(R) Software Adrenaline delivers strategic insight and game-changing
conversations that shape the rapidly evolving mobile landscape. Sign up now.
http://pubads.g.doubleclick.net/gampad/clk?id=63431311&iu=/4140/ostg.clktrk
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
