Anlysis:
1)During lock of sponse SI, AMFD send removal of assignment to SU2 and
quiesced state to SU1.
2)AMFD gets quiesced response from SU1 and its sends removal to Su1 for SI
Sep 4 16:39:27.893139 osafamfd [2318:sgproc.cc:0751] >> avd_su_si_assign_evh:
id:104, node:2010f, act:5, 'safSu=SU1,safSg=SGONE,safApp=TWONAPP',
'safSi=TWONSI1,safApp=TWONAPP', ha:3, err:1, single:0
Sep 4 16:39:27.894016 osafamfd [2318:siass.cc:0558] >> avd_susi_del_send: SI
'safSi=TWONSI1,safApp=TWONAPP', SU 'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
3)During the processing of remove callback component faulted with sufailover
recovery, but
AMFD gets success for assignment:
Sep 4 16:39:28.069876 osafamfnd [2328:err.cc:0395] NO
'safComp=COMP2SU1TWONAPP,safSu=SU1,safSg=SGONE,safApp=TWONAPP' faulted due to
'csiRemovecallbackFailed' : Recovery is 'suFailover'
Sep 4 16:39:28.151234 osafamfd [2318:sgproc.cc:0751] >> avd_su_si_assign_evh:
id:109, node:2010f, act:4, 'safSu=SU1,safSg=SGONE,safApp=TWONAPP',
'safSi=TWONSI1,safApp=TWONAPP', ha:3, err:1, single:0
Here AMFD deletes the susi for SI1
4)AMFD gets su-failover request for SU1:
Sep 4 16:39:28.178174 osafamfd [2318:sgproc.cc:0468] >> avd_su_oper_state_evh:
id:114, node:2010f, 'safSu=SU1,safSg=SGONE,safApp=TWONAPP' state:2
Here AMFD sends deletion for SI2,SI3 and SI4 to SU1 and SU2 and active
modification for SI5 to SU2.
Sep 4 16:39:28.185986 osafamfd [2318:siass.cc:0558] >> avd_susi_del_send: SI
'safSi=TWONSI2,safApp=TWONAPP', SU 'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
Sep 4 16:39:28.189938 osafamfd [2318:siass.cc:0558] >> avd_susi_del_send: SI
'safSi=TWONSI2,safApp=TWONAPP', SU 'safSu=SU2,safSg=SGONE,safApp=TWONAPP'
Sep 4 16:39:28.192625 osafamfd [2318:siass.cc:0558] >> avd_susi_del_send: SI
'safSi=TWONSI3,safApp=TWONAPP', SU 'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
Sep 4 16:39:28.195139 osafamfd [2318:siass.cc:0558] >> avd_susi_del_send: SI
'safSi=TWONSI3,safApp=TWONAPP', SU 'safSu=SU2,safSg=SGONE,safApp=TWONAPP'
Sep 4 16:39:28.198414 osafamfd [2318:siass.cc:0558] >> avd_susi_del_send: SI
'safSi=TWONSI4,safApp=TWONAPP', SU 'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
Sep 4 16:39:28.202968 osafamfd [2318:siass.cc:0558] >> avd_susi_del_send: SI
'safSi=TWONSI4,safApp=TWONAPP', SU 'safSu=SU2,safSg=SGONE,safApp=TWONAPP'
Sep 4 16:39:28.206751 osafamfd [2318:siass.cc:0517] >> avd_susi_mod_send: SI
'safSi=TWONSI5,safApp=TWONAPP', SU 'safSu=SU2,safSg=SGONE,safApp=TWONAPP'
ha_state:1
AMFD deletes SUSIs in SU1 and instantiated SU5
5) AMFD gets deletion for SI1 in SU2:
Sep 4 16:39:28.403824 osafamfd [2318:sgproc.cc:0751] >> avd_su_si_assign_evh:
id:106, node:2020f, act:4, 'safSu=SU2,safSg=SGONE,safApp=TWONAPP',
'safSi=TWONSI1,safApp=TWONAPP', ha:2, err:1, single:0
Sep 4 16:39:28.403845 osafamfd [2318:mbcsv_api.c:0773] >>
mbcsv_process_snd_ckpt_reque
6) AMFD gets removal response from SU2 for SI2 and active modification response
for SI5
Sep 4 16:39:28.592662 osafamfd [2318:sgproc.cc:0751] >> avd_su_si_assign_evh:
id:109, node:2020f, act:4, 'safSu=SU2,safSg=SGONE,safApp=TWONAPP',
'safSi=TWONSI2,safApp=TWONAPP', ha:2, err:1, single:0
Sep 4 16:39:28.697919 osafamfd [2318:sgproc.cc:0751] >> avd_su_si_assign_evh:
id:110, node:2020f, act:5, 'safSu=SU2,safSg=SGONE,safApp=TWONAPP',
'safSi=TWONSI5,safApp=TWONAPP', ha:1, err:1, single:0
Sep 4 16:39:28.697938 osafamfd [2318:mbcsv_api.c:0773] >>
mbcsv_process_snd_ckpt_request: Sending checkpoint data to all STANDBY peers,
as per the send-type specified
Threre are no further AMFD traces here after.
While processing these assignment at SU2, one of the component restarts
Sep 4 16:39:28.375691 osafamfnd [2846:err.cc:0395] NO
'safComp=COMP2SU2TWONAPP,safSu=SU2,safSg=SGONE,safApp=TWONAPP' faulted due to
'csiRemovecallbackFailed' : Recovery is 'componentRestart'
This happens before AMFND gets the removal for SI2:
Sep 4 16:39:28.443749 osafamfnd [2846:su.cc:0371] >>
avnd_evt_avd_info_su_si_assign_evh: 'safSu=SU2,safSg=SGONE,safApp=TWONAPP'
Sep 4 16:39:28.443759 osafamfnd [2846:sidb.cc:0904] >> avnd_su_si_rec_get:
'safSu=SU2,safSg=SGONE,safApp=TWONAPP' : 'safSi=TWONSI2,safApp=TWONAPP'
Sep 4 16:39:28.443768 osafamfnd [2846:sidb.cc:0914] << avnd_su_si_rec_get
Sep 4 16:39:28.443797 osafamfnd [2846:susm.cc:0183] >> avnd_su_siq_rec_buf:
'safSu=SU2,safSg=SGONE,safApp=TWONAPP'
Sep 4 16:39:28.443814 osafamfnd [2846:sidb.cc:0904] >> avnd_su_si_rec_get:
'safSu=SU2,safSg=SGONE,safApp=TWONAPP' : 'safSi=TWONSI2,safApp=TWONAPP'
Sep 4 16:39:28.443821 osafamfnd [2846:sidb.cc:0914] << avnd_su_si_rec_get
Sep 4 16:39:28.443841 osafamfnd [2846:susm.cc:0227] << avnd_su_siq_rec_buf
Sep 4 16:39:28.443848 osafamfnd [2846:susm.cc:0311] >> avnd_su_si_msg_prc:
'safSu=SU2,safSg=SGONE,safApp=TWONAPP', act=4, ha_state=0, single_csi=0
That is while handling the removal callback for SI1.
After restart of the component and it reassignment, AMFND responded only for
SI2 and Si5 to AMFD but not for SI3 and SI4
There are traces in AMFND of dropping the callback responses:
Sep 4 16:39:28.760079 osafamfnd [2846:cbq.cc:0240] >> avnd_evt_ava_resp_evh
Sep 4 16:39:28.760086 osafamfnd [2846:proxy.cc:0502] TR
safComp=COMP2SU2TWONAPP,safSu=SU2,safSg=SGONE,safApp=TWONAPP: Type=15
Sep 4 16:39:28.760093 osafamfnd [2846:proxy.cc:0605] >> avnd_int_ext_comp_val:
safComp=COMP2SU2TWONAPP,safSu=SU2,safSg=SGONE,safApp=TWONAPP
Sep 4 16:39:28.760100 osafamfnd [2846:cbq.cc:0334] << avnd_evt_ava_resp_evh:
Empty comp callback record
comp=safComp=COMP2SU2TWONAPP,safSu=SU2,safSg=SGONE,safApp=TWONAPP, callback
type=fc800001
This issue could not be reproduced. Attached 1046.tgz contains the
configuration and traces from both the controller after successful
verification.
Attachment: 1046.tgz (4.8 MB; application/x-compressed)
---
** [tickets:#1046] unlock of si fails after fault and one SI is only partially
assigned**
**Status:** unassigned
**Milestone:** 4.3.3
**Created:** Thu Sep 04, 2014 11:16 AM UTC by surender khetavath
**Last Updated:** Thu Sep 04, 2014 11:16 AM UTC
**Owner:** nobody
changeset : 5697
model : 2n
configuration : 1App,1SG,5SUs with 3comps each, 5SIs with 3CSIs each
si-si deps configured as SI1<-SI2<-SI3<-SI4.
SU1 is active, SU2 is standby.
SU1 is mapped to SC-1 and SU2 to SC-2,SU3 to PL-3 and SU4,5 to PL-4
saAmfSGAutoRepair=1(True)
SuFailover=1(True)
Test:
lock SI1
reject in the remove cbk
unlock SI1
Unlock of Si times out
su admin repair also times out.
safSi=TWONSI1,safApp=TWONAPP
saAmfSIAdminState=LOCKED(2)
saAmfSIAssignmentState=UNASSIGNED(1)
safSi=TWONSI5,safApp=TWONAPP
saAmfSIAdminState=UNLOCKED(1)
saAmfSIAssignmentState=PARTIALLY_ASSIGNED(3)
safSi=TWONSI3,safApp=TWONAPP
saAmfSIAdminState=UNLOCKED(1)
saAmfSIAssignmentState=UNASSIGNED(1)
safSi=TWONSI4,safApp=TWONAPP
saAmfSIAdminState=UNLOCKED(1)
saAmfSIAssignmentState=UNASSIGNED(1)
safSi=TWONSI2,safApp=TWONAPP
saAmfSIAdminState=UNLOCKED(1)
saAmfSIAssignmentState=UNASSIGNED(1)
safSu=SU1,safSg=SGONE,safApp=TWONAPP
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=DISABLED(2)
saAmfSUPresenceState=UNINSTANTIATED(1)
saAmfSUReadinessState=OUT-OF-SERVICE(1)
safSu=SU2,safSg=SGONE,safApp=TWONAPP
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=IN-SERVICE(2)
safSu=SU3,safSg=SGONE,safApp=TWONAPP
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=IN-SERVICE(2)
safSu=SU4,safSg=SGONE,safApp=TWONAPP
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=IN-SERVICE(2)
safSu=SU5,safSg=SGONE,safApp=TWONAPP
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=IN-SERVICE(2)
safSISU=safSu=SU2\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI3,safApp=TWONAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU2\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI4,safApp=TWONAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU2\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI5,safApp=TWONAPP
saAmfSISUHAState=ACTIVE(1)
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.------------------------------------------------------------------------------
Slashdot TV.
Video for Nerds. Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
