Analysis:
1)Lock of Node resulted in :
Apr 18 16:00:58.492365 osafamfd [16935:avd_sgNWayfsm.c:0690] >>
avd_sg_nway_su_admin_fail: su 'safSu=SU1,safSg=SGONE,safApp=NWAYAPP'
sg_fsm_state:0
Quiesced to SI1 and SI2 in SU1
Apr 18 16:00:58.492377 osafamfd [16935:avd_siass.c:0599] >> avd_susi_mod_send:
SI 'safSi=NWAYSI1,safApp=NWAYAPP', SU 'safSu=SU1,safSg=SGONE,safApp=NWAYAPP'
ha_state:3
Apr 18 16:00:58.499392 osafamfd [16935:avd_siass.c:0599] >> avd_susi_mod_send:
SI 'safSi=NWAYSI2,safApp=NWAYAPP', SU 'safSu=SU1,safSg=SGONE,safApp=NWAYAPP'
ha_state:3
2)AMF got quiesced response for SI1:
Apr 18 16:00:58.659629 osafamfd [16935:avd_sgproc.c:0542] >>
avd_su_si_assign_evh: id:103, node:2010f, act:5,
'safSu=SU1,safSg=SGONE,safApp=NWAYAPP', 'safSi=NWAYSI1,safApp=NWAYAPP', ha:3,
err:1, single:0
It sends delete to SU1 and SU3 for SI1:
Apr 18 16:00:58.660715 osafamfd [16935:avd_siass.c:0640] >> avd_susi_del_send:
SI 'safSi=NWAYSI1,safApp=NWAYAPP', SU 'safSu=SU1,safSg=SGONE,safApp=NWAYAPP'
Apr 18 16:00:58.661358 osafamfd [16935:avd_siass.c:0640] >> avd_susi_del_send:
SI 'safSi=NWAYSI1,safApp=NWAYAPP', SU 'safSu=SU3,safSg=SGONE,safApp=NWAYAPP'
3)AMF gets assignment response for SI2 quiesced state:
Apr 18 16:00:58.811108 osafamfd [16935:avd_sgproc.c:0542] >>
avd_su_si_assign_evh: id:104, node:2010f, act:5,
'safSu=SU1,safSg=SGONE,safApp=NWAYAPP', 'safSi=NWAYSI2,safApp=NWAYAPP', ha:3,
err:1, single:0
Apr 18 16:00:58.812897 osafamfd [16935:avd_sgNWayfsm.c:2690] <<
find_pref_standby_susi
Apr 18 16:00:58.812918 osafamfd [16935:avd_siass.c:0749] >>
avd_susi_role_failover: 'safSi=NWAYSI2,safApp=NWAYAPP'
'safSu=SU4,safSg=SGONE,safApp=NWAYAPP'
Apr 18 16:00:58.812939 osafamfd [16935:avd_siDep.c:1773] >>
avd_sidep_is_si_failover_possible: SI: 'safSi=NWAYSI2,safApp=NWAYAPP'
Sends delete for SI2 in SU1 and SU4:
Apr 18 16:00:58.812950 osafamfd [16935:avd_siass.c:0640] >> avd_susi_del_send:
SI 'safSi=NWAYSI2,safApp=NWAYAPP', SU 'safSu=SU1,safSg=SGONE,safApp=NWAYAPP'
Apr 18 16:00:58.814525 osafamfd [16935:avd_siass.c:0640] >> avd_susi_del_send:
SI 'safSi=NWAYSI2,safApp=NWAYAPP', SU 'safSu=SU4,safSg=SGONE,safApp=NWAYAPP'
4)Delete response for SI1 in SU1:
Apr 18 16:00:59.008373 osafamfd [16935:avd_sgproc.c:0542] >>
avd_su_si_assign_evh: id:105, node:2010f, act:4,
'safSu=SU1,safSg=SGONE,safApp=NWAYAPP', 'safSi=NWAYSI1,safApp=NWAYAPP', ha:3,
err:1, single:0
Apr 18 16:00:59.011137 osafamfd [16935:avd_siass.c:0504] << avd_susi_delete
Apr 18 16:00:59.011148 osafamfd [16935:avd_sgNWayfsm.c:2850] TR sg_fsm_state 2
=> 1
5)Delete response for SI1 in SU3:
Apr 18 16:00:59.048793 osafamfd [16935:avd_sgproc.c:0542] >>
avd_su_si_assign_evh: id:74, node:2030f, act:4,
'safSu=SU3,safSg=SGONE,safApp=NWAYAPP', 'safSi=NWAYSI1,safApp=NWAYAPP', ha:2,
err:1, single:0
Creation of new susi for SI1 in SU2:
Apr 18 16:00:59.053186 osafamfd [16935:avd_sgproc.c:0069] >>
avd_new_assgn_susi: 'safSu=SU2,safSg=SGONE,safApp=NWAYAPP'
'safSi=NWAYSI1,safApp=NWAYAPP' state=1
Apr 18 16:00:59.066226 osafamfd [16935:avd_sgproc.c:1619] <<
avd_sg_su_oper_list_add
Apr 18 16:00:59.066233 osafamfd [16935:avd_sgNWayfsm.c:1574] TR sg_fsm_state 0
=> 1
5)Deletion response for SI2 in SU1:
Apr 18 16:00:59.122911 osafamfd [16935:avd_sgproc.c:0542] >>
avd_su_si_assign_evh: id:106, node:2010f, act:4,
'safSu=SU1,safSg=SGONE,safApp=NWAYAPP', 'safSi=NWAYSI2,safApp=NWAYAPP', ha:3,
err:1, single:0
6)Deletion response for SI2 in SU4:
Apr 18 16:00:59.205075 osafamfd [16935:avd_sgproc.c:0542] >>
avd_su_si_assign_evh: id:73, node:2040f, act:4,
'safSu=SU4,safSg=SGONE,safApp=NWAYAPP', 'safSi=NWAYSI2,safApp=NWAYAPP', ha:2,
err:1, single:0
Apr 18 16:00:59.205095 osafamfd [16935:mbcsv_api.c:0773] >>
mbcsv_process_snd_ckpt_request: Sen
7) Response for SUSI creation for SI1 in SU2:
Apr 18 16:00:59.291110 osafamfd [16935:avd_sgproc.c:0542] >>
avd_su_si_assign_evh: id:123, node:2020f, act:2,
'safSu=SU2,safSg=SGONE,safApp=NWAYAPP', 'safSi=NWAYSI1,safApp=NWAYAPP', ha:1,
err:1, single:0
New SUSI creation for SI2 in SU2:
Apr 18 16:00:59.296298 osafamfd [16935:avd_sgproc.c:0069] >>
avd_new_assgn_susi: 'safSu=SU2,safSg=SGONE,safApp=NWAYAPP'
'safSi=NWAYSI2,safApp=NWAYAPP' state=1
Apr 18 16:00:59.296305 osafamfd [16935:avd_siass.c:0161] >> avd_susi_create:
safSu=SU2,safSg=SGONE,safApp=NWAYAPP safSi=NWAYSI2,safApp=NWAYAPP state=1
Apr 18 16:00:59.318001 osafamfd [16935:avd_sgNWayfsm.c:1574] TR sg_fsm_state 0
=> 1
Apr 18 16:00:59.318013 osafamfd [16935:mbcsv_api.c:0773] >>
mbcsv_process_snd_ckpt_request: Sending checkpoint data to all STANDBY peers,
as per the send-type specified
8)Response for new SUSI creation for SI2 in SU2:
Apr 18 16:00:59.510610 osafamfd [16935:avd_sgproc.c:0542] >>
avd_su_si_assign_evh: id:124, node:2020f, act:2,
'safSu=SU2,safSg=SGONE,safApp=NWAYAPP', 'safSi=NWAYSI2,safApp=NWAYAPP', ha:1,
err:1, single:0
Apr 18 16:00:59.510691 osafamfd [16935:mbcsv_api.c:0773] >>
mbcsv_process_snd_ckpt_request: Sending checkpoint data to all STANDBY peers,
as per the send-type specified
New SUSI creation for SI1 in SU3 and SI2 in SU4:
Apr 18 16:00:59.515588 osafamfd [16935:avd_sgproc.c:0069] >>
avd_new_assgn_susi: 'safSu=SU3,safSg=SGONE,safApp=NWAYAPP'
'safSi=NWAYSI1,safApp=NWAYAPP' state=2
Apr 18 16:00:59.515621 osafamfd [16935:avd_siass.c:0161] >> avd_susi_create:
safSu=SU3,safSg=SGONE,safApp=NWAYAPP safSi=NWAYSI1,safApp=NWAYAPP state=2
Apr 18 16:00:59.525356 osafamfd [16935:avd_sgproc.c:0069] >>
avd_new_assgn_susi: 'safSu=SU4,safSg=SGONE,safApp=NWAYAPP'
'safSi=NWAYSI2,safApp=NWAYAPP' state=2
Apr 18 16:00:59.525383 osafamfd [16935:avd_siass.c:0161] >> avd_susi_create:
safSu=SU4,safSg=SGONE,safApp=NWAYAPP safSi=NWAYSI2,safApp=NWAYAPP state=2
Apr 18 16:00:59.525451 osafamfd [16935:avd_siass.c
9)Response for new SUSI creation in SU3 for SI1:
Apr 18 16:00:59.768955 osafamfd [16935:avd_sgproc.c:0542] >>
avd_su_si_assign_evh: id:75, node:2030f, act:2,
'safSu=SU3,safSg=SGONE,safApp=NWAYAPP', 'safSi=NWAYSI1,safApp=NWAYAPP', ha:2,
err:1, single:0
10)Response for new SUSI creation in SU4 for SI2:
Apr 18 16:00:59.778165 osafamfd [16935:avd_sgproc.c:0542] >>
avd_su_si_assign_evh: id:74, node:2040f, act:2,
'safSu=SU4,safSg=SGONE,safApp=NWAYAPP', 'safSi=NWAYSI2,safApp=NWAYAPP', ha:2,
err:1, single:0
The issue is AMF did not respond to IMM for the completion of admin operation.
The problem is handled in sgproc.cc:avd_su_si_assign_evh() for nway active model
} else {
if (n2d_msg->msg_info.n2d_su_si_assign.error ==
NCSCC_RC_SUCCESS) {
if ((su->sg_of_su->sg_redundancy_model ==
SA_AMF_N_WAY_REDUNDANCY_MODEL) &&
(su->sg_of_su->sg_fsm_state ==
AVD_SG_FSM_STABLE)) {
for (temp_su =
su->sg_of_su->list_of_su; temp_su != NULL;
temp_su =
temp_su->sg_list_su_next) {
su_complete_admin_op(temp_su,
SA_AIS_OK);
}
} else
; // wait for SG to become STABLE
}
}
This same code should be kept open for Nway model also because SG is becoming
stable after assignments is the SUs not hosted by the locked node. So after
last assignments in SU3 and SU4, this part for code should respond to IMM for
Nway model also.
---
** [tickets:#864] campaign fails due to a failed admin op on SC-1 in NWAY model
**
**Status:** unassigned
**Milestone:** future
**Created:** Fri Apr 18, 2014 11:01 AM UTC by surender khetavath
**Last Updated:** Fri Apr 18, 2014 11:01 AM UTC
**Owner:** nobody
changeset : 5143
Model : NWAY
A campaign is modelled to add PL-5 node and a SU on that node.
While expanding the cluster, smf tries to perform lock operation on SC-1, which
eventually fails.
syslog on SC-1:
Apr 18 16:14:21 SC-1 osafsmfd[16960]: ER Fail to invoke admin operation, too
many SA_AIS_ERR_TRY_AGAIN, giving up.
dn=[safAmfNode=SC-1,safAmfCluster=myAmfCluster], opId=[2]
Apr 18 16:14:21 SC-1 osafsmfd[16960]: ER Failed to call admin operation 2 on
safAmfNode=SC-1,safAmfCluster=myAmfCluster
Apr 18 16:14:21 SC-1 osafsmfd[16960]: ER Failed to Lock activation units in
step=safSmfStep=0004
Apr 18 16:14:21 SC-1 osafsmfd[16960]: ER Step undoing failed
Apr 18 16:14:21 SC-1 osafsmfd[16960]: ER Step safSmfStep=0004 in procedure
safSmfProc=OpenSAF-upgrade failed, step result 5
Apr 18 16:14:21 SC-1 osafsmfd[16960]: NO CAMP: Procedure
safSmfProc=OpenSAF-upgrade returned FAILED
the SU1 on SC-1 is out-of-service though SC-1 is up
safSu=SU1,safSg=SGONE,safApp=NWAYAPP
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=OUT-OF-SERVICE(1)
safSu=SC-1,safSg=NoRed,safApp=OpenSAF
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=IN-SERVICE(2)
safSu=PL-4,safSg=NoRed,safApp=OpenSAF
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=IN-SERVICE(2)
safSu=SC-1,safSg=2N,safApp=OpenSAF
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=IN-SERVICE(2)
SUSI assignments
/etc/init.d/opensafd status
safSISU=safSu=SC-1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-2\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SC-2\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=PL-3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed4,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU3\,safSg=SGONE\,safApp=NWAYAPP,safSi=NWAYSI5,safApp=NWAYAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU4\,safSg=SGONE\,safApp=NWAYAPP,safSi=NWAYSI2,safApp=NWAYAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU3\,safSg=SGONE\,safApp=NWAYAPP,safSi=NWAYSI4,safApp=NWAYAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU3\,safSg=SGONE\,safApp=NWAYAPP,safSi=NWAYSI1,safApp=NWAYAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU2\,safSg=SGONE\,safApp=NWAYAPP,safSi=NWAYSI1,safApp=NWAYAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=NWAYAPP,safSi=NWAYSI3,safApp=NWAYAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU3\,safSg=SGONE\,safApp=NWAYAPP,safSi=NWAYSI3,safApp=NWAYAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU2\,safSg=SGONE\,safApp=NWAYAPP,safSi=NWAYSI2,safApp=NWAYAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU4\,safSg=SGONE\,safApp=NWAYAPP,safSi=NWAYSI5,safApp=NWAYAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=NWAYAPP,safSi=NWAYSI4,safApp=NWAYAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=PL-5\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed5,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
