In AMFND logs on ctrl1 a lot of faults are present and AMFND handled it
correctly. For the analysis, I have considered the faults which occurred when
shutdown operation was actually performed by AMFD i.e.
Jul 9 12:40:43.180256 osafamfd [3725:avd_sg.c:1336] >> avd_sg_admin_state_set:
safSg=SGONE,safApp=TWONAPP AdmState UNLOCKED => SHUTTING_DOWN.
According to AMFND logs the fault during shutdown operation occurred not be
cause of failed operation response in quiescing callback but due to
"qscingCompleteTimeout" with recovery componentrestart. Faults due to
"qscingCompleteTimeout" continued till escalation level reached to suFailover.
Here as part of recovery AMNFD gets quiesced assignments from AMFD for SI3, SI4
and SI5 respectively. For SI3 it should not get since it is sponsor for SI4.
For SI3 AMFND did not issue any callback to COMP2 and COMP3 because they are
faulted. But it did not issue callback to COMP1 also even though it is in
healthy INSTANITATED (after faults reassignment is already going on it). So
here operation done response was not generated for SI3. For SI4 and SI5
callbacks were issued to COMP1 and when responses came, AMFND responded to AMFD
with operation done event separately for SI4 and SI5. Since operation done
event was not genereated for SI3, SG is in unstable state and SUSI in SU1 are
in quiesced stat
e. Due to this subsequent admin operation will be rejected with "WA SG not in
STABLE state (safSg=SGONE,safApp=TWONAPP)".
---
** [tickets:#492] Assignments are not removed during SG shutdown with faulty
component**
**Status:** unassigned
**Created:** Tue Jul 09, 2013 07:32 AM UTC by surender khetavath
**Last Updated:** Tue Jul 09, 2013 07:32 AM UTC
**Owner:** Praveen
Changeset : 4325
Model : TWON
Configuration: 1SG,5SUs having 3comps each, 5SIs with 3Csis each.
Intially: 5Node cluster, SU1 mapped to SC-1,SU2 to SC-2,SU3-PL3,SU4&SU5 to PL-4
SU1 was active and SU2 standby
si-si deps configured as SI1<-SI2<-SI3<-SI4
Test:
Shutdown of SG. A component in active SU is made to reply with FAILED_OP in
quiescing cbk. sg goes to locked state but assignments are not removed.
Later unlock of SG fails with time-out
SU states:
safSu=SU2,safSg=SGONE,safApp=TWONAPP
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=OUT-OF-SERVICE(1)
safSu=SU1,safSg=SGONE,safApp=TWONAPP
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=DISABLED(2)
saAmfSUPresenceState=TERMINATING(4)
saAmfSUReadinessState=OUT-OF-SERVICE(1)
safSu=SU3,safSg=SGONE,safApp=TWONAPP
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=OUT-OF-SERVICE(1)
safSu=SU4,safSg=SGONE,safApp=TWONAPP
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=OUT-OF-SERVICE(1)
safSu=SU5,safSg=SGONE,safApp=TWONAPP
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=UNINSTANTIATED(1)
saAmfSUReadinessState=OUT-OF-SERVICE(1)
sg state:
safSg=SGONE,safApp=TWONAPP
saAmfSGAdminState=LOCKED(2)
SUSI states:
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI4,safApp=TWONAPP
saAmfSISUHAState=QUIESCED(3)
safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI2,safApp=TWONAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI5,safApp=TWONAPP
saAmfSISUHAState=QUIESCED(3)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI1,safApp=TWONAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-2\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI3,safApp=TWONAPP
saAmfSISUHAState=QUIESCED(3)
safSISU=safSu=PL-5\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed5,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-2\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=PL-3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed4,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
syslog on active ctrl ie sc-1:
Jul 9 12:43:09 SC-1 osafamfd[3725]: WA SG not in STABLE state
(safSg=SGONE,safApp=TWONAPP)
Jul 9 12:43:10 SC-1 osafamfd[3725]: WA SG not in STABLE state
(safSg=SGONE,safApp=TWONAPP)
Jul 9 12:43:11 SC-1 osafamfd[3725]: WA SG not in STABLE state
(safSg=SGONE,safApp=TWONAPP)
Jul 9 12:43:12 SC-1 osafamfd[3725]: WA SG not in STABLE state
(safSg=SGONE,safApp=TWONAPP)
Jul 9 12:43:13 SC-1 osafamfd[3725]: WA SG not in STABLE state
(safSg=SGONE,safApp=TWONAPP)
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
