Attached 403_simple.xml, AMF configuration to reproduce the problem.
Attachment: 403_simple.xml (16.3 kB; text/xml)
---
** [tickets:#403] amf: Few Assignments stuck in quiesced state and few remain
in active state**
**Status:** review
**Created:** Fri May 31, 2013 05:31 AM UTC by Praveen
**Last Updated:** Fri Oct 11, 2013 08:54 AM UTC
**Owner:** Praveen
Migrated from http://devel.opensaf.org/ticket/2912.
Model : TwoN
changeset : 3796
Configuration : Sg,3SUs with 3comps each, 5SIs with 3CSIs each.
SI-SI deps configured. SI1<-SI2<-SI3<-SI4
CSI-CSI deps configured as :
CSI1/SI1 <- CSI2/SI1 <- CSI3/SI1
CSI1/SI5 <- CSI2/SI5 <- CSI3/SI5
scenario:
--------------------------------------------------------------------------------
1. Bringup the model. SU1 is active and SU2 is standby.
2. lock the Su1.( this lock op eventually timed-out )
The comp3/SU1 was made to crash( programing error).
Amf restarted the comp3 for 3times. Later escalated to SuRestart?. As part of
SU Restart it restarted all the 3comps and put the SU1 to
RESTARTING->INSTANTIATED state. But the comp3 didnot instantiate as it crashes
again.
Finally the assignments are :
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI1,safApp=TWONAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU2\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI1,safApp=TWONAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI2,safApp=TWONAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU2\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI2,safApp=TWONAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI3,safApp=TWONAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU2\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI3,safApp=TWONAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI4,safApp=TWONAPP
saAmfSISUHAState=QUIESCED(3)
safSISU=safSu=SU2\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI4,safApp=TWONAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI5,safApp=TWONAPP
saAmfSISUHAState=QUIESCED(3)
safSISU=safSu=SU2\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI5,safApp=TWONAPP
saAmfSISUHAState=STANDBY(2)
Note the SI4 and SI5 are stucked in Quiesced state for SU1.
/var/log/messages show:
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]: Assigning
'safSi=TWONSI4,safApp=TWONAPP' QUIESCED to
'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]: Assigning
'safSi=TWONSI5,safApp=TWONAPP' QUIESCED to
'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
CMD is cleanup
APPDN is
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]:
'safComp=COMP3SU1TWONAPP,safSu=SU1,safSg=SGONE,safApp=TWONAPP' faulted due to
'avaDown(8)' : Recovery is 'componentRestart(2)'
some thing is spawnd cleanup for
safComp=COMP3SU1TWONAPP,safSu=SU1,safSg=SGONE,safApp=TWONAPP
CMD is instantiate
APPDN is SYSAPPCOMPDN
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer locally disconnected.
Marking it as doomed 10 <721, 2010f> (COMP3SU1TWONAPP)
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer locally disconnected.
Marking it as doomed 11 <722, 2010f> (@COMP3SU1TWONAPP)
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer disconnected 10 <721,
2010f> (COMP3SU1TWONAPP)
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer disconnected 11 <722,
2010f> (@COMP3SU1TWONAPP)
some thing is spawnd instantiate for
safComp=COMP3SU1TWONAPP,safSu=SU1,safSg=SGONE,safApp=TWONAPP
Nov 20 10:44:35 linux-xc76 sysAppComp: logtrace: trace enabled to file
/tmp/sysAppImmTrace, mask=0xffffffff
Nov 20 10:44:35 linux-xc76 sysAppComp: IMMA library TRACE initialize done
pid:4164 svid:26 file:/tmp/sysAppImmTrace
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer connected: 40
(COMP3SU1TWONAPP) <1161, 2010f>
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer (applier) connected:
41 (@COMP3SU1TWONAPP) <1162, 2010f>
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]: Assigned
'safSi=TWONSI5,safApp=TWONAPP' QUIESCED to
'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
CMD is cleanup
APPDN is
some thing is spawnd cleanup for
safComp=COMP3SU1TWONAPP,safSu=SU1,safSg=SGONE,safApp=TWONAPP
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]:
'safComp=COMP3SU1TWONAPP,safSu=SU1,safSg=SGONE,safApp=TWONAPP' faulted due to
'avaDown(8)' : Recovery is 'componentRestart(2)'
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer locally disconnected.
Marking it as doomed 40 <1161, 2010f> (COMP3SU1TWONAPP)
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer locally disconnected.
Marking it as doomed 41 <1162, 2010f> (@COMP3SU1TWONAPP)
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer disconnected 40 <1161,
2010f> (COMP3SU1TWONAPP)
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer disconnected 41 <1162,
2010f> (@COMP3SU1TWONAPP)
CMD is instantiate
APPDN is SYSAPPCOMPDN
some thing is spawnd instantiate for
safComp=COMP3SU1TWONAPP,safSu=SU1,safSg=SGONE,safApp=TWONAPP
Nov 20 10:44:35 linux-xc76 sysAppComp: logtrace: trace enabled to file
/tmp/sysAppImmTrace, mask=0xffffffff
Nov 20 10:44:35 linux-xc76 sysAppComp: IMMA library TRACE initialize done
pid:4187 svid:26 file:/tmp/sysAppImmTrace
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer connected: 42
(COMP3SU1TWONAPP) <1178, 2010f>
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer (applier) connected:
43 (@COMP3SU1TWONAPP) <1179, 2010f>
CMD is cleanup
APPDN is
some thing is spawnd cleanup for
safComp=COMP3SU1TWONAPP,safSu=SU1,safSg=SGONE,safApp=TWONAPP
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]:
'safComp=COMP3SU1TWONAPP,safSu=SU1,safSg=SGONE,safApp=TWONAPP' faulted due to
'avaDown(8)' : Recovery is 'suRestart(10)'
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]:
'safSu=SU1,safSg=SGONE,safApp=TWONAPP' Presence State INSTANTIATED => RESTARTING
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer locally disconnected.
Marking it as doomed 42 <1178, 2010f> (COMP3SU1TWONAPP)
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer locally disconnected.
Marking it as doomed 43 <1179, 2010f> (@COMP3SU1TWONAPP)
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer disconnected 42 <1178,
2010f> (COMP3SU1TWONAPP)
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer disconnected 43 <1179,
2010f> (@COMP3SU1TWONAPP)
CMD is instantiate
APPDN is SYSAPPCOMPDN
some thing is spawnd instantiate for
safComp=COMP3SU1TWONAPP,safSu=SU1,safSg=SGONE,safApp=TWONAPP
Nov 20 10:44:35 linux-xc76 sysAppComp: logtrace: trace enabled to file
/tmp/sysAppImmTrace, mask=0xffffffff
Nov 20 10:44:35 linux-xc76 sysAppComp: IMMA library TRACE initialize done
pid:4210 svid:26 file:/tmp/sysAppImmTrace
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Delete of PERSISTENT runtime
object 'COMP2SU1TWONAPP' by Impl: COMP2SU1TWONAPP
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer disconnected 12 <732,
2010f> (COMP2SU1TWONAPP)
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer disconnected 13 <733,
2010f> (@COMP2SU1TWONAPP)
CMD is instantiate
APPDN is SYSAPPCOMPDN
some thing is spawnd instantiate for
safComp=COMP2SU1TWONAPP,safSu=SU1,safSg=SGONE,safApp=TWONAPP
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer connected: 44
(COMP3SU1TWONAPP) <1195, 2010f>
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer (applier) connected:
45 (@COMP3SU1TWONAPP) <1196, 2010f>
Nov 20 10:44:35 linux-xc76 sysAppComp: logtrace: trace enabled to file
/tmp/sysAppImmTrace, mask=0xffffffff
Nov 20 10:44:35 linux-xc76 sysAppComp: IMMA library TRACE initialize done
pid:4229 svid:26 file:/tmp/sysAppImmTrace
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Delete of PERSISTENT runtime
object 'COMP1SU1TWONAPP' by Impl: COMP1SU1TWONAPP
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer disconnected 14 <743,
2010f> (COMP1SU1TWONAPP)
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer disconnected 15 <744,
2010f> (@COMP1SU1TWONAPP)
CMD is instantiate
APPDN is SYSAPPCOMPDN
some thing is spawnd instantiate for
safComp=COMP1SU1TWONAPP,safSu=SU1,safSg=SGONE,safApp=TWONAPP
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer connected: 46
(COMP2SU1TWONAPP) <1206, 2010f>
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Implementer (applier) connected:
47 (@COMP2SU1TWONAPP) <1207, 2010f>
Nov 20 10:44:35 linux-xc76 osafimmnd[2885]: Create of PERSISTENT runtime
object 'COMP2SU1TWONAPP' by Impl COMP2SU1TWONAPP
Nov 20 10:44:35 linux-xc76 sysAppComp: logtrace: trace enabled to file
/tmp/sysAppImmTrace, mask=0xffffffff
Nov 20 10:44:35 linux-xc76 sysAppComp: IMMA library TRACE initialize done
pid:4248 svid:26 file:/tmp/sysAppImmTrace
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]:
'safSu=SU1,safSg=SGONE,safApp=TWONAPP' Presence State RESTARTING => INSTANTIATED
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]: Assigning
'safSi=TWONSI1,safApp=TWONAPP' ACTIVE to 'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]: Assigning
'safSi=TWONSI2,safApp=TWONAPP' ACTIVE to 'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]: Assigning
'safSi=TWONSI3,safApp=TWONAPP' ACTIVE to 'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]: Assigning
'safSi=TWONSI4,safApp=TWONAPP' QUIESCED to
'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
Nov 20 10:44:35 linux-xc76 osafamfnd[2951]: Assigning
'safSi=TWONSI5,safApp=TWONAPP' QUIESCED to
'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
Nov 20 10:44:36 linux-xc76 osafimmnd[2885]: Implementer connected: 48
(COMP1SU1TWONAPP) <1229, 2010f>
Nov 20 10:44:36 linux-xc76 osafimmnd[2885]: Implementer (applier) connected:
49 (@COMP1SU1TWONAPP) <1230, 2010f>
Nov 20 10:44:36 linux-xc76 osafimmnd[2885]: Create of PERSISTENT runtime
object 'COMP1SU1TWONAPP' by Impl COMP1SU1TWONAPP
Nov 20 10:44:36 linux-xc76 osafamfnd[2951]: Assigned
'safSi=TWONSI2,safApp=TWONAPP' ACTIVE to 'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
Nov 20 10:44:36 linux-xc76 osafamfnd[2951]: Assigned
'safSi=TWONSI3,safApp=TWONAPP' ACTIVE to 'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
Nov 20 10:44:36 linux-xc76 osafamfnd[2951]: Assigned
'safSi=TWONSI1,safApp=TWONAPP' ACTIVE to 'safSu=SU1,safSg=SGONE,safApp=TWONAPP'
logs are huge and couldn't attach. states of entities is attached.
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
