Analysis:
1) Lock timestamp:
Jul 9 16:32:20.638345 osafamfd [8400:avd_node.c:0770] >>
avd_node_admin_lock_unlock_shutdown: safAmfNode=SC-1,safAmfCluster=myAmfCluster
Jul 9 16:32:20.638358 osafamfd [8400:avd_node.c:0660] >> node_admin_state_set:
safAmfNode=SC-1,safAmfCluster=myAmfCluster AdmState UNLOCKED => LOCKED
2)When nodefailover got escalated, SC-1 gets del response from SU1
Jul 9 16:32:20.913263 osafamfd [8400:avd_sgproc.c:0536] >>
avd_su_si_assign_evh: id:170, node:2010f, act:4,
'safSu=SU1,safSg=SGONE,safApp=TWONAPP', '', ha:2, err:1, single:0
3)SUSIs were deleted for SU1 in AMFD database:
Jul 9 16:32:20.913973 osafamfd [8400:avd_sgproc.c:1693] >>
avd_sg_su_asgn_del_util: 'safSu=SU1,safSg=SGONE,safApp=TWONAPP', del:1, q:0
4)As a part of nodefailover SU2 was chosen for new standby assignments:
Jul 9 16:32:22.517301 osafamfd [8400:avd_sgproc.c:0252] >>
avd_su_oper_state_evh: id:174, node:2010f,
safSu=SU1,safSg=SGONE,safApp=TWONAPP' state:2
After performing failover of the SU1 to SU2 AMFD generally performs the reboot
of the node if amfnd gives all components termination event to amfd. But AMFND
was still busy in termination of components. During this time AMFD was updating
IMM for runtime modification of SUSIs in SU1 and SU2. It could perform it only
for SI1 and it got all component termination indication from amfd so it
rebooted itself.
Jul 9 16:32:22.514959 osafamfd [8400:avd_imm.c:1561] >> job_exec_imm_objdelete:
Delete
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI1,safApp=TWONAPP
Jul 9 16:32:22.517251 osafamfd [8400:avd_imm.c:1624] << avd_job_fifo_execute: 1
Since only runtime update of SI1 was done to IMM for SU1 before AMFD reboots
itself, susi status is showing only four SUSIs for SI2-SI5 in SU1. No runtime
SUSI update for SU2 was done before the active AMFD reboots so
"/etc/init.d/opensafd status" is not showing SUSIs for SU2.
I think this problem has already been encountered recently or at least it has
been part of discussion in the context of some other ticket.
---
** [tickets:#494] Recovery escalated to Nodefailover but the si assignments are
not removed**
**Status:** unassigned
**Created:** Tue Jul 09, 2013 11:18 AM UTC by surender khetavath
**Last Updated:** Tue Jul 09, 2013 11:18 AM UTC
**Owner:** nobody
Changeset : 4325
Model : TWON
Configuration: 1SG,5SUs having 3comps each, 5SIs with 3Csis each.
Intially: 5Node cluster, SU1 mapped to SC-1,SU2 to SC-2,SU3-PL3,SU4&SU5 to PL-4
SU3 was active and SU1 standby
si-si deps configured as SI1<-SI2<-SI3<-SI4
Test:
Lock of node SC-1. A component in SU1 is made to reject the remove cbk.
NodeFailover and SC-1 went for reboot.But assignments are not removed.
SUSI states:
/etc/init.d/opensafd status
safSISU=safSu=SU3\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI4,safApp=TWONAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU3\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI1,safApp=TWONAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SC-1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU3\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI2,safApp=TWONAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU3\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI3,safApp=TWONAPP
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-2\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-2\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI5,safApp=TWONAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI2,safApp=TWONAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI4,safApp=TWONAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SU1\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI3,safApp=TWONAPP
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=PL-3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed4,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=PL-5\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed5,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SU3\,safSg=SGONE\,safApp=TWONAPP,safSi=TWONSI5,safApp=TWONAPP
saAmfSISUHAState=ACTIVE(1)
SI states:
safSi=TWONSI4,safApp=TWONAPP
saAmfSIAdminState=UNLOCKED(1)
saAmfSIAssignmentState=PARTIALLY_ASSIGNED(3)
safSi=TWONSI1,safApp=TWONAPP
saAmfSIAdminState=UNLOCKED(1)
saAmfSIAssignmentState=PARTIALLY_ASSIGNED(3)
safSi=TWONSI5,safApp=TWONAPP
saAmfSIAdminState=UNLOCKED(1)
saAmfSIAssignmentState=PARTIALLY_ASSIGNED(3)
safSi=TWONSI3,safApp=TWONAPP
saAmfSIAdminState=UNLOCKED(1)
saAmfSIAssignmentState=PARTIALLY_ASSIGNED(3)
safSi=TWONSI2,safApp=TWONAPP
saAmfSIAdminState=UNLOCKED(1)
saAmfSIAssignmentState=PARTIALLY_ASSIGNED(3)
NODE state:
safAmfNode=SC-1,safAmfCluster=myAmfCluster
saAmfNodeAdminState=UNLOCKED(1)
saAmfNodeOperState=ENABLED(1)
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
