I tried to reproduce this ticket on opensaf version 5.18.04
Setup : Two payloads with three controllers.(headless mode)
1)root@mohan-VirtualBox:/etc/opensaf# amf-state siass
safSISU=safSu=SC-1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed4,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=PL-5\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed5,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-2\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-2\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=STANDBY(2)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
2)I have done failover by bringing down active, standby and spare in the order.
a)on SC-1
root@mohan-VirtualBox:/home/mohan# cat /etc/opensaf/node_name
SC-1
root@mohan-VirtualBox:/etc/opensaf# /etc/init.d/opensafd stop
[....] Stopping opensafd (via systemctl): opensafd.service
[.ok
b)on SC-2
root@mohan-VirtualBox:/home/mohan# cat /etc/opensaf/node_name
SC-2
root@mohan-VirtualBox:/etc/opensaf# amf-state siass
safSISU=safSu=SC-3\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=STANDBY(2)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed4,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=PL-5\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed5,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-2\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-2\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
root@mohan-VirtualBox:/etc/opensaf# /etc/init.d/opensafd stop
[....] Stopping opensafd (via systemctl): opensafd.service
[.ok
C)on SC-3
root@mohan-VirtualBox:/home/mohan# cat /etc/opensaf/node_name
SC-3
root@mohan-VirtualBox:/etc/opensaf# amf-state siass
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed4,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=PL-5\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed5,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-3\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
root@mohan-VirtualBox:/etc/opensaf# /etc/init.d/opensafd stop
[....] Stopping opensafd (via systemctl): opensafd.service
[.ok
d)on pl-4
root@mohan-VirtualBox:/home/mohan# cat /etc/opensaf/node_name
PL-4
root@mohan-VirtualBox:/home/mohan# amf-state siass
safSISU=safSu=SC-3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed4,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-3\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=PL-5\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed5,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
3)I tried all scenarious(stop all controllers without giving time gap and with
less time gap )
to reproduce this ticket, but it is not reproduced.
4)I observed that ckptnd is not started and the node is not rebooted(payloads)
So, i am closing the ticket, please reopen if reproduced with traces.
---
** [tickets:#1867] HEADLESS : Payloads went for reboot, in headless state as
CPSV got TIMEOUT rc for CLM API**
**Status:** assigned
**Milestone:** 5.18.09
**Created:** Wed Jun 08, 2016 10:54 AM UTC by Srikanth R
**Last Updated:** Mon Sep 03, 2018 07:34 AM UTC
**Owner:** Mohan Kanakam
Version : Opensaf 5.0. GA
Setup : Two payloads with three controllers.
Steps performed :
-> Initially all the nodes are part of the cluster.
-> Induced failover by bringing down active, standby and spare in the order.
Aug 7 20:30:08 SCALE_SLOT-94 kernel: [5993776.936794] TIPC: Lost contact with
<1.1.1>
Aug 7 20:30:08 SCALE_SLOT-94 osafimmnd[2748]: NO Sleep done registering IMMND
with MDS
Aug 7 20:30:08 SCALE_SLOT-94 osafimmnd[2748]: NO MDS: mds_register_callback:
dest 2040fa5bb6016 already exist
Aug 7 20:30:08 SCALE_SLOT-94 osafimmnd[2748]: NO SUCCESS IN REGISTERING IMMND
WITH MDS
Aug 7 20:30:08 SCALE_SLOT-94 osafimmnd[2748]: NO Re-introduce-me
highestProcessed:6859 highestReceived:6859
Aug 7 20:30:13 SCALE_SLOT-94 osafimmnd[2748]: WA MDS Send Failed to
service:IMMD rc:2
Aug 7 20:30:14 SCALE_SLOT-94 osafamfnd[2767]: WA AMF director unexpectedly
crashed
-> On the both payloads, CKPTND restarted with the following error in syslog.
Aug 7 20:30:17 SCALE_SLOT-94 osafckptnd[2787]: ER cpnd clm node get failed
with return value:5
Aug 7 20:30:17 SCALE_SLOT-94 osafamfnd[2767]: NO
'safComp=CPND,safSu=PL-4,safSg=NoRed,safApp=OpenSAF' faulted due to 'avaDown' :
Recovery is 'componentRestart'
Aug 7 20:30:17 SCALE_SLOT-94 osafckptnd[14434]: Started
-> But CKPTND Instantation failed and finally the node went for reboot.
Aug 7 20:30:27 SCALE_SLOT-94 osafimmnd[2748]: NO Re-introduce-me
highestProcessed:6859 highestReceived:6859
Aug 7 20:30:27 SCALE_SLOT-94 osafimmnd[2748]: WA MDS Send Failed to
service:IMMD rc:2
Aug 7 20:30:27 SCALE_SLOT-94 osafamfnd[2767]: NO Instantiation of
'safComp=CPND,safSu=PL-4,safSg=NoRed,safApp=OpenSAF' failed
Aug 7 20:30:27 SCALE_SLOT-94 osafamfnd[2767]: NO Reason: component
registration timer expired
Aug 7 20:30:27 SCALE_SLOT-94 osafckptnd[14451]: Started
...
Aug 7 20:30:38 SCALE_SLOT-94 osafamfnd[2767]: NO Instantiation of
'safComp=CPND,safSu=PL-4,safSg=NoRed,safApp=OpenSAF' failed
Aug 7 20:30:38 SCALE_SLOT-94 osafamfnd[2767]: NO Reason: component
registration timer expired
Aug 7 20:30:38 SCALE_SLOT-94 osafimmnd[2748]: NO Re-introduce-me
highestProcessed:6859 highestReceived:6859
Aug 7 20:30:38 SCALE_SLOT-94 osafimmnd[2748]: WA MDS Send Failed to
service:IMMD rc:2
Aug 7 20:30:38 SCALE_SLOT-94 osafamfnd[2767]: WA
'safComp=CPND,safSu=PL-4,safSg=NoRed,safApp=OpenSAF' Presence State RESTARTING
=> INSTANTIATION_FAILED
Aug 7 20:30:38 SCALE_SLOT-94 osafamfnd[2767]: NO avnd_di_oper_send() deferred
as AMF director is offline
Aug 7 20:30:38 SCALE_SLOT-94 osafamfnd[2767]: WA Director is down. Remove all
SIs from 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF'
Aug 7 20:30:38 SCALE_SLOT-94 osafamfnd[2767]: NO Component Failover trigerred
for 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF': Failed component:
'safComp=CPND,safSu=PL-4,safSg=NoRed,safApp=OpenSAF'
Aug 7 20:30:38 SCALE_SLOT-94 osafamfnd[2767]: ER
'safComp=CPND,safSu=PL-4,safSg=NoRed,safApp=OpenSAF'got Inst failed
Aug 7 20:30:38 SCALE_SLOT-94 osafamfnd[2767]: Rebooting OpenSAF NodeId =
132111 EE Name = , Reason: NCS component Instantiation failed, OwnNodeId =
132111, SupervisionTime = 60
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
