Hi Minh,
Thanks. Yes, I will send the traces of the TC mentioned below
after I am done with some other tests. The devel lists has limitation of 400 KB
limit of attachment, so will see how can I send it, may be I can attach in the
ticket as temporary.
Thanks
-Nagu
> -----Original Message-----
> From: minh chau [mailto:minh.c...@dektech.com.au]
> Sent: 11 February 2016 10:17
> To: Nagendra Kumar; hans.nordeb...@ericsson.com;
> gary....@dektech.com.au; Praveen Malviya
> Cc: opensaf-devel@lists.sourceforge.net
> Subject: Re: [devel] FW: [PATCH 0 of 5] Review Request for amf: Add support
> for cloud resilience [#1620] V2 (delayed failover issue)
>
> Hi Nagu,
>
> There's known limitation in IMM with configuration 2+1 or 1+1, which results
> in IMM reload at second controller restart. We're discussing to avoid the
> crash Meanwhile I'm also looking at the delayed failover issues which you
> reported on TC#2 #12 #13 #15 We have similar automated tests but they all
> pass, so I guess your test has something special.
> Can you run those tests once and send me syslog + amfd/amfnd traces?
>
> Thanks,
> Minh
> On 10/02/16 23:00, Nagendra Kumar wrote:
> > TC #16. Same configuration as #12: Run SI shutdown and keep sleep
> of 5 sec before saAmfCSIQuiescingComplete and stop controller and then
> after sleep, reject saAmfCSIQuiescingComplete with
> SA_AIS_ERR_FAILED_OPERATION. All the assignment from SU1 on PL-3 and
> SU2 on PL-4 are removed and SI admin state is 2(locked):
> > saAmfSIAdminState SA_UINT32_T 2 (0x2)
> >
> > "Si going into locked state" is different behaviour when controller is up
> > and
> running and run this test case. In case, controller is available, SI will be
> in
> unlocked state and all the assignments will be on SU2 as Act and SU3 as
> Standby (on PL-4). This need either correction or documentation.
> >
> > TC #17. Same configuration as #12: Run SG shutdown and
> keep sleep of 5 sec before saAmfCSIQuiescingComplete and stop controller
> and then after sleep, reject saAmfCSIQuiescingComplete with
> SA_AIS_ERR_FAILED_OPERATION. Amfnd crashes[Please note that this test
> case works with controller up]:
> > Syslog and bt:
> > Feb 10 11:44:29 PM_PL-3 osafamfnd[15508]: NO component with
> > QUIESCED/QUIESCING assignment failed Feb 10 11:44:29 PM_PL-3
> osafamfnd[15508]: NO recovery action 'comp restart' escalated to 'comp
> failover'
> > Feb 10 11:44:29 PM_PL-3 osafamfnd[15508]: NO SU failover probation
> > timer started (timeout: 1200000000000 ns) Feb 10 11:44:29 PM_PL-3
> > osafamfnd[15508]: NO Performing failover of
> 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' (SU failover count: 1) Feb
> 10 11:44:29 PM_PL-3 osafamfnd[15508]: NO
> 'safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'
> recovery action escalated from 'componentRestart' to 'componentFailover'
> > Feb 10 11:44:29 PM_PL-3 osafamfnd[15508]: NO
> 'safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'
> faulted due to 'csiSetcallbackFailed' : Recovery is 'componentFailover'
> > Feb 10 11:44:29 PM_PL-3 osafamfnd[15508]: NO
> > 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State
> INSTANTIATED => TERMINATING Feb 10 11:44:29 PM_PL-3
> osafamfnd[15508]: NO Removed 'safSi=AmfDemo,safApp=AmfDemo1' from
> 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'
> > Feb 10 11:44:29 PM_PL-3 amf_demo[15721]: saAmfHAStateGet FAILED - 7
> > Feb 10 11:44:29 PM_PL-3 amf_demo[15721]: exiting (caught term signal)
> > Feb 10 11:44:29 PM_PL-3 osafamfnd[15508]: NO avnd_di_oper_send()
> > deferred as AMF director is offline Feb 10 11:44:29 PM_PL-3
> > osafimmnd[15760]: AL AMF Node Director is down, terminate this process
> >
> > Program terminated with signal 11, Segmentation fault.
> > #0 0x0000000000412b50 in
> > avnd_comp_cmplete_all_assignment(avnd_cb_tag*, avnd_comp_tag*) ()
> > (gdb) bt
> > #0 0x0000000000412b50 in
> > avnd_comp_cmplete_all_assignment(avnd_cb_tag*, avnd_comp_tag*) ()
> > #1 0x000000000040a093 in
> > avnd_comp_clc_terming_cleansucc_hdler(avnd_cb_tag*,
> avnd_comp_tag*) ()
> > #2 0x000000000040c7d4 in avnd_comp_clc_fsm_run(avnd_cb_tag*,
> > avnd_comp_tag*, avnd_comp_clc_pres_fsm_ev) ()
> > #3 0x000000000040ce49 in avnd_evt_clc_resp_evh(avnd_cb_tag*,
> > avnd_evt_tag*) ()
> > #4 0x000000000042133f in avnd_main_process() () at main.cc:667
> > #5 0x0000000000405517 in main () at main.cc:186
> > (gdb) thread apply all bt
> >
> > Thread 4 (Thread 0x7fdaf3c05b00 (LWP 15512)):
> > #0 0x00007fdaf2b2b415 in __lll_unlock_wake () from
> > /lib64/libpthread.so.0
> > #1 0x00007fdaf2b27ac4 in _L_unlock_553 () from /lib64/libpthread.so.0
> > #2 0x00007fdaf2b279f7 in __pthread_mutex_unlock_usercnt () from
> > /lib64/libpthread.so.0
> > #3 0x00007fdaf37edac3 in ncs_os_lock () from
> > /usr/local/lib/libopensaf_core.so.0
> > #4 0x00007fdaf37e084d in ncs_ipc_send () from
> > /usr/local/lib/libopensaf_core.so.0
> > #5 0x000000000041eea1 in avnd_evt_send(avnd_cb_tag*, avnd_evt_tag*)
> > ()
> > #6 0x000000000040a2cb in
> > comp_clc_resp_callback(NCS_OS_PROC_EXECUTE_TIMED_CB_INFO*) ()
> > #7 0x00007fdaf37ecdfb in give_exec_mod_cb () from
> > /usr/local/lib/libopensaf_core.so.0
> > #8 0x00007fdaf37ecfde in ncs_exec_mod_hdlr () from
> > /usr/local/lib/libopensaf_core.so.0
> > #9 0x00007fdaf2b247b6 in start_thread () from /lib64/libpthread.so.0
> > #10 0x00007fdaf20da9cd in clone () from /lib64/libc.so.6
> > #11 0x0000000000000000 in ?? ()
> >
> > Thread 3 (Thread 0x7fdaf3c25b00 (LWP 15510)):
> > #0 0x00007fdaf20d14f6 in poll () from /lib64/libc.so.6
> > #1 0x00007fdaf3817623 in mdtm_process_recv_events () from
> > /usr/local/lib/libopensaf_core.so.0
> > #2 0x00007fdaf2b247b6 in start_thread () from /lib64/libpthread.so.0
> > #3 0x00007fdaf20da9cd in clone () from /lib64/libc.so.6
> > #4 0x0000000000000000 in ?? ()
> >
> > Thread 2 (Thread 0x7fdaf3c58b00 (LWP 15509)):
> > #0 0x00007fdaf20d14f6 in poll () from /lib64/libc.so.6
> > #1 0x00007fdaf37db22f in osaf_ppoll () from
> > /usr/local/lib/libopensaf_core.so.0
> > #2 0x00007fdaf37e2acf in ncs_tmr_wait () from
> > /usr/local/lib/libopensaf_core.so.0
> > #3 0x00007fdaf2b247b6 in start_thread () from /lib64/libpthread.so.0
> > #4 0x00007fdaf20da9cd in clone () from /lib64/libc.so.6
> > #5 0x0000000000000000 in ?? ()
> >
> > Thread 1 (Thread 0x7fdaf3c28720 (LWP 15508)):
> > #0 0x0000000000412b50 in
> > avnd_comp_cmplete_all_assignment(avnd_cb_tag*, avnd_comp_tag*) ()
> > #1 0x000000000040a093 in
> > avnd_comp_clc_terming_cleansucc_hdler(avnd_cb_tag*,
> avnd_comp_tag*) ()
> > #2 0x000000000040c7d4 in avnd_comp_clc_fsm_run(avnd_cb_tag*,
> > avnd_comp_tag*, avnd_comp_clc_pres_fsm_ev) ()
> > #3 0x000000000040ce49 in avnd_evt_clc_resp_evh(avnd_cb_tag*,
> > avnd_evt_tag*) ()
> > #4 0x000000000042133f in avnd_main_process() () at main.cc:667
> > #5 0x0000000000405517 in main () at main.cc:186
> >
> > TC #18. Same configuration as #12: Run SG lock and keep gdb
> in amf_csi_remove_callback and stop controller and then start the
> controller and make is up. Now release Amfnd from gdb so that it can
> respond to csi remove(Please note that controller has reboot and is available
> now). Now, issue SG unlock. Amfnd crashes on PL-3 and PL-4 at the same
> location[Please note that this test case works with controller up]:
> > Syslog and Bt:
> > Feb 10 16:35:51 PM_PL-3 amf_demo[26623]: CSI Remove for all CSIs Feb
> > 10 16:35:51 PM_PL-3 osafamfnd[26545]: NO Removed
> 'safSi=AmfDemo,safApp=AmfDemo1' from
> 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'
> > Feb 10 16:36:02 PM_PL-3 osafamfnd[26545]: NO Assigning
> 'safSi=AmfDemo,safApp=AmfDemo1' ACTIVE to
> 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'
> > Feb 10 16:36:02 PM_PL-3 amf_demo[26623]: CSI Set - add
> 'safCsi=AmfDemo,safSi=AmfDemo,safApp=AmfDemo1' HAState Active
> > Feb 10 16:36:02 PM_PL-3 amf_demo[26623]: name: abcdef, value: val1
> > Feb 10 16:36:02 PM_PL-3 amf_demo[26623]: name: abcdef, value: val2
> > Feb 10 16:36:02 PM_PL-3 osafamfnd[26545]: NO Assigned
> 'safSi=AmfDemo,safApp=AmfDemo1' ACTIVE to
> 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'
> > Feb 10 16:36:02 PM_PL-3 osafamfnd[26545]: di.cc:850:
> avnd_di_susi_resp_send: Assertion 'si' failed.
> > Feb 10 16:36:02 PM_PL-3 osafclmna[26536]: AL AMF Node Director is
> > down, terminate this process
> >
> > Core was generated by `/usr/local/lib/opensaf/osafamfnd --
> tracemask=0xffffffff'.
> > Program terminated with signal 6, Aborted.
> > #0 0x00007f022ebd9b55 in raise () from /lib64/libc.so.6
> > (gdb) bt
> > #0 0x00007f022ebd9b55 in raise () from /lib64/libc.so.6
> > #1 0x00007f022ebdb131 in abort () from /lib64/libc.so.6
> > #2 0x00007f023038331b in __osafassert_fail () from
> > /usr/local/lib/libopensaf_core.so.0
> > #3 0x000000000041b399 in avnd_di_susi_resp_send(avnd_cb_tag*,
> > avnd_su_tag*, avnd_su_si_rec*) ()
> > #4 0x000000000042e9fa in avnd_su_si_oper_done(avnd_cb_tag*,
> > avnd_su_tag*, avnd_su_si_rec*) ()
> > #5 0x0000000000411622 in avnd_comp_csi_assign_done(avnd_cb_tag*,
> > avnd_comp_tag*, avnd_comp_csi_rec*) ()
> > #6 0x0000000000407397 in avnd_evt_ava_resp_evh(avnd_cb_tag*,
> > avnd_evt_tag*) ()
> > #7 0x000000000042133f in avnd_main_process() () at main.cc:667
> > #8 0x0000000000405517 in main () at main.cc:186
> >
> > TC #19. Same configuration as #12: Run Node lock and keep
> sleep of 5 sec in amf_csi_set_callback and stop controller. Reject quisced
> assignment in amf_csi_set_callback, Amfnd crashes. Syslog and gdb is the
> same as in TC #17.
> >
> > TC #20. Same configuration as #12: Issue Node shutdown:
> and keep sleep of 5 sec in amf_csi_set_callback before sending
> saAmfResponse() and stop controller. Amfnd crashes:
> > Syslog:
> >
> > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO component with
> QUIESCED/QUIESCING a ssignment
> failed
> > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO recovery action 'comp
> restart' esca lated to 'comp
> failover'
> > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO SU failover probation
> timer started (timeout:
> 1200000000000 ns)
> > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO Performing failover of
> 'safSu=SU1,s
> afSg=AmfDemo,safApp=AmfDemo1'
> (SU failover count: 1)
> > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO
> 'safComp=AmfDemo,safSu=SU1,safSg=Am
> fDemo,safApp=AmfDemo1' recovery action escalated from
> 'componentRestart' to 'com
> ponentFailover'
> > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO
> 'safComp=AmfDemo,safSu=SU1,safSg=Am
> fDemo,safApp=AmfDemo1' faulted due to 'csiSetcallbackFailed' : Recovery is
> 'comp onentFailover'
> > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO
> 'safSu=SU1,safSg=AmfDemo,safApp=Amf
> Demo1'
> Presence State INSTANTIATED => TERMINATING
> > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO Removed
> 'safSi=AmfDemo,safApp=AmfDe mo1'
> from
> 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'
> > Feb 10 17:21:10 PM_PL-3 amf_demo[29519]: saAmfHAStateGet FAILED - 7
> > Feb 10 17:21:10 PM_PL-3 osafimmnd[29561]: AL AMF Node Director is
> down, terminat e this process
> > Feb 10 17:21:10 PM_PL-3 amf_demo[29519]: AL AMF Node Director is
> down, terminate this process
> > Feb 10 17:21:10 PM_PL-3 amf_demo[29519]: exiting (caught term signal)
> > Feb 10 17:21:10 PM_PL-3 osafclmna[29321]: AL AMF Node Director is
> down, terminat e this process
> >
> > Bt:
> > Core was generated by `/usr/local/lib/opensaf/osafamfnd --
> tracemask=0xffffffff'.
> > Program terminated with signal 11, Segmentation fault.
> > #0 0x00000000004117c9 in avnd_comp_csi_assign_done(avnd_cb_tag*,
> > avnd_comp_tag*, avnd_comp_csi_rec*) ()
> > (gdb) bt
> > #0 0x00000000004117c9 in avnd_comp_csi_assign_done(avnd_cb_tag*,
> > avnd_comp_tag*, avnd_comp_csi_rec*) ()
> > #1 0x0000000000406a3b in
> > avnd_evt_ava_csi_quiescing_compl_evh(avnd_cb_tag*, avnd_evt_tag*) ()
> > #2 0x000000000042133f in avnd_main_process() () at main.cc:667
> > #3 0x0000000000405517 in main () at main.cc:186
> >
> >
> > Thanks
> > -Nagu
> >
> >> -----Original Message-----
> >> From: Nagendra Kumar
> >> Sent: 09 February 2016 21:39
> >> To: minh chau; hans.nordeb...@ericsson.com;
> gary....@dektech.com.au;
> >> Praveen Malviya
> >> Cc: opensaf-devel@lists.sourceforge.net
> >> Subject: Re: [devel] FW: [PATCH 0 of 5] Review Request for amf: Add
> >> support for cloud resilience [#1620] V2
> >>
> >> 15. Same configuration as Test Case #12, SI lock. Keep gdb in both
> >> the SUs for csi remove and keep timeout as 100 sec. Slock SI and stop
> controller.
> >> Start controller and allow csi remove to timeout.
> >> Two things:
> >> SU2 has Standby assignment(which is wrong), SU1 has not
> assignment.
> >> Error at PL-4 : SU-SI record addition failed
> >>
> >> PM_SC-1:/home/nagu/views/staging # amf-state siass safSISU=safSu=PL-
> >> 4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF
> >> saAmfSISUHAState=ACTIVE(1)
> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
> >> safSISU=safSu=PL-
> >> 3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF
> >> saAmfSISUHAState=ACTIVE(1)
> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
> >>
> safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s
> >> afApp=AmfDemo1
> >> saAmfSISUHAState=STANDBY(2)
> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
> >> safSISU=safSu=SC-
> >> 1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF
> >> saAmfSISUHAState=ACTIVE(1)
> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
> >> safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC-
> >> 2N,safApp=OpenSAF
> >> saAmfSISUHAState=ACTIVE(1)
> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
> >>
> >> Syslog of PL-4:
> >>
> >> Feb 9 21:24:50 PM_PL-4 osafamfnd[7998]: NO
> >> 'safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1' component restart
> probation
> >> timer started (timeout: 60000000000 ns) Feb 9 21:24:50 PM_PL-4
> >> osafamfnd[7998]: NO Restarting a component of
> >> 'safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1' (comp restart count: 1)
> Feb
> >> 9 21:24:50 PM_PL-4 osafamfnd[7998]: NO
> >> 'safComp=AmfDemo,safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1'
> >> faulted due to 'csiRemovecallbackTimeout' : Recovery is
> 'componentRestart'
> >> Feb 9 21:24:55 PM_PL-4 amf_demo_script: killproc
> >> /opt/amf_demo/amf_demo failed Feb 9 21:24:55 PM_PL-4
> >> amf_demo[8200]:
> >> 'safComp=AmfDemo,safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1'
> >> started Feb 9 21:24:55 PM_PL-4 osafamfnd[7998]: NO Removed
> >> 'safSi=AmfDemo1,safApp=AmfDemo1' from
> >> 'safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1'
> >> Feb 9 21:24:55 PM_PL-4 amf_demo[8200]: HC started with AMF Feb 9
> >> 21:24:55 PM_PL-4 amf_demo[8200]: Registered with AMF Feb 9 21:24:55
> >> PM_PL-4 amf_demo[8200]: CSI Set - add
> >> 'safCsi=AmfDemo,safSi=AmfDemo,safApp=AmfDemo1' HAState Standby
> >> Feb 9 21:24:55 PM_PL-4 amf_demo[8200]: name: abcdef, value: val1
> >> Feb 9 21:24:55 PM_PL-4 amf_demo[8200]: name: abcdef, value: val2
> >> Feb 9 21:24:55 PM_PL-4 osafamfnd[7998]: CR SU-SI record addition
> >> failed, SU= safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 :
> >> SI=safSi=AmfDemo,safApp=AmfDemo1 Feb 9 21:24:55 PM_PL-4
> >> amf_demo[8200]: Health check 1 Feb 9 21:25:50 PM_PL-4
> >> osafamfnd[7998]: NO 'safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1'
> >> Component or SU restart probation timer expired
> >>
> >> Thanks
> >> -Nagu
> >>
> >> ---------------------------------------------------------------------
> >> ---------
> >> Site24x7 APM Insight: Get Deep Visibility into Application
> >> Performance APM + Mobile APM + RUM: Monitor 3 App instances at just
> >> $35/Month Monitor end-to-end web transactions and take corrective
> >> actions now Troubleshoot faster and improve end-user experience.
> Signup Now!
> >> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> >> _______________________________________________
> >> Opensaf-devel mailing list
> >> Opensaf-devel@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/opensaf-devel
>
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel
