Please note that these test cases with SI-Dep need to be run.
Thanks
-Nagu
----- Original Message -----
From: nagendr...@oracle.com
To: minh.c...@dektech.com.au, hans.nordeb...@ericsson.com,
gary....@dektech.com.au, praveen.malv...@oracle.com
Cc: opensaf-devel@lists.sourceforge.net
Sent: Monday, March 28, 2016 8:33:25 PM GMT +05:30 Chennai, Kolkata, Mumbai,
New Delhi
Subject: Re: [devel] [PATCH 01 of 15] amfd: Add support for cloud resilience at
common libs [#1620]
Hi,
Please find the result of remaining test cases. This is partial testing
results and I am continuing with other test cases and would be only able to
share them by Thursday.
TC #1: Configuration and logs(Admin Op TC 1) is attached in the ticket.
Steps:
1. Start SC-1 and PL-3 and PL-4. 2N SG, SU1(safSu=ABC1,safSg=2N,safApp=Test)
hosted on PL-3 and SU2(safSu=ABC2,safSg=2N,safApp=Test) hosted on PL-4. SU
having two components, Two SIs(each SI has one CSI). One CSI is on one Comp
i.e. Maps like : One SI->one CSI-> One Comp.
immcfg -f /opt/amf_demo/2N_2comps.xml
amf-adm unlock-in safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock-in safSu=ABC2,safSg=2N,safApp=Test
amf-adm unlock safSu=ABC2,safSg=2N,safApp=Test
2. Now, Issue lock on SU1(say) (safSu=ABC1,safSg=2N,safApp=Test). Keep gdb in
one of the component in Quiesced cbk (amf_csi_set_callback). Another components
responds to Quisced cbk. Now, stop controller and start it.
3. After controller comes up and Cluster Timer expiry,
Amf gives Act assignment to SU2 (say) safSu=ABC2,safSg=2N,safApp=Test.
This is fundamentally wrong because, Quisced cbk is still not responded.
PM_PL-4:/opt/amf_demo # Mar 18 16:42:36 PM_PL-4 osafamfnd[2659]: NO Assigning
'all (2) SIs' ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 18 16:42:36 PM_PL-4 osafamfnd[2659]: NO Assigning 'safSi=A,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 18 16:42:36 PM_PL-4 osafamfnd[2659]: NO Assigning 'safSi=B,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 18 16:42:36 PM_PL-4 amf_demo[2765]: CSI Set - HAState Active for all
assigned CSIs
Mar 18 16:42:36 PM_PL-4 amf_demo[2773]: CSI Set - HAState Active for all
assigned CSIs
Mar 18 16:42:36 PM_PL-4 osafamfnd[2659]: NO Assigned 'safSi=A,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 18 16:42:36 PM_PL-4 osafamfnd[2659]: NO Assigned 'safSi=B,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 18 16:42:36 PM_PL-4 osafamfnd[2659]: NO Assigned 'all SIs' ACTIVE of
'safSu=ABC2,safSg=2N,safApp=Test'
4. Now, respond to quisced cbk, SU1 gets Quisced and Remove. This is Ok.
TC #2: Configuration and logs(Admin Op TC 2) is attached in the ticket.
Same as TC #1 except that don't respond to Quisced callback from gdb and let it
Timeout.
Same problem as in TC #1 happens. SU1 is terminated:
Mar 18 17:14:44 PM_PL-3 osafamfnd[18888]: NO saAmfSUFailover is true for
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:14:44 PM_PL-3 osafamfnd[18888]: NO SU failover probation timer
started (timeout: 1200000000000 ns)
Mar 18 17:14:44 PM_PL-3 osafamfnd[18888]: NO Performing failover of
'safSu=ABC1,safSg=2N,safApp=Test' (SU failover count: 1)
Mar 18 17:14:44 PM_PL-3 osafamfnd[18888]: NO
'safComp=A1,safSu=ABC1,safSg=2N,safApp=Test' recovery action escalated from
'componentFailover' to 'suFailover'
Mar 18 17:14:44 PM_PL-3 osafamfnd[18888]: NO
'safComp=A1,safSu=ABC1,safSg=2N,safApp=Test' faulted due to
'csiSetcallbackTimeout' : Recovery is 'suFailover'
Mar 18 17:14:44 PM_PL-3 osafamfnd[18888]: NO Terminating components of
'safSu=ABC1,safSg=2N,safApp=Test'(abruptly & unordered)
Mar 18 17:14:44 PM_PL-3 osafamfnd[18888]: NO 'safSu=ABC1,safSg=2N,safApp=Test'
Presence State INSTANTIATED => TERMINATING
Mar 18 17:14:44 PM_PL-3 osafamfnd[18888]: NO 'safSu=ABC1,safSg=2N,safApp=Test'
Presence State TERMINATING => TERMINATING
Mar 18 17:14:44 PM_PL-3 osafamfnd[18888]: NO 'safSu=ABC1,safSg=2N,safApp=Test'
Presence State TERMINATING => TERMINATING
Mar 18 17:14:44 PM_PL-3 logger: killing
safComp=A1,safSu=ABC1,safSg=2N,safApp=Test : PID : 18983
Mar 18 17:14:44 PM_PL-3 logger: killing
safComp=B1,safSu=ABC1,safSg=2N,safApp=Test : PID : 18991
Mar 18 17:14:44 PM_PL-3 osafamfnd[18888]: NO 'safSu=ABC1,safSg=2N,safApp=Test'
Presence State TERMINATING => UNINSTANTIATED
Mar 18 17:14:44 PM_PL-3 osafamfnd[18888]: NO Terminated all components in
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:14:44 PM_PL-3 osafamfnd[18888]: NO Informing director of sufailover
Then unlock SU1 and repair SU1, SU1 gets Standby assignment and then gets
Remove assignment( why remove assignment?), which is wrong.:
Mar 18 17:15:19 PM_PL-3 osafamfnd[18888]: NO Repair request for
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:15:19 PM_PL-3 osafamfnd[18888]: NO 'safSu=ABC1,safSg=2N,safApp=Test'
Presence State UNINSTANTIATED => UNINSTANTIATED
Mar 18 17:15:19 PM_PL-3 osafamfnd[18888]: NO 'safSu=ABC1,safSg=2N,safApp=Test'
Presence State UNINSTANTIATED => INSTANTIATING
Mar 18 17:15:19 PM_PL-3 amf_demo[19100]:
'safComp=A1,safSu=ABC1,safSg=2N,safApp=Test' started
Mar 18 17:15:19 PM_PL-3 amf_demo[19100]: Registered with AMF
Mar 18 17:15:19 PM_PL-3 amf_demo[19108]:
'safComp=B1,safSu=ABC1,safSg=2N,safApp=Test' started
Mar 18 17:15:19 PM_PL-3 osafamfnd[18888]: NO 'safSu=ABC1,safSg=2N,safApp=Test'
Presence State INSTANTIATING => INSTANTIATED
Mar 18 17:15:19 PM_PL-3 amf_demo[19108]: Registered with AMF
Mar 18 17:15:19 PM_PL-3 osafamfnd[18888]: NO Assigning 'safSi=A,safApp=Test'
STANDBY to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:15:19 PM_PL-3 osafamfnd[18888]: NO Assigning 'safSi=B,safApp=Test'
STANDBY to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:15:21 PM_PL-3 amf_demo[19100]: CSI Set - add
'safCsi=A1,safSi=A,safApp=Test' HAState Standby
Mar 18 17:15:21 PM_PL-3 osafamfnd[18888]: NO Assigned 'safSi=A,safApp=Test'
STANDBY to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:15:21 PM_PL-3 amf_demo[19108]: CSI Set - add
'safCsi=B1,safSi=B,safApp=Test' HAState Standby
Mar 18 17:15:21 PM_PL-3 osafamfnd[18888]: NO Assigned 'safSi=B,safApp=Test'
STANDBY to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:15:21 PM_PL-3 osafamfnd[18888]: NO Removing 'all (2) SIs' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:15:21 PM_PL-3 osafamfnd[18888]: NO Removing 'safSi=A,safApp=Test'
from 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:15:21 PM_PL-3 osafamfnd[18888]: NO Removing 'safSi=B,safApp=Test'
from 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:15:21 PM_PL-3 amf_demo[19100]: CSI Remove for all CSIs
Mar 18 17:15:21 PM_PL-3 amf_demo[19108]: CSI Remove for all CSIs
Mar 18 17:15:21 PM_PL-3 osafamfnd[18888]: NO Removed 'safSi=A,safApp=Test' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:15:21 PM_PL-3 osafamfnd[18888]: NO Removed 'safSi=B,safApp=Test' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:15:21 PM_PL-3 osafamfnd[18888]: NO Removed 'all SIs' from
'safSu=ABC1,safSg=2N,safApp=Test'
TC #3 : Repeat the TC #2 with saAmfSutDefSUFailover as false.
Extra SUSI Remove(ER susi_assign_evh: 'safSu=ABC1,safSg=2N,safApp=Test' has no
assignments) is coming to PL-3:
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO SU failover probation timer
started (timeout: 1200000000000 ns)
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO Performing failover of
'safSu=ABC1,safSg=2N,safApp=Test' (SU failover count: 1)
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO
'safComp=A1,safSu=ABC1,safSg=2N,safApp=Test' faulted due to
'csiSetcallbackTimeout' : Recovery is 'componentFailover'
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO 'safSu=ABC1,safSg=2N,safApp=Test'
Presence State INSTANTIATED => TERMINATING
Mar 18 17:28:21 PM_PL-3 logger: killing
safComp=A1,safSu=ABC1,safSg=2N,safApp=Test : PID : 19917
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO Assigned 'safSi=A,safApp=Test'
QUIESCED to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO Assigned 'safSi=B,safApp=Test'
QUIESCED to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO Assigned 'all SIs' QUIESCED of
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO Removing 'all (2) SIs' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO Removing 'safSi=A,safApp=Test'
from 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO Removing 'safSi=B,safApp=Test'
from 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:28:21 PM_PL-3 amf_demo[19925]: CSI Remove for all CSIs
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO Removed 'safSi=A,safApp=Test' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO Removed 'safSi=B,safApp=Test' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO Removed 'all SIs' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: ER susi_assign_evh:
'safSu=ABC1,safSg=2N,safApp=Test' has no assignments
Mar 18 17:28:21 PM_PL-3 amf_demo[19996]:
'safComp=A1,safSu=ABC1,safSg=2N,safApp=Test' started
Mar 18 17:28:21 PM_PL-3 osafamfnd[19819]: NO 'safSu=ABC1,safSg=2N,safApp=Test'
Presence State TERMINATING => INSTANTIATED
Mar 18 17:28:21 PM_PL-3 amf_demo[19996]: Registered with AMF
TC #4: Repeat TC #1 till step 2. In step 3, after controller comes up and
before Cluster Timer expiry, stop PL-3.
The SUSI assignments are:
safSISU=safSu=ABC2\,safSg=2N\,safApp=Test,safSi=B,safApp=Test
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=ABC1\,safSg=2N\,safApp=Test,safSi=B,safApp=Test
saAmfSISUHAState=QUIESCED(3)
safSISU=safSu=ABC1\,safSg=2N\,safApp=Test,safSi=A,safApp=Test
saAmfSISUHAState=QUIESCED(3)
safSISU=safSu=ABC2\,safSg=2N\,safApp=Test,safSi=A,safApp=Test
saAmfSISUHAState=ACTIVE(1)
Unlock SU2 gives following error:
amf-adm unlock safSu=ABC1,safSg=2N,safApp=Test
Mar 18 17:55:21.712332 osafamfd [5505:imm.cc:1998] >> report_admin_op_error:
inv:60129542145, res:6, Error String: 'SG state is not stable'
Start PL-3 and unlock SU2 again, but the SUSI state remain as above.
TC #5: Configuration and logs(Admin Op TC 5) is attached in the ticket.
Run TC #1 till step 1.
2. Now, Issue lock on SU1(say) (safSu=ABC1,safSg=2N,safApp=Test). Keep gdb in
one of the component in Active cbk (amf_csi_set_callback) of SU2 component on
PL-4.
SU1 goes to Quiesced, SU2 goes to Act and stuck in gdb. Stop controller and
start controller.
3. After controller comes up and after Cluster Timer expiry, Amf gives remove
to SU1, which is wrong, because SU2 still hasn't responded.
Mar 18 18:21:39 PM_PL-3 osafamfnd[25307]: NO Removing 'all (2) SIs' from
'safSu=ABC1,safSg=2N,safApp=Test'
After SU2 comp responds to Act assignment, the following error prints comes:
Mar 18 18:21:51 PM_SC-1 osafamfd[8652]: EM sg_2n_fsm.cc:2359:
safSu=ABC2,safSg=2N,safApp=Test (31)
TC #6: Configuration and logs(Admin Op TC 6) is attached in the ticket.
Repeat TC #5 till step 3 before responding to Act cbk. Then issue unlock op on
SU1, SU1 gets Standby assignments, which is wrong, because SU2 is still not
responded, admin op has to be rejected.
amf-adm unlock safSu=ABC1,safSg=2N,safApp=Test
Mar 18 18:39:53 PM_PL-3 osafamfnd[26910]: NO Assigning 'safSi=A,safApp=Test'
STANDBY to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 18:39:53 PM_PL-3 osafamfnd[26910]: NO Assigning 'safSi=B,safApp=Test'
STANDBY to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 18:39:55 PM_PL-3 amf_demo[26999]: CSI Set - add
'safCsi=A1,safSi=A,safApp=Test' HAState Standby
Mar 18 18:39:55 PM_PL-3 osafamfnd[26910]: NO Assigned 'safSi=A,safApp=Test'
STANDBY to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 18 18:39:55 PM_PL-3 amf_demo[27007]: CSI Set - add
'safCsi=B1,safSi=B,safApp=Test' HAState Standby
Mar 18 18:39:55 PM_PL-3 osafamfnd[26910]: NO Assigned 'safSi=B,safApp=Test'
STANDBY to 'safSu=ABC1,safSg=2N,safApp=Test'
TC #7: Repeat TC #5 till step 2. After controller comes up and before Cluster
Timer expiry, stop PL-4, the following susi exists, which is wrong:
safSISU=safSu=ABC2\,safSg=2N\,safApp=Test,safSi=B,safApp=Test
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=ABC2\,safSg=2N\,safApp=Test,safSi=A,safApp=Test
saAmfSISUHAState=ACTIVE(1)
Also, unlock SU1 doesn't result in any assignment.
TC #8: Same as Test Case #2. Instead, keep gdb in remove cbk on SU1. When SU1
is repaired and unlocked, the Standby assignment is given and then remove
assignments (Why remove assignment ?).
Mar 21 12:07:42 PM_PL-3 osafamfnd[6473]: NO Repair request for
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 21 12:07:42 PM_PL-3 osafamfnd[6473]: NO 'safSu=ABC1,safSg=2N,safApp=Test'
Presence State UNINSTANTIATED => UNINSTANTIATED
Mar 21 12:07:42 PM_PL-3 osafamfnd[6473]: NO 'safSu=ABC1,safSg=2N,safApp=Test'
Presence State UNINSTANTIATED => INSTANTIATING
Mar 21 12:07:42 PM_PL-3 amf_demo[6659]:
'safComp=A1,safSu=ABC1,safSg=2N,safApp=Test' started
Mar 21 12:07:42 PM_PL-3 amf_demo[6659]: Registered with AMF
Mar 21 12:07:42 PM_PL-3 amf_demo[6667]:
'safComp=B1,safSu=ABC1,safSg=2N,safApp=Test' started
Mar 21 12:07:42 PM_PL-3 osafamfnd[6473]: NO 'safSu=ABC1,safSg=2N,safApp=Test'
Presence State INSTANTIATING => INSTANTIATED
Mar 21 12:07:42 PM_PL-3 amf_demo[6667]: Registered with AMF
Mar 21 12:07:47 PM_PL-3 osafamfnd[6473]: NO Assigning 'safSi=A,safApp=Test'
STANDBY to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 21 12:07:47 PM_PL-3 osafamfnd[6473]: NO Assigning 'safSi=B,safApp=Test'
STANDBY to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 21 12:07:49 PM_PL-3 amf_demo[6659]: CSI Set - add
'safCsi=A1,safSi=A,safApp=Test' HAState Standby
Mar 21 12:07:49 PM_PL-3 osafamfnd[6473]: NO Assigned 'safSi=A,safApp=Test'
STANDBY to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 21 12:07:49 PM_PL-3 amf_demo[6667]: CSI Set - add
'safCsi=B1,safSi=B,safApp=Test' HAState Standby
Mar 21 12:07:49 PM_PL-3 osafamfnd[6473]: NO Assigned 'safSi=B,safApp=Test'
STANDBY to 'safSu=ABC1,safSg=2N,safApp=Test'
Mar 21 12:07:49 PM_PL-3 osafamfnd[6473]: NO Removing 'all (2) SIs' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 21 12:07:49 PM_PL-3 osafamfnd[6473]: NO Removing 'safSi=A,safApp=Test' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 21 12:07:49 PM_PL-3 osafamfnd[6473]: NO Removing 'safSi=B,safApp=Test' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 21 12:07:49 PM_PL-3 amf_demo[6667]: CSI Remove for all CSIs
Mar 21 12:07:49 PM_PL-3 amf_demo[6659]: CSI Remove for all CSIs
Mar 21 12:07:49 PM_PL-3 osafamfnd[6473]: NO Removed 'safSi=A,safApp=Test' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 21 12:07:49 PM_PL-3 osafamfnd[6473]: NO Removed 'safSi=B,safApp=Test' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 21 12:07:49 PM_PL-3 osafamfnd[6473]: NO Removed 'all SIs' from
'safSu=ABC1,safSg=2N,safApp=Test'
TC #9: Configure Spare SU3 on PL-4. Configuration is the same as TC #1.
Keep gdb in SU2 comp for amf_csi_set_callback.
Lock SU1, SU1 goes to Quisced and then Su2 is given Act, it is stuck in gdb.
Now restart the controller.
When controller comes up, it gives Remove assignment to SU1 and then Standby
assignment to SU3, which is wrong as SU1
is yet to respond for Act assignment.
PL-3 logs after controller comes back:
Mar 21 12:32:09 PM_PL-3 amf_demo[9657]: CSI Remove for all CSIs
Mar 21 12:32:09 PM_PL-3 osafamfnd[9560]: NO Removed 'safSi=A,safApp=Test' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 21 12:32:09 PM_PL-3 osafamfnd[9560]: NO Removed 'safSi=B,safApp=Test' from
'safSu=ABC1,safSg=2N,safApp=Test'
Mar 21 12:32:09 PM_PL-3 osafamfnd[9560]: NO Removed 'all SIs' from
'safSu=ABC1,safSg=2N,safApp=Test'
PL-4 logs after controller comes back:
PM_PL-4:/opt/amf_demo # Mar 21 12:32:09 PM_PL-4 osafamfnd[27583]: NO Assigning
'safSi=A,safApp=Test' STANDBY to 'safSu=ABC3,safSg=2N,safApp=Test'
Mar 21 12:32:09 PM_PL-4 osafamfnd[27583]: NO Assigning 'safSi=B,safApp=Test'
STANDBY to 'safSu=ABC3,safSg=2N,safApp=Test'
Mar 21 12:32:09 PM_PL-4 amf_demo[27680]: CSI Set - add
'safCsi=A1,safSi=A,safApp=Test' HAState Standby
Mar 21 12:32:09 PM_PL-4 osafamfnd[27583]: NO Assigned 'safSi=A,safApp=Test'
STANDBY to 'safSu=ABC3,safSg=2N,safApp=Test'
Mar 21 12:32:09 PM_PL-4 amf_demo[27688]: CSI Set - add
'safCsi=B1,safSi=B,safApp=Test' HAState Standby
Mar 21 12:32:09 PM_PL-4 osafamfnd[27583]: NO Assigned 'safSi=B,safApp=Test'
STANDBY to 'safSu=ABC3,safSg=2N,safApp=Test'
TC #10: Configuration same as TC #9. Logs attached in the ticket (Admin Op TC
10).
Keep gdb in SU1 comp for amf_csi_set_callback. Lock SU1, SU1 goes to Quisced
and it is stuck in gdb. Now stop the controller, responds from gdb, start
controller.
When controller comes up and before Cluster Timer expiry, stop PL-4.
The SUSI assignments are:
safSISU=safSu=ABC2\,safSg=2N\,safApp=Test,safSi=B,safApp=Test
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=ABC2\,safSg=2N\,safApp=Test,safSi=A,safApp=Test
saAmfSISUHAState=STANDBY(2)
The unlock of SU1 fails with " SG state is not stable".
Start PL-4, there is no assignment given to SU2 and SU3.
TC #11: Configuration same as TC #9. Logs attached in the ticket (Admin Op TC
11).
Keep gdb in SU1 comp for amf_csi_set_callback. Lock SU1, SU1 goes to Quisced
and it is stuck in gdb. Now stop the controller, responds from gdb, start
controller.
When controller comes up and before Cluster Timer expiry, stop and start PL-4:
Amfnd logs below. It looks Amfd is sending SUSI remove to Amfnd when it comes
up, which is wrong:
Mar 21 14:38:58 PM_PL-4 osafamfnd[6810]: ER susi_assign_evh:
'safSu=ABC2,safSg=2N,safApp=Test' has no assignments
The SUSI and SG fsm states(Unstable) are in the same state as TC #10.
TC #12: Configuration same as TC #9.
1. Start SC-1 and PL-3 and PL-4. 2N SG, SU1(safSu=ABC1,safSg=2N,safApp=Test)
hosted on PL-3 and SU2(safSu=ABC2,safSg=2N,safApp=Test) hosted on PL-4. SU
having two components, Two SIs(each SI has one CSI). One CSI is on one Comp
i.e. Maps like : One SI->one CSI-> One Comp.
immcfg -f /opt/amf_demo/2N_2comps.xml
amf-adm unlock-in safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock-in safSu=ABC2,safSg=2N,safApp=Test
2. Now, Issue unlock on SU1(say) (safSu=ABC1,safSg=2N,safApp=Test). Keep gdb in
one of the component in Act cbk (amf_csi_set_callback). Another components
responds to Ack cbk. Now, stop controller and start it.
3. After controller comes up and before Cluster Timer Expiry, issue unlock SU2.
Allowing unlock before cluster timer expiry is itself wrong(But, the same thing
I simulated when PL-4 comes with SU2 unlocked on it, SU2 is getting Standby
assignment).
After Cluster expiry, SU2 is given Standby assignment, which is wrong because
SU1 is still didn't respond to Act.
TC #13: Configuration same as TC #12 except saAmfSutDefSUFailover as false:
Repeat TC #12 till step 2. Logs (Admin Op TC 13) attached in the ticket.
After controller comes up and before Cluster Timer Expiry, issue unlock SU2 and
lock of SU1.
After cluster timer expiry, kill comp1(in gdb) in SU1, assignment is not given
to SU2.
SG becomes unstable and unlock of SU1 fails.
TC #14: Same configuration as TC #12:
immcfg -f /opt/amf_demo/2N_2comps.xml
amf-adm unlock-in safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock-in safSu=ABC2,safSg=2N,safApp=Test
Now, SU1 has Act assignment on PL-3.
Unlock SU2 and keep gdb in Standby assignment in comp1 of SU2, reboot the
controller.
When controller comes up and before Cluster Timer expiry, stop PL-3.
Now, respond from gdb from Standby assignment. SU1 SUSI is still there and
there is no failover:
safSISU=safSu=ABC2\,safSg=2N\,safApp=Test,safSi=A,safApp=Test
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=ABC2\,safSg=2N\,safApp=Test,safSi=B,safApp=Test
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=ABC1\,safSg=2N\,safApp=Test,safSi=B,safApp=Test
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=ABC1\,safSg=2N\,safApp=Test,safSi=A,safApp=Test
saAmfSISUHAState=ACTIVE(1)
TC #15: Same configuration as TC #12: Logs (Admin Op TC 15)
immcfg -f /opt/amf_demo/2N_2comps.xml
amf-adm unlock-in safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock-in safSu=ABC2,safSg=2N,safApp=Test
Now, SU1 has Act assignment on PL-3.
Unlock SU2 and keep gdb in Standby assignment in comp1 of SU2, stop PL-3.
Restart controller, when controller comes up, allow Standby assignment to
Timeout.
After csiset timeout, Active and Quiesced assignments on SU2 are going at the
same time(which is wrong):
Mar 22 13:52:50 PM_PL-4 osafamfnd[4576]: NO SU failover probation timer started
(timeout: 1200000000000 ns)
Mar 22 13:52:50 PM_PL-4 osafamfnd[4576]: NO Performing failover of
'safSu=ABC2,safSg=2N,safApp=Test' (SU failover count: 1)
Mar 22 13:52:50 PM_PL-4 osafamfnd[4576]: NO
'safComp=A1,safSu=ABC2,safSg=2N,safApp=Test' faulted due to
'csiSetcallbackTimeout' : Recovery is 'componentFailover'
Mar 22 13:52:50 PM_PL-4 osafamfnd[4576]: NO 'safSu=ABC2,safSg=2N,safApp=Test'
Presence State INSTANTIATED => TERMINATING
Mar 22 13:52:50 PM_PL-4 logger: killing
safComp=A1,safSu=ABC2,safSg=2N,safApp=Test : PID : 4703
Mar 22 13:52:50 PM_PL-4 osafamfnd[4576]: NO Assigned 'safSi=A,safApp=Test'
STANDBY to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 13:52:50 PM_PL-4 osafamfnd[4576]: NO Assigning 'all (2) SIs' ACTIVE to
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 13:52:50 PM_PL-4 osafamfnd[4576]: NO Assigning 'safSi=A,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 13:52:50 PM_PL-4 osafamfnd[4576]: NO Assigning 'safSi=B,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 13:52:50 PM_PL-4 osafamfnd[4576]: NO Assigning 'all (2) SIs' QUIESCED to
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 13:52:50 PM_PL-4 osafamfnd[4576]: NO Assigning 'safSi=A,safApp=Test'
QUIESCED to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 13:52:50 PM_PL-4 osafamfnd[4576]: NO Assigning 'safSi=B,safApp=Test'
QUIESCED to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 13:52:52 PM_PL-4 amf_demo[4711]: CSI Set - HAState Active for all
assigned CSIs
Mar 22 13:52:52 PM_PL-4 amf_demo[4711]: saAmfHAStateGet FAILED - 7
Mar 22 13:52:54 PM_PL-4 amf_demo[4711]: CSI Set - HAState Quiesced for all
assigned CSIs
Mar 22 13:52:54 PM_PL-4 osafamfnd[4576]: NO Assigned 'safSi=A,safApp=Test'
QUIESCED to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 13:52:54 PM_PL-4 osafamfnd[4576]: NO Assigned 'safSi=B,safApp=Test'
QUIESCED to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 13:52:54 PM_PL-4 osafamfnd[4576]: NO Assigned 'all SIs' QUIESCED of
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 13:52:54 PM_PL-4 amf_demo[4711]: saAmfHAStateGet FAILED - 7
Mar 22 13:52:54 PM_PL-4 osafamfnd[4576]: NO Removing 'all (2) SIs' from
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 13:52:54 PM_PL-4 osafamfnd[4576]: NO Removing 'safSi=A,safApp=Test' from
'safSu=ABC2,safSg=2N,safApp=Test'
TC #16: Same configuration as TC #12: Logs (Admin Op TC 16)
immcfg -f /opt/amf_demo/2N_2comps.xml
amf-adm unlock-in safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock-in safSu=ABC2,safSg=2N,safApp=Test
Now, SU1 has Act assignment on PL-3.
Unlock SU2 and keep gdb in Standby assignment in comp1 of SU2.
Stop controller and then stop PL-3. Start controller, when controller comes up,
allow Standby assignment to Timeout.
After csiset timeout, Active and Quiesced assignments on SU2 are going at the
same time(which is wrong). The same as above:
Mar 22 14:22:08 PM_PL-4 osafamfnd[5573]: NO SU failover probation timer started
(timeout: 1200000000000 ns)
Mar 22 14:22:08 PM_PL-4 osafamfnd[5573]: NO Performing failover of
'safSu=ABC2,safSg=2N,safApp=Test' (SU failover count: 1)
Mar 22 14:22:08 PM_PL-4 osafamfnd[5573]: NO
'safComp=A1,safSu=ABC2,safSg=2N,safApp=Test' faulted due to
'csiSetcallbackTimeout' : Recovery is 'componentFailover'
Mar 22 14:22:08 PM_PL-4 osafamfnd[5573]: NO 'safSu=ABC2,safSg=2N,safApp=Test'
Presence State INSTANTIATED => TERMINATING
Mar 22 14:22:08 PM_PL-4 logger: killing
safComp=A1,safSu=ABC2,safSg=2N,safApp=Test : PID : 5654
Mar 22 14:22:08 PM_PL-4 osafamfnd[5573]: NO Assigned 'safSi=A,safApp=Test'
STANDBY to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:08 PM_PL-4 osafamfnd[5573]: NO Assigning 'all (2) SIs' ACTIVE to
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:08 PM_PL-4 osafamfnd[5573]: NO Assigning 'safSi=A,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:08 PM_PL-4 osafamfnd[5573]: NO Assigning 'safSi=B,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:08 PM_PL-4 osafamfnd[5573]: NO Assigning 'all (2) SIs' QUIESCED to
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:08 PM_PL-4 osafamfnd[5573]: NO Assigning 'safSi=A,safApp=Test'
QUIESCED to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:08 PM_PL-4 osafamfnd[5573]: NO Assigning 'safSi=B,safApp=Test'
QUIESCED to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:10 PM_PL-4 amf_demo[5662]: CSI Set - HAState Active for all
assigned CSIs
Mar 22 14:22:10 PM_PL-4 amf_demo[5662]: saAmfHAStateGet FAILED - 7
Mar 22 14:22:12 PM_PL-4 amf_demo[5662]: CSI Set - HAState Quiesced for all
assigned CSIs
Mar 22 14:22:12 PM_PL-4 osafamfnd[5573]: NO Assigned 'safSi=A,safApp=Test'
QUIESCED to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:12 PM_PL-4 osafamfnd[5573]: NO Assigned 'safSi=B,safApp=Test'
QUIESCED to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:12 PM_PL-4 osafamfnd[5573]: NO Assigned 'all SIs' QUIESCED of
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:12 PM_PL-4 amf_demo[5662]: saAmfHAStateGet FAILED - 7
Mar 22 14:22:12 PM_PL-4 osafamfnd[5573]: NO Removing 'all (2) SIs' from
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:12 PM_PL-4 osafamfnd[5573]: NO Removing 'safSi=A,safApp=Test' from
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:12 PM_PL-4 osafamfnd[5573]: NO Removing 'safSi=B,safApp=Test' from
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:22:12 PM_PL-4 amf_demo[5662]: CSI Remove for all CSIs
TC #17: Same configuration as TC #12: Logs (Admin Op TC 17)
immcfg -f /opt/amf_demo/2N_2comps.xml
amf-adm unlock-in safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock-in safSu=ABC2,safSg=2N,safApp=Test
Now, SU1 has Act assignment on PL-3.
Unlock SU2 and keep gdb in Standby assignment in comp1 of SU2, stop PL-3.
Restart controller, when controller comes up, PL-4 reboots:
Mar 22 14:36:31 PM_PL-4 osafamfnd[6268]: WA AMF director unexpectedly crashed
Mar 22 14:36:31 PM_PL-4 osafamfnd[6268]: NO Checking
'safSu=ABC2,safSg=2N,safApp=Test' for pending messages
Mar 22 14:36:31 PM_PL-4 osafamfnd[6268]: ER Pending messages from director.
Rebooting to re-sync.
Mar 22 14:36:31 PM_PL-4 osafamfnd[6268]: Rebooting OpenSAF NodeId = 132111 EE
Name = , Reason: local AVD down(Adest) or both AVD down(Vdest) received,
OwnNodeId = 132111, SupervisionTime = 60
This is normal and obvious case and since Amfd is knowing that it has sent Act
assignment to PL-4 when PL-3 was stopped.
This is similar to TC #16. Looks just timing issue.
TC #18: Same configuration as TC #12 except saAmfSutDefSUFailover as true: Logs
and configuration attached (Admin Op TC 18) in the ticket.
immcfg -f /opt/amf_demo/2N_2comps.xml
amf-adm unlock-in safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock-in safSu=ABC2,safSg=2N,safApp=Test
Now, SU1 has Act assignment on PL-3.
Unlock SU2 and keep gdb in Standby assignment in comp1 of SU2.
Stop controller and then stop PL-3. Start controller, when controller comes up,
allow Standby assignment to Timeout.
After csiset timeout, SUfailover happens, so issue repair admin on SU2. Two
times Act assignments are being given to SU2, one new Act and another modified
to Act(Giving two times Act is wrong, also giving modification from Act to Act
is wrong):
Mar 22 14:51:42 PM_PL-4 osafamfnd[7643]: NO saAmfSUFailover is true for
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:51:42 PM_PL-4 osafamfnd[7643]: NO SU failover probation timer started
(timeout: 1200000000000 ns)
Mar 22 14:51:42 PM_PL-4 osafamfnd[7643]: NO Performing failover of
'safSu=ABC2,safSg=2N,safApp=Test' (SU failover count: 1)
Mar 22 14:51:42 PM_PL-4 osafamfnd[7643]: NO
'safComp=A1,safSu=ABC2,safSg=2N,safApp=Test' recovery action escalated from
'componentFailover' to 'suFailover'
Mar 22 14:51:42 PM_PL-4 osafamfnd[7643]: NO
'safComp=A1,safSu=ABC2,safSg=2N,safApp=Test' faulted due to
'csiSetcallbackTimeout' : Recovery is 'suFailover'
Mar 22 14:51:42 PM_PL-4 osafamfnd[7643]: NO Terminating components of
'safSu=ABC2,safSg=2N,safApp=Test'(abruptly & unordered)
Mar 22 14:51:42 PM_PL-4 osafamfnd[7643]: NO 'safSu=ABC2,safSg=2N,safApp=Test'
Presence State INSTANTIATED => TERMINATING
Mar 22 14:51:42 PM_PL-4 osafamfnd[7643]: NO 'safSu=ABC2,safSg=2N,safApp=Test'
Presence State TERMINATING => TERMINATING
Mar 22 14:51:42 PM_PL-4 osafamfnd[7643]: NO 'safSu=ABC2,safSg=2N,safApp=Test'
Presence State TERMINATING => TERMINATING
Mar 22 14:51:42 PM_PL-4 logger: killing
safComp=B1,safSu=ABC2,safSg=2N,safApp=Test : PID : 7728
Mar 22 14:51:42 PM_PL-4 logger: killing
safComp=A1,safSu=ABC2,safSg=2N,safApp=Test : PID : 7720
Mar 22 14:51:42 PM_PL-4 osafamfnd[7643]: NO 'safSu=ABC2,safSg=2N,safApp=Test'
Presence State TERMINATING => UNINSTANTIATED
Mar 22 14:51:42 PM_PL-4 osafamfnd[7643]: NO Terminated all components in
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:51:42 PM_PL-4 osafamfnd[7643]: NO Informing director of sufailover
Mar 22 14:51:49 PM_PL-4 osafamfnd[7643]: NO Repair request for
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:51:49 PM_PL-4 osafamfnd[7643]: NO 'safSu=ABC2,safSg=2N,safApp=Test'
Presence State UNINSTANTIATED => UNINSTANTIATED
Mar 22 14:51:49 PM_PL-4 osafamfnd[7643]: NO 'safSu=ABC2,safSg=2N,safApp=Test'
Presence State UNINSTANTIATED => INSTANTIATING
Mar 22 14:51:49 PM_PL-4 amf_demo[7809]:
'safComp=A1,safSu=ABC2,safSg=2N,safApp=Test' started
Mar 22 14:51:49 PM_PL-4 amf_demo[7809]: Registered with AMF
Mar 22 14:51:49 PM_PL-4 amf_demo[7817]:
'safComp=B1,safSu=ABC2,safSg=2N,safApp=Test' started
Mar 22 14:51:49 PM_PL-4 osafamfnd[7643]: NO 'safSu=ABC2,safSg=2N,safApp=Test'
Presence State INSTANTIATING => INSTANTIATED
Mar 22 14:51:49 PM_PL-4 amf_demo[7817]: Registered with AMF
Mar 22 14:51:49 PM_PL-4 osafamfnd[7643]: NO Assigning 'safSi=A,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:51:49 PM_PL-4 osafamfnd[7643]: NO Assigning 'safSi=B,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:51:51 PM_PL-4 amf_demo[7809]: CSI Set - add
'safCsi=A1,safSi=A,safApp=Test' HAState Active
Mar 22 14:51:51 PM_PL-4 osafamfnd[7643]: NO Assigned 'safSi=A,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:51:51 PM_PL-4 amf_demo[7817]: CSI Set - add
'safCsi=B1,safSi=B,safApp=Test' HAState Active
Mar 22 14:51:51 PM_PL-4 osafamfnd[7643]: NO Assigned 'safSi=B,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:51:51 PM_PL-4 osafamfnd[7643]: NO Assigning 'all (2) SIs' ACTIVE to
'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:51:51 PM_PL-4 osafamfnd[7643]: NO Assigning 'safSi=A,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:51:51 PM_PL-4 osafamfnd[7643]: NO Assigning 'safSi=B,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:51:53 PM_PL-4 osafamfnd[7643]: NO Assigned 'safSi=A,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:51:53 PM_PL-4 osafamfnd[7643]: NO Assigned 'safSi=B,safApp=Test'
ACTIVE to 'safSu=ABC2,safSg=2N,safApp=Test'
Mar 22 14:51:53 PM_PL-4 osafamfnd[7643]: NO Assigned 'all SIs' ACTIVE of
'safSu=ABC2,safSg=2N,safApp=Test'
TC #19: Configuration and logs attached (Admin Op TC 19).
immcfg -f /opt/amf_demo/2N_2comps.xml
amf-adm unlock-in safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock-in safSu=ABC2,safSg=2N,safApp=Test
Now, SU1 has Act assignment on PL-3.
Unlock SU2 and keep gdb in Standby assignment in comp1 of SU2.
Restart controller and before Cluster Timer Expiry:
Kill comp 1 of SU1:
PM_PL-3:~ # ps -e | grep demo
29214 ? 00:00:00 amf_demo
29222 ? 00:00:00 amf_demo
PM_PL-3:~ # kill -9 29222
Amfd crashes with the following logs:
Mar 22 18:39:44 PM_SC-1 osafamfd[16724]: sg_2n_fsm.cc:534: avd_sg_2n_act_susi:
Assertion 'a_susi_1->su == a_susi_2->su' failed.
Mar 22 18:39:44 PM_SC-1 osafamfnd[16734]: WA AMF director unexpectedly crashed
Bt:
Program terminated with signal 6, Aborted.
#0 0x00007f839a2b2b55 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x00007f839a2b2b55 in raise () from /lib64/libc.so.6
#1 0x00007f839a2b4131 in abort () from /lib64/libc.so.6
#2 0x00007f839c08938b in __osafassert_fail ()
from /usr/local/lib/libopensaf_core.so.0
#3 0x000000000044b2e1 in avd_sg_2n_act_susi () at sg_2n_fsm.cc:534
#4 0x000000000044f0d7 in SG_2N::susi_success_sg_realign(AVD_SU*,
avd_su_si_rel_ tag*,
AVSV_SUSI_ACT, SaAmfHAStateT) () at sg_2n_fsm.cc:1663
#5 0x000000000044ff5c in SG_2N::susi_success(cl_cb_tag*, AVD_SU*,
avd_su_si_rel _tag*,
AVSV_SUSI_ACT, SaAmfHAStateT) () at sg_2n_fsm.cc:2362
#6 0x000000000046ee3e in avd_su_si_assign_evh(cl_cb_tag*, avd_evt_tag*) ()
at sgproc.cc:1176
#7 0x0000000000433764 in process_event(cl_cb_tag*, avd_evt_tag*) ()
at main.cc:808
#8 0x0000000000407c7e in main () at main.cc:729
(gdb)
PL-4 reboots with the following logs:
Mar 22 18:38:36 PM_PL-4 osafamfnd[11778]: WA AMF director unexpectedly crashed
Mar 22 18:38:36 PM_PL-4 osafamfnd[11778]: NO Checking
'safSu=ABC2,safSg=2N,safApp=Test' for pending messages
Mar 22 18:38:36 PM_PL-4 osafamfnd[11778]: ER Pending messages from director.
Rebooting to re-sync.
Mar 22 18:38:36 PM_PL-4 osafamfnd[11778]: Rebooting OpenSAF NodeId = 132111 EE
Name = , Reason: local AVD down(Adest) or both AVD down(Vdest) received,
OwnNodeId = 132111, SupervisionTime = 60
Mar 22 18:38:36 PM_PL-4 opensaf_reboot: Rebooting local node; timeout=60
Syslogs of SC-1, PL-3 and PL-4 are attached.
TC #20: Configuration and logs (Admin Op TC 20) attached in the ticket.
immcfg -f /opt/amf_demo/2N_2comps.xml
amf-adm unlock-in safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock safSu=ABC1,safSg=2N,safApp=Test
amf-adm unlock-in safSu=ABC2,safSg=2N,safApp=Test
Now, SU1 has Act assignment on PL-3.
Unlock SU2 and keep gdb in Standby assignment in comp1 of SU2.
Restart controller and before cluster timer expiry, kill comp 1 of SU1:
kill -9 13149
And respond success from Standby cbk, but reject Act assignment by responding
SA_AIS_ERR_FAILED_OPERATION from Comp1 and Comp2.
SG goes into Unstable case and no assignments were given to SU1 or SU2.
The following errors came at SC-1 before Cluster Timer Expiry:
Mar 23 14:30:00 PM_SC-1 osafamfd[20123]: EM sg_2n_fsm.cc:1439:
safSu=ABC2,safSg=2N,safApp=Test (31)
Mar 23 14:30:00 PM_SC-1 osafamfd[20123]: EM sg_2n_fsm.cc:1439:
safSu=ABC2,safSg=2N,safApp=Test (31)
Thanks
-Nagu
> -----Original Message-----
> From: Nagendra Kumar
> Sent: 15 March 2016 14:26
> To: minh chau; hans.nordeb...@ericsson.com; gary....@dektech.com.au;
> Praveen Malviya
> Cc: opensaf-devel@lists.sourceforge.net
> Subject: Re: [devel] [PATCH 01 of 15] amfd: Add support for cloud resilience
> at common libs [#1620]
>
> Hi Minh,
>
> I am going to execute test cases with all the patches
> applied. I will
> share the test results once I am done.
>
>
>
> Can you please send all the patches (latest if any) in a tar file.
>
>
>
> Please note that few of the test cases executed on previous patch sets would
> be invalid because of 'immediate esclation' implemented in the latest
> patches(#1-#4).
>
>
>
> Thanks
>
> -Nagu
>
>
>
> From: minh chau [mailto:minh.c...@dektech.com.au]
> Sent: 15 March 2016 04:51
> To: Nagendra Kumar; hans.nordeb...@ericsson.com;
> gary....@dektech.com.au; Praveen Malviya
> Cc: opensaf-devel@lists.sourceforge.net
> Subject: Re: [PATCH 01 of 15] amfd: Add support for cloud resilience at
> common libs [#1620]
>
>
>
> Hi Nagu, Praveen
>
> Since #1-#4 have been acked, can you please push them?
> #5 and #11_2 allows comp/su failover during headless, so we may have to
> visit them later.
> However, the patches: #9 #10 #11_1 #12 #13 are bug fixes that does not
> relate to *delayed failover* and needed for #1-#4. Can you please have a
> look?
>
> Thanks,
> Minh
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140
_______________________________________________
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140
_______________________________________________
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel
