TC #46: Related to Alarms and Notification: Notification and alarms, configuration and logs(TC 46) attached in the ticket. 1. Start single controller and a payload. 2N Appl, SU1(Act) on PL-3 and SU2(Stb) on PL-4 as below: immcfg -f /opt/amf_demo/AppConfig-2N.xml_2 amf-adm unlock-in safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock-in safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1
2. Keep sleep and exit 1 in instantiation script as below: start|instantiate) #sleep 7 shift start $* sleep 3 exit 1 RETVAL=$? 3. Stop controller. 4. Kill demo. Demo will move into instantiation failed state. 5. Start controller and subscribe for all notifications (ntfsubscribe > /tmp/notification) on controller. 6. Now, SU2 will get Act assignment. In the notification, two extra notifications "state change" for Standby came for SU2, which is wrong because SU2 assignment didn't change: === Feb 17 11:04:22 - State Change === eventType = SA_NTF_OBJECT_STATE_CHANGE notificationObject = "safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1" notifyingObject = "safApp=safAmfService" notificationClassId = SA_NTF_VENDOR_ID_SAF.SA_SVC_AMF.110 (0x6e) additionalText = "The HA state of SI safSi=AmfDemo1,safApp=AmfDemo1 assigned to SU safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 changed" - additionalInfo: 0 - infoId = 2 infoType = 10 infoValue = "safSi=AmfDemo1,safApp=AmfDemo1" sourceIndicator = SA_NTF_OBJECT_OPERATION State ID = SA_AMF_HA_STATE Old State: New State: SA_AMF_HA_STANDBY === Feb 17 11:04:22 - State Change === eventType = SA_NTF_OBJECT_STATE_CHANGE notificationObject = "safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1" notifyingObject = "safApp=safAmfService" notificationClassId = SA_NTF_VENDOR_ID_SAF.SA_SVC_AMF.110 (0x6e) additionalText = "The HA state of SI safSi=AmfDemo,safApp=AmfDemo1 assigned to SU safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 changed" - additionalInfo: 0 - infoId = 2 infoType = 10 infoValue = "safSi=AmfDemo,safApp=AmfDemo1" sourceIndicator = SA_NTF_OBJECT_OPERATION State ID = SA_AMF_HA_STATE Old State: New State: SA_AMF_HA_STANDBY TC #47: Related to Alarms and Notification: Notification and alarms, configuration and logs(TC 47) attached in the ticket. 1. Start single controller and a payload. 2N Appl, SU1(Act) on PL-3 and SU2(Stb) on PL-4 as below: immcfg -f /opt/amf_demo/AppConfig-2N.xml_2 amf-adm unlock-in safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock-in safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 2. Keep gdb in csi set of Act component and Lock SU (amf-adm lock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1). 3. Stop controller and then respond from gdb. 4. Start controller and subscribe for all notifications(ntfsubscribe > /tmp/notification_after_reboot). 5. The notification should be in continuation of the previous notification, but it looks SI transitions from UNASSIGNED to PARTIALLY_ASSIGNED. Ideally it should be from FULLY_ASSIGNED to PARTIALLY_ASSIGNED: === Feb 17 12:04:15 - State Change === eventType = SA_NTF_OBJECT_STATE_CHANGE notificationObject = "safSi=AmfDemo1,safApp=AmfDemo1" notifyingObject = "safApp=safAmfService" notificationClassId = SA_NTF_VENDOR_ID_SAF.SA_SVC_AMF.111 (0x6f) additionalText = "The Assignment state of SI safSi=AmfDemo1,safApp=AmfDemo1 changed" sourceIndicator = SA_NTF_OBJECT_OPERATION State ID = SA_AMF_ASSIGNMENT_STATE Old State: SA_AMF_ASSIGNMENT_UNASSIGNED New State: SA_AMF_ASSIGNMENT_PARTIALLY_ASSIGNED === Feb 17 12:04:15 - State Change === eventType = SA_NTF_OBJECT_STATE_CHANGE notificationObject = "safSi=AmfDemo,safApp=AmfDemo1" notifyingObject = "safApp=safAmfService" notificationClassId = SA_NTF_VENDOR_ID_SAF.SA_SVC_AMF.111 (0x6f) additionalText = "The Assignment state of SI safSi=AmfDemo,safApp=AmfDemo1 changed" sourceIndicator = SA_NTF_OBJECT_OPERATION State ID = SA_AMF_ASSIGNMENT_STATE Old State: SA_AMF_ASSIGNMENT_UNASSIGNED New State: SA_AMF_ASSIGNMENT_PARTIALLY_ASSIGNED I have attached notification_normal_scenario_without_reboot when I locked SU and then collected notifications. Ideally, notifications should be like that even the node is rebooted. TC #48: Related to Alarms and Notification: Notification and alarms, configuration and logs(TC 48) attached in the ticket. 1. Start single controller and a payload. 2N Appl, SU1(Act) on PL-3 and SU2(Stb) on PL-4 as below: immcfg -f /opt/amf_demo/AppConfig-2N.xml_2 amf-adm unlock-in safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock-in safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 2. Keep gdb in csi set of Act component and Lock SI (amf-adm lock safSi=AmfDemo,safApp=AmfDemo1). 3. Stop controller and then respond from gdb. 4. Start controller and subscribe for all notifications(ntfsubscribe > /tmp/notification_after_reboot). 5. Apart from some problems reported above, additionally, SI unassignment alarm is missing for safSi=AmfDemo,safApp=AmfDemo1 (notification_after_reboot attached). The following alarm is missing, which comes under normal scenario if the controller is there(You can find it in notification_normal_scenario_without_reboot attached): === Feb 17 17:14:10 - Alarm === eventType = SA_NTF_ALARM_PROCESSING notificationObject = "safSi=AmfDemo,safApp=AmfDemo1" notifyingObject = "safApp=safAmfService" notificationClassId = SA_NTF_VENDOR_ID_SAF.SA_SVC_AMF.5 (0x5) additionalText = "SI designated by safSi=AmfDemo,safApp=AmfDemo1 has no current active assignments to any SU" probableCause = SA_NTF_SOFTWARE_ERROR perceivedSeverity = SA_NTF_SEVERITY_MAJOR TC #49: Related to Alarms and Notification: Notification and alarms, configuration and logs(TC 49) attached in the ticket. 1. Start single controller and a payload. 2N Appl, SU1(Act) on PL-3: immcfg -f /opt/amf_demo/AppConfig-2N.xml_2 amf-adm unlock-in safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 2. Keep gdb in csi set of Act component and shutdown SU1 (amf-adm shutdown safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1). 3. Stop controller and then respond from gdb. 4. Start controller and subscribe for all notifications(ntfsubscribe > /tmp/notification_after_reboot). 5. The state change notification for SA_AMF_HA_QUIESCED is missing(when controller is up, we get SA_AMF_HA_QUIESCED only), we are getting instead state change notification of SA_AMF_HA_QUIESCING: === Feb 17 19:28:09 - State Change === eventType = SA_NTF_OBJECT_STATE_CHANGE notificationObject = "safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1" notifyingObject = "safApp=safAmfService" notificationClassId = SA_NTF_VENDOR_ID_SAF.SA_SVC_AMF.110 (0x6e) additionalText = "The HA state of SI safSi=AmfDemo1,safApp=AmfDemo1 assigned to SU safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 changed" - additionalInfo: 0 - infoId = 2 infoType = 10 infoValue = "safSi=AmfDemo1,safApp=AmfDemo1" sourceIndicator = SA_NTF_OBJECT_OPERATION State ID = SA_AMF_HA_STATE Old State: New State: SA_AMF_HA_QUIESCING === Feb 17 19:28:09 - State Change === eventType = SA_NTF_OBJECT_STATE_CHANGE notificationObject = "safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1" notifyingObject = "safApp=safAmfService" notificationClassId = SA_NTF_VENDOR_ID_SAF.SA_SVC_AMF.110 (0x6e) additionalText = "The HA state of SI safSi=AmfDemo,safApp=AmfDemo1 assigned to SU safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 changed" - additionalInfo: 0 - infoId = 2 infoType = 10 infoValue = "safSi=AmfDemo,safApp=AmfDemo1" sourceIndicator = SA_NTF_OBJECT_OPERATION State ID = SA_AMF_HA_STATE Old State: New State: SA_AMF_HA_QUIESCING TC #50: Related to Alarms and Notification: Notification and alarms, configuration and logs(TC 50) attached in the ticket. 1. Start single controller and a payload. 2N Appl, SU1(Act) on PL-3 as below: immcfg -f /opt/amf_demo/AppConfig-2N.xml_2 amf-adm unlock-in safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 2. Keep gdb in csi set of Act component and shutdown SI (amf-adm shutdown safSi=AmfDemo,safApp=AmfDemo1). 3. Stop controller and then respond from gdb. 4. Start controller and subscribe for all notifications(ntfsubscribe > /tmp/notification_after_reboot). 5. The following notification is missing(missing in notification_after_reboot, but you can find it in notification_normal_scenario_without_reboot): === Feb 17 19:38:49 - State Change === eventType = SA_NTF_OBJECT_STATE_CHANGE notificationObject = "safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1" notifyingObject = "safApp=safAmfService" notificationClassId = SA_NTF_VENDOR_ID_SAF.SA_SVC_AMF.110 (0x6e) additionalText = "The HA state of SI safSi=AmfDemo,safApp=AmfDemo1 assigned to SU safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 changed" - additionalInfo: 0 - infoId = 2 infoType = 10 infoValue = "safSi=AmfDemo,safApp=AmfDemo1" sourceIndicator = SA_NTF_OBJECT_OPERATION State ID = SA_AMF_HA_STATE Old State: New State: SA_AMF_HA_QUIESCED Thanks -Nagu > -----Original Message----- > From: Nagendra Kumar > Sent: 16 February 2016 21:21 > To: minh chau; hans.nordeb...@ericsson.com; gary....@dektech.com.au; > Praveen Malviya > Cc: opensaf-devel@lists.sourceforge.net > Subject: Re: [devel] FW: [PATCH 0 of 5] Review Request for amf: Add support > for cloud resilience [#1620] V2 > > TC #43: Related to Alarms and Notification: Notification and alarms (TC 43) > attached in the ticket. > Start single controller and a payload. Stop controller. Start controller and > subscribe for all notifications (ntfsubscribe > /tmp/notification) on > controller. > The following notifications/alarms looks inappropriate: > 1. State change notification of safSu=PL- > 3,safSg=NoRed,safApp=OpenSAF from Disable to Enable. Since PL-3 is > already running, the oper state shouldn't change. > 2. Pres state notification of safSu=PL-3,safSg=NoRed,safApp=OpenSAF > from UNINSTANTIATED to INSTANTIATED. [PL-3 is up]. > 3. The Assignment state of safSi=NoRed2,safApp=OpenSAF change. > 4. HA state of SI safSi=NoRed2,safApp=OpenSAF. > 5. The following shouldn't come because PL-3 is up. Also since SC-2 and > PL-4 are down, so these shouldn't come because these don't come when SC- > 1 comes first and no other node is up and running, so why these come when > one payload is up: > a. SI designated by safSi=NoRed1,safApp=OpenSAF has no current > active assignments to any SU > b. SI designated by safSi=NoRed3,safApp=OpenSAF has no current > active assignments to any SU > c. SI designated by safSi=NoRed4,safApp=OpenSAF has no current > active assignments to any SU > d. SI designated by safSi=SC-2N,safApp=OpenSAF has no current > active assignments to any SU > 6. The following shouldn't come as controller never sent alarm for it. > This doesn't come when controller is coming up and no other node is up, so > different when one payload is up: > Previous raised alarm of safSi=NoRed1,safApp=OpenSAF is now > cleared > Previous raised alarm of safSi=SC-2N,safApp=OpenSAF is now > cleared > 7. Why oper state of PL-3 should change? > Oper state safAmfNode=PL-3,safAmfCluster=myAmfCluster changed > deom Dis to Enabled. > > TC #44: The same problem for applications SUs and SIs that are assigned > on > PL-3. > TC #45: Related to Alarms and Notification: Notification and alarms, > configuration and logs(TC 45) attached in the ticket. > 1. Start single controller and a payload. 2N Appl, SU1(Act) on PL-3 as below: > immcfg -f /opt/amf_demo/AppConfig-2N.xml_2 amf-adm unlock-in > safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 > amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 > > 2. Keep sleep and exit 1 in instantiation script as below: > start|instantiate) > #sleep 7 > shift > start $* > sleep 3 > exit 1 > RETVAL=$? > 3. Stop controller. > 4. Kill demo. Demo will move into instantiation failed state. > 5. Start controller and subscribe for all notifications (ntfsubscribe > > /tmp/notification) on controller. > > In the notification, there is no "oper state change"(pasted below) > notification for safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1. This > notification comes when controller is up and you kill demo forcing it to go > into inst failed state: > === Feb 16 20:33:18 - State Change === eventType = > SA_NTF_OBJECT_STATE_CHANGE notificationObject = > "safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1" > notifyingObject = "safApp=safAmfService" > notificationClassId = SA_NTF_VENDOR_ID_SAF.SA_SVC_AMF.108 (0x6c) > additionalText = "Oper state safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 > changed" > sourceIndicator = SA_NTF_OBJECT_OPERATION State ID = > SA_AMF_OP_STATE Old State: SA_AMF_OPERATIONAL_ENABLED New State: > SA_AMF_OPERATIONAL_DISABLED > > Thanks > -Nagu > > > -----Original Message----- > > From: Nagendra Kumar > > Sent: 15 February 2016 19:31 > > To: minh chau; HYPERLINK "mailto:hans.nordeb...@ericsson.com" > > hans.nordeb...@ericsson.com; HYPERLINK "mailto:gary....@dektech.com.au" > > gary....@dektech.com.au; > > Praveen Malviya > > Cc: HYPERLINK "mailto:opensaf-devel@lists.sourceforge.net" > > opensaf-devel@lists.sourceforge.net > > Subject: Re: [devel] FW: [PATCH 0 of 5] Review Request for amf: Add > > support for cloud resilience [#1620] V2 > > > > TC #41: For Nway Act: Just change the red model to Nway. > > I just tested for Si lock and it looks same problem as in Nway and NpM: > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > afApp=AmfDemo1 > > saAmfSISUHAState=ACTIVE(1) > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > ,safApp=AmfDemo1 > > saAmfSISUHAState=QUIESCED(3) > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > So, it looks to me that previous test cases of SI shutdown, SG > > lock/shutdown, Node lock are also valid for Nway Act as well. > > I am not testing those test cases as it is taking too much time to > > test. Please test accordingly. > > > > TC #42: For No red: Just change the red model to No Red. > > Same as TC #41. I could test for Si lock only: > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > afApp=AmfDemo1 > > saAmfSISUHAState=QUIESCED(3) > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > So, other test cases of SI shutdown, SG lock/shutdown, Node lock are > > also valid for No Red as well. Please check once. > > > > Thanks > > -Nagu > > > -----Original Message----- > > > From: Nagendra Kumar > > > Sent: 15 February 2016 19:16 > > > To: minh chau; HYPERLINK "mailto:hans.nordeb...@ericsson.com" > > > HYPERLINK "mailto:hans.nordeb...@ericsson.com" > > > hans.nordeb...@ericsson.com; HYPERLINK > > > "mailto:gary....@dektech.com.au" HYPERLINK > > > "mailto:gary....@dektech.com.au" gary....@dektech.com.au; Praveen > > > Malviya > > > Cc: HYPERLINK "mailto:opensaf-devel@lists.sourceforge.net" > > > HYPERLINK "mailto:opensaf-devel@lists.sourceforge.net" > > > opensaf-devel@lists.sourceforge.net > > > Subject: Re: [devel] FW: [PATCH 0 of 5] Review Request for amf: Add > > > support for cloud resilience [#1620] V2 > > > > > > TC #36: For Nway: Same as TC #32, just change the red model to Nway. > > > The assignments are proper after controller start: > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > ,safApp=AmfDemo1 > > > saAmfSISUHAState=QUIESCED(3) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > afApp=AmfDemo1 > > > saAmfSISUHAState=STANDBY(2) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > ,safApp=AmfDemo1 > > > saAmfSISUHAState=STANDBY(2) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > afApp=AmfDemo1 > > > saAmfSISUHAState=ACTIVE(1) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > TC #37: For Nway: Same as TC #33, just change the red model to Nway. > > > The assignments are proper after controller start: > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > ,safApp=AmfDemo1 > > > saAmfSISUHAState=QUIESCED(3) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > afApp=AmfDemo1 > > > saAmfSISUHAState=QUIESCED(3) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > TC #38: For Nway: Same as TC #34, just change the red model to Nway. > > > The assignments are proper after controller start: > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > afApp=AmfDemo1 > > > saAmfSISUHAState=STANDBY(2) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > afApp=AmfDemo1 > > > saAmfSISUHAState=ACTIVE(1) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > ,safApp=AmfDemo1 > > > saAmfSISUHAState=QUIESCING(4) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > ,safApp=AmfDemo1 > > > saAmfSISUHAState=STANDBY(2) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > TC #39: For Nway: Same as TC #35, just change the red model to Nway. > > > The assignments are proper after controller start: > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > ,safApp=AmfDemo1 > > > saAmfSISUHAState=QUIESCING(4) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > afApp=AmfDemo1 > > > saAmfSISUHAState=QUIESCING(4) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > TC #40: For Nway: Same as TC #35 except Node shutdown, just change > > > the red model to Nway. > > > amf-adm shutdown safAmfNode=PL-3,safAmfCluster=myAmfCluster > > > The assignments are proper after controller start: > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > ,safApp=AmfDemo1 > > > saAmfSISUHAState=QUIESCING(4) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > afApp=AmfDemo1 > > > saAmfSISUHAState=QUIESCING(4) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > afApp=AmfDemo1 > > > saAmfSISUHAState=STANDBY(2) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > ,safApp=AmfDemo1 > > > saAmfSISUHAState=STANDBY(2) > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > Thanks > > > -Nagu > > > > > > > -----Original Message----- > > > > From: Nagendra Kumar > > > > Sent: 15 February 2016 18:53 > > > > To: minh chau; HYPERLINK "mailto:hans.nordeb...@ericsson.com" > > > > HYPERLINK "mailto:hans.nordeb...@ericsson.com" > > > > hans.nordeb...@ericsson.com; > > HYPERLINK "mailto:gary....@dektech.com.au" > HYPERLINK "mailto:gary....@dektech.com.au" gary....@dektech.com.au; > > > > Praveen Malviya > > > > Cc: HYPERLINK "mailto:opensaf-devel@lists.sourceforge.net" > > > > HYPERLINK "mailto:opensaf-devel@lists.sourceforge.net" > > > > opensaf-devel@lists.sourceforge.net > > > > Subject: Re: [devel] FW: [PATCH 0 of 5] Review Request for amf: > > > > Add support for cloud resilience [#1620] V2 > > > > > > > > TC #32: For NpM: Logs and Configuration file(Four SUs and two SIs, > > > > SU1 Act and SU2 Standby) "TC 32" are attached in ticket. > > > > > > > > immcfg -f /opt/amf_demo/AppConfig-NpM_1620.xml > > > > amf-adm unlock-in safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 > > > > amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 > > > > amf-adm unlock-in safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 > > > > amf-adm unlock safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 > > > > amf-adm lock safSi=AmfDemo1,safApp=AmfDemo1 > > > > > > > > Lock SI(as above command) and keep gdb in Quisced callback of > > > > SU1's > > > comp. > > > > Stop controller and respond from gdb. Start controller. > > > > Assignments are not in proper > state(safSi=AmfDemo1,safApp=AmfDemo1 > > > is > > > > still assigned to SU1 as Quisced and SU2 as Standby). > > > > Before SI lock: > > > > > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > > afApp=AmfDemo1 > > > > saAmfSISUHAState=ACTIVE(1) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > > ,safApp=AmfDemo1 > > > > saAmfSISUHAState=ACTIVE(1) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > > afApp=AmfDemo1 > > > > saAmfSISUHAState=STANDBY(2) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > > ,safApp=AmfDemo1 > > > > saAmfSISUHAState=STANDBY(2) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > After SI lock: > > > > > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > > ,safApp=AmfDemo1 > > > > saAmfSISUHAState=QUIESCED(3) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > > afApp=AmfDemo1 > > > > saAmfSISUHAState=ACTIVE(1) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > > afApp=AmfDemo1 > > > > saAmfSISUHAState=STANDBY(2) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > > ,safApp=AmfDemo1 > > > > saAmfSISUHAState=STANDBY(2) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > TC #33: For NpM: Logs and Configuration file(Four SUs and two SIs, > > > > SU1 Act and SU2 Standby) "TC 33" are attached in ticket. > > > > immcfg -f /opt/amf_demo/AppConfig-NpM_1620.xml > > > > amf-adm unlock-in safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 > > > > amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 > > > > amf-adm unlock-in safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 > > > > amf-adm unlock safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 > > > > amf-adm unlock-in safSu=SU3,safSg=AmfDemo,safApp=AmfDemo1 > > > > amf-adm unlock safSu=SU3,safSg=AmfDemo,safApp=AmfDemo1 > > > > > > > > amf-adm lock safSg=AmfDemo,safApp=AmfDemo1 > > > > > > > > Lock SG(as above command) and keep gdb in Quisced callback of > > > > SU1's comp. Stop controller and respond from gdb. Start controller. > > > > Assignments are not in proper state: > > > > > > > > > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > > ,safApp=AmfDemo1 > > > > saAmfSISUHAState=QUIESCED(3) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > > afApp=AmfDemo1 > > > > saAmfSISUHAState=QUIESCED(3) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > TC #34: For NpM: Same as TC #32 except SI shutdown case. After SI > > > > lock, assignments are not proper: > > > > > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > > ,safApp=AmfDemo1 > > > > saAmfSISUHAState=STANDBY(2) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > > ,safApp=AmfDemo1 > > > > saAmfSISUHAState=QUIESCING(4) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > > afApp=AmfDemo1 > > > > saAmfSISUHAState=ACTIVE(1) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > > afApp=AmfDemo1 > > > > saAmfSISUHAState=STANDBY(2) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > TC #35: For NpM: Same as TC #33 except SG shutdown case. After SI > > > > lock, assignments are not proper: > > > > > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > > ,safApp=AmfDemo1 > > > > saAmfSISUHAState=QUIESCING(4) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > > afApp=AmfDemo1 > > > > saAmfSISUHAState=QUIESCING(4) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 > > > > ,safApp=AmfDemo1 > > > > saAmfSISUHAState=STANDBY(2) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > > afApp=AmfDemo1 > > > > saAmfSISUHAState=STANDBY(2) > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > Thanks > > > > -Nagu > > > > > -----Original Message----- > > > > > From: Nagendra Kumar > > > > > Sent: 12 February 2016 20:05 > > > > > To: minh chau; HYPERLINK "mailto:hans.nordeb...@ericsson.com" > > > > > HYPERLINK "mailto:hans.nordeb...@ericsson.com" > > > > > hans.nordeb...@ericsson.com; > > > HYPERLINK "mailto:gary....@dektech.com.au" > HYPERLINK "mailto:gary....@dektech.com.au" gary....@dektech.com.au; > > > > > Praveen Malviya > > > > > Cc: HYPERLINK "mailto:opensaf-devel@lists.sourceforge.net" > > > > > HYPERLINK "mailto:opensaf-devel@lists.sourceforge.net" > > > > > opensaf-devel@lists.sourceforge.net > > > > > Subject: Re: [devel] FW: [PATCH 0 of 5] Review Request for amf: > > > > > Add support for cloud resilience [#1620] V2 > > > > > > > > > > TC #27: Same configuration as TC #24: > > > > > Add a new Csi in running demo appl: Keep gdb in SU1 comp in > > > > > amf_csi_set_callback and add new csi to existing si. Stop > > > > > controller, and then respond from gdb. Start controller. Only > > > > > Act assignment is given to SU1 component. Standby csi assignment > > > > > is not given to SU2 > > > > component: > > > > > > > > > > > > > > > safCSIComp=safComp=AmfDemo\,safSu=SU1\,safSg=AmfDemo\,safApp=Am > > > > > fDemo1,safCsi=AmfDemo,safSi=AmfDemo,safApp=AmfDemo1 > > > > > > > > > > > > > > > safCSIComp=safComp=AmfDemo\,safSu=SU2\,safSg=AmfDemo\,safApp=Am > > > > > fDemo1,safCsi=AmfDemo,safSi=AmfDemo,safApp=AmfDemo1 > > > > > > > > > > > > > > > safCSIComp=safComp=AmfDemo\,safSu=SU1\,safSg=AmfDemo\,safApp=Am > > > > > fDemo1,safCsi=AmfDemo1,safSi=AmfDemo,safApp=AmfDemo1 > > > > > > > > > > Logs " TC 27" are attached in ticket. > > > > > > > > > > TC #28: Same configuration as TC #24: > > > > > Delete a Csi in running demo appl: Add a new csi and keep gdb in > > > > > SU1 comp in amf_csi_remove_callback and then delete csi. Stop > > > > > controller, and then respond from gdb. Start controller. Amfd crashes: > > > > > > > > > > Feb 12 15:52:32 PM_SC-1 osafamfnd[16096]: Started Feb 12 > > > > > 15:52:32 > > > > > PM_SC-1 osafamfnd[16096]: NO Sending node up due to > NCSMDS_UP > > > Feb > > > > 12 > > > > > 15:52:32 PM_SC-1 osafamfd[16086]: NO Received node_up from > > > > > 2010f: msg_id 1 > > > > > Feb 12 15:52:32 PM_SC-1 osafamfd[16086]: csi.cc:1470: > > > > > avd_compcsi_recreate: Assertion 'csi' failed. > > > > > Feb 12 15:52:32 PM_SC-1 osafamfnd[16096]: WA AMF director > > > > unexpectedly > > > > > crashed Feb 12 15:52:32 PM_SC-1 osafamfnd[16096]: WA AMF > > > > > director unexpectedly crashed > > > > > > > > > > Logs " TC 28" are attached in ticket. > > > > > > > > > > TC #29: Configuration: One controller, and two PLs. 2N SG, SU1 > > > > > and > > > > > SU2 with > > > > > 3 comp with one csi and SI associated. Si under si deps. > > > > > safSi=B,safApp=Test depends on safSi=A,safApp=Test and > > > > > safSi=C,safApp=Test depends on safSi=B,safApp=Test. SU1(Act) on > > > > > PL-3 and > > > > SU2(Standby) on PL-4. > > > > > > > > > > Lock SI A and keep gdb in quisced callback and stop controller. > > > > > Respond from gdb and start controller. After controller comes up > > > > > SU2 has two Standby assignment and no active assignments, which > > > > > looks > > > > serious problem. > > > > > > > > > > safSISU=safSu=ABC2\,safSg=2N\,safApp=Test,safSi=B,safApp=Test > > > > > saAmfSISUHAState=STANDBY(2) > > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > safSISU=safSu=ABC2\,safSg=2N\,safApp=Test,safSi=C,safApp=Test > > > > > saAmfSISUHAState=STANDBY(2) > > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > At the same time, there are two error message in controller: > > > > > Feb 12 19:32:01 PM_SC-1 osafamfd[1572]: EM sg_2n_fsm.cc:1439: > > > > > safSu=ABC2,safSg=2N,safApp=Test (31) Feb 12 19:32:01 PM_SC-1 > > > > > osafamfd[1572]: EM sg_2n_fsm.cc:1439: > > > > > safSu=ABC2,safSg=2N,safApp=Test (31) > > > > > > > > > > Logs " TC 29" are attached in ticket. > > > > > > > > > > TC #30: Configuration same as TC #29: This case, Lock SI B, > > > > > which is dependent on SI A and is sponsor of SI C. > > > > > Only assignment of SI B is removed and SI C assignment remains: > > > > > safSISU=safSu=ABC1\,safSg=2N\,safApp=Test,safSi=C,safApp=Test > > > > > saAmfSISUHAState=ACTIVE(1) > > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > safSISU=safSu=ABC2\,safSg=2N\,safApp=Test,safSi=A,safApp=Test > > > > > saAmfSISUHAState=STANDBY(2) > > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > safSISU=safSu=ABC1\,safSg=2N\,safApp=Test,safSi=A,safApp=Test > > > > > saAmfSISUHAState=ACTIVE(1) > > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > safSISU=safSu=ABC2\,safSg=2N\,safApp=Test,safSi=C,safApp=Test > > > > > saAmfSISUHAState=STANDBY(2) > > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > Logs " TC 30" are attached in ticket. > > > > > > > > > > TC #31: Same configuration as TC #30. Lock SI B, which is > > > > > dependent on SI A and is sponsor of SI C. Assignment of SI B is > > > > > removed and tolerance timer will start running. > > > > > Reboot the controller. The assignment of SI C should be removed > > > > > because its sponsor is in locked state. > > > > > Logs " TC 30" are attached in ticket. > > > > > > > > > > Thanks > > > > > -Nagu > > > > > > > > > > > -----Original Message----- > > > > > > From: Nagendra Kumar > > > > > > Sent: 11 February 2016 20:06 > > > > > > To: minh chau; HYPERLINK "mailto:hans.nordeb...@ericsson.com" > > > > > > HYPERLINK "mailto:hans.nordeb...@ericsson.com" > > > > > > hans.nordeb...@ericsson.com; > > > > HYPERLINK "mailto:gary....@dektech.com.au" > > > > HYPERLINK "mailto:gary....@dektech.com.au" gary....@dektech.com.au; > > > > > > Praveen Malviya > > > > > > Cc: HYPERLINK "mailto:opensaf-devel@lists.sourceforge.net" > > > > > > HYPERLINK "mailto:opensaf-devel@lists.sourceforge.net" > > > > > > opensaf-devel@lists.sourceforge.net > > > > > > Subject: Re: [devel] FW: [PATCH 0 of 5] Review Request for amf: > > > > > > Add > > > > > support > > > > > > for cloud resilience [#1620] V2 > > > > > > > > > > > > TC #21: Node shutdown and same as TC #17. Logs " TC 21" are > > > > > > attached in ticket. > > > > > > > > > > > > TC #22: Node lock and unlock(2nd time), result same as TC #18. > > > > > > Logs " TC > > > > > 22" > > > > > > are attached in ticket. > > > > > > > > > > > > TC #23: Node Group shutdown operation: Stop controller during > > > > > > shutdown operation and it will remain in shutdown state even > > > > > > if all the assignments > > > > > are > > > > > > removed. > > > > > > > > > > > > TC #24: Configuration: Start SC-1 and PL-3 and PL-4, create 2N > > > > > > Red model > > > > > > SU1(Act) on PL-3 and SU2(Standby) on PL-4 . Create a node > > > > > > group having > > > > > PL- > > > > > > 3 and PL-4. > > > > > > Now, lock and lock-in the node group. Reboot the controller. > > > > > > Check the admin state of node group, it will be 3. Now, do > > > > > > unlock-in of node group, > > > > > > SU1 and SU2 are not instantiated(which is wrong). Node group > > > > > > admin state > > > > > is > > > > > > 2 now(locked). Logs " TC 24" are attached in ticket. > > > > > > > > > > > > TC #25: Same configuration as TC #24: > > > > > > Lock the node group and keep gdb in amf_csi_set_callback in > > > > > > SU1's > > > > > > comp(Act) and amf_csi_remove_callback in SU2's comp(Standby). > > > > > > Reboot the controller. When SC-1 comes up, respond from gdb > > > > > > from > > > > > > SU1 and > > > > > > SU2 components. The following error comes, that means extra > > > > > > susi remove is > > > > > > coming: > > > > > > Feb 11 19:46:27 PM_PL-3 osafamfnd[16881]: ER susi_assign_evh: > > > > > > 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' has no > assignments > > > > > > > > > > > > Logs " TC 25" are attached in ticket. > > > > > > > > > > > > TC #26: Same configuration as TC #24: > > > > > > Lock the node group and keep gdb in amf_csi_remove_callback in > > > > > > SU1's > > > > > > comp(Act) and amf_csi_remove_callback in SU2's comp(Standby). > > > > > > Reboot the controller. When SC-1 comes up, respond from gdb > > > > > > from > > > > > > SU1 and > > > > > > SU2 components. Amfnd on PL-3 and PL-4 crashes: > > > > > > PL-3: > > > > > > Feb 11 19:56:41 PM_PL-3 osafamfnd[17561]: di.cc:850: > > > > > > avnd_di_susi_resp_send: Assertion 'si' failed. > > > > > > Feb 11 19:56:42 PM_PL-3 osafclmna[17552]: AL AMF Node Director > > > > > > is > > > > > down, > > > > > > terminate this process > > > > > > PL-4: > > > > > > Feb 11 19:56:40 PM_PL-4 osafamfnd[10912]: di.cc:850: > > > > > > avnd_di_susi_resp_send: Assertion 'si' failed. > > > > > > Feb 11 19:56:40 PM_PL-4 osafimmnd[10893]: NO Global discard > > > > > > node received for nodeId:2030f pid:17542 Feb 11 19:56:40 > > > > > > PM_PL-4 > > > > > > osafimmnd[10893]: NO Implementer disconnected 13 <0, > > > 2030f(down)> > > > > > > (MsgQueueService131855) Feb 11 19:56:40 PM_PL-4 > > > amf_demo[11007]: > > > > > AL > > > > > > AMF Node Director is down, terminate this process > > > > > > > > > > > > Logs " TC 26" are attached in ticket. > > > > > > > > > > > > Thanks > > > > > > -Nagu > > > > > > > -----Original Message----- > > > > > > > From: Nagendra Kumar > > > > > > > Sent: 10 February 2016 17:30 > > > > > > > To: minh chau; HYPERLINK > > > > > > > "mailto:hans.nordeb...@ericsson.com" > > > > > > > HYPERLINK "mailto:hans.nordeb...@ericsson.com" > > > > > > > hans.nordeb...@ericsson.com; > > > > > HYPERLINK "mailto:gary....@dektech.com.au" > > > > > HYPERLINK "mailto:gary....@dektech.com.au" gary....@dektech.com.au; > > > > > > > Praveen Malviya > > > > > > > Cc: HYPERLINK "mailto:opensaf-devel@lists.sourceforge.net" > > > > > > > HYPERLINK "mailto:opensaf-devel@lists.sourceforge.net" > > > > > > > opensaf-devel@lists.sourceforge.net > > > > > > > Subject: Re: [devel] FW: [PATCH 0 of 5] Review Request for amf: > > > > > > > Add support for cloud resilience [#1620] V2 > > > > > > > > > > > > > > TC #16. Same configuration as #12: Run SI shutdown and > > keep sleep > > > > > > > of 5 sec before saAmfCSIQuiescingComplete and stop > > > > > > > controller and then after sleep, reject > > > > > > > saAmfCSIQuiescingComplete with > SA_AIS_ERR_FAILED_OPERATION. > > > > > > > All the assignment from SU1 on > > > > > > > PL- > > > 3 > > > > > and > > > > > > > SU2 on PL-4 are removed and SI admin state is 2(locked): > > > > > > > saAmfSIAdminState SA_UINT32_T 2 > > > > > > > (0x2) > > > > > > > > > > > > > > "Si going into locked state" is different behaviour when > > > > > > > controller is up and running and run this test case. In > > > > > > > case, controller is available, SI will be in unlocked state > > > > > > > and all the assignments will be on SU2 as Act and SU3 as Standby > (on PL-4). > > > > > > > This need either > > > > > correction > > > > > > or documentation. > > > > > > > > > > > > > > TC #17. Same configuration as #12: Run SG shutdown > > and > > > > > > keep sleep > > > > > > > of 5 sec before saAmfCSIQuiescingComplete and stop > > > > > > > controller and then after sleep, reject > > > > > > > saAmfCSIQuiescingComplete with > SA_AIS_ERR_FAILED_OPERATION. > > > > > > > Amfnd crashes[Please note that > > > this > > > > > test > > > > > > > case works with controller up]: > > > > > > > Syslog and bt: > > > > > > > Feb 10 11:44:29 PM_PL-3 osafamfnd[15508]: NO component with > > > > > > > QUIESCED/QUIESCING assignment failed Feb 10 11:44:29 PM_PL- > 3 > > > > > > > osafamfnd[15508]: NO recovery action 'comp restart' > > > > > > > escalated to 'comp failover' > > > > > > > Feb 10 11:44:29 PM_PL-3 osafamfnd[15508]: NO SU failover > > > > > > > probation timer started (timeout: 1200000000000 ns) Feb 10 > > > > > > > 11:44:29 PM_PL-3 > > > > > > > osafamfnd[15508]: NO Performing failover of > > > > > > > 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' (SU failover > > count: > > > 1) > > > > > Feb > > > > > > > 10 11:44:29 PM_PL-3 osafamfnd[15508]: NO > > > > > > > > > > 'safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' > > > > > > > recovery action escalated from 'componentRestart' to > > > > > > 'componentFailover' > > > > > > > Feb 10 11:44:29 PM_PL-3 osafamfnd[15508]: NO > > > > > > > > > > 'safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' > > > > > > > faulted due to 'csiSetcallbackFailed' : Recovery is > > 'componentFailover' > > > > > > > Feb 10 11:44:29 PM_PL-3 osafamfnd[15508]: NO > > > > > > > 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State > > > > > > INSTANTIATED > > > > > > > => TERMINATING Feb 10 11:44:29 PM_PL-3 osafamfnd[15508]: > NO > > > > > > Removed > > > > > > > 'safSi=AmfDemo,safApp=AmfDemo1' from > > > > > > > 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' > > > > > > > Feb 10 11:44:29 PM_PL-3 amf_demo[15721]: saAmfHAStateGet > > > FAILED > > > > - > > > > > 7 > > > > > > > Feb 10 11:44:29 PM_PL-3 amf_demo[15721]: exiting (caught > > > > > > > term > > > > > > > signal) Feb 10 11:44:29 PM_PL-3 osafamfnd[15508]: NO > > > > > > > avnd_di_oper_send() deferred as AMF director is offline Feb > > > > > > > 10 > > > > > > > 11:44:29 PM_PL-3 > > > > > > > osafimmnd[15760]: AL AMF Node Director is down, terminate > > > > > > > this > > > > > process > > > > > > > > > > > > > > Program terminated with signal 11, Segmentation fault. > > > > > > > #0 0x0000000000412b50 in > > > > > > > avnd_comp_cmplete_all_assignment(avnd_cb_tag*, > > > avnd_comp_tag*) > > > > () > > > > > > > (gdb) bt > > > > > > > #0 0x0000000000412b50 in > > > > > > > avnd_comp_cmplete_all_assignment(avnd_cb_tag*, > > > avnd_comp_tag*) > > > > () > > > > > > > #1 0x000000000040a093 in > > > > > > > avnd_comp_clc_terming_cleansucc_hdler(avnd_cb_tag*, > > > > > > avnd_comp_tag*) > > > > > > > () > > > > > > > #2 0x000000000040c7d4 in > > avnd_comp_clc_fsm_run(avnd_cb_tag*, > > > > > > > avnd_comp_tag*, avnd_comp_clc_pres_fsm_ev) () > > > > > > > #3 0x000000000040ce49 in > > > > > > > avnd_evt_clc_resp_evh(avnd_cb_tag*, > > > > > > > avnd_evt_tag*) () > > > > > > > #4 0x000000000042133f in avnd_main_process() () at > > > > > > > main.cc:667 > > > > > > > #5 0x0000000000405517 in main () at main.cc:186 > > > > > > > (gdb) thread apply all bt > > > > > > > > > > > > > > Thread 4 (Thread 0x7fdaf3c05b00 (LWP 15512)): > > > > > > > #0 0x00007fdaf2b2b415 in __lll_unlock_wake () from > > > > > > > /lib64/libpthread.so.0 > > > > > > > #1 0x00007fdaf2b27ac4 in _L_unlock_553 () from > > > > > > > /lib64/libpthread.so.0 > > > > > > > #2 0x00007fdaf2b279f7 in __pthread_mutex_unlock_usercnt () > > > > > > > from > > > > > > > /lib64/libpthread.so.0 > > > > > > > #3 0x00007fdaf37edac3 in ncs_os_lock () from > > > > > > > /usr/local/lib/libopensaf_core.so.0 > > > > > > > #4 0x00007fdaf37e084d in ncs_ipc_send () from > > > > > > > /usr/local/lib/libopensaf_core.so.0 > > > > > > > #5 0x000000000041eea1 in avnd_evt_send(avnd_cb_tag*, > > > > > avnd_evt_tag*) > > > > > > > () > > > > > > > #6 0x000000000040a2cb in > > > > > > > > > > comp_clc_resp_callback(NCS_OS_PROC_EXECUTE_TIMED_CB_INFO*) > > > > () > > > > > > > #7 0x00007fdaf37ecdfb in give_exec_mod_cb () from > > > > > > > /usr/local/lib/libopensaf_core.so.0 > > > > > > > #8 0x00007fdaf37ecfde in ncs_exec_mod_hdlr () from > > > > > > > /usr/local/lib/libopensaf_core.so.0 > > > > > > > #9 0x00007fdaf2b247b6 in start_thread () from > > > > > > > /lib64/libpthread.so.0 > > > > > > > #10 0x00007fdaf20da9cd in clone () from /lib64/libc.so.6 > > > > > > > #11 0x0000000000000000 in ?? () > > > > > > > > > > > > > > Thread 3 (Thread 0x7fdaf3c25b00 (LWP 15510)): > > > > > > > #0 0x00007fdaf20d14f6 in poll () from /lib64/libc.so.6 > > > > > > > #1 0x00007fdaf3817623 in mdtm_process_recv_events () from > > > > > > > /usr/local/lib/libopensaf_core.so.0 > > > > > > > #2 0x00007fdaf2b247b6 in start_thread () from > > > > > > > /lib64/libpthread.so.0 > > > > > > > #3 0x00007fdaf20da9cd in clone () from /lib64/libc.so.6 > > > > > > > #4 0x0000000000000000 in ?? () > > > > > > > > > > > > > > Thread 2 (Thread 0x7fdaf3c58b00 (LWP 15509)): > > > > > > > #0 0x00007fdaf20d14f6 in poll () from /lib64/libc.so.6 > > > > > > > #1 0x00007fdaf37db22f in osaf_ppoll () from > > > > > > > /usr/local/lib/libopensaf_core.so.0 > > > > > > > #2 0x00007fdaf37e2acf in ncs_tmr_wait () from > > > > > > > /usr/local/lib/libopensaf_core.so.0 > > > > > > > #3 0x00007fdaf2b247b6 in start_thread () from > > > > > > > /lib64/libpthread.so.0 > > > > > > > #4 0x00007fdaf20da9cd in clone () from /lib64/libc.so.6 > > > > > > > #5 0x0000000000000000 in ?? () > > > > > > > > > > > > > > Thread 1 (Thread 0x7fdaf3c28720 (LWP 15508)): > > > > > > > #0 0x0000000000412b50 in > > > > > > > avnd_comp_cmplete_all_assignment(avnd_cb_tag*, > > > avnd_comp_tag*) > > > > () > > > > > > > #1 0x000000000040a093 in > > > > > > > avnd_comp_clc_terming_cleansucc_hdler(avnd_cb_tag*, > > > > > > avnd_comp_tag*) > > > > > > > () > > > > > > > #2 0x000000000040c7d4 in > > avnd_comp_clc_fsm_run(avnd_cb_tag*, > > > > > > > avnd_comp_tag*, avnd_comp_clc_pres_fsm_ev) () > > > > > > > #3 0x000000000040ce49 in > > > > > > > avnd_evt_clc_resp_evh(avnd_cb_tag*, > > > > > > > avnd_evt_tag*) () > > > > > > > #4 0x000000000042133f in avnd_main_process() () at > > > > > > > main.cc:667 > > > > > > > #5 0x0000000000405517 in main () at main.cc:186 > > > > > > > > > > > > > > TC #18. Same configuration as #12: Run SG lock and > > keep gdb > > > > > > in > > > > > > > amf_csi_remove_callback and stop controller and then start > > > > > > > the controller and make is up. Now release Amfnd from gdb so > > > > > > > that it can respond to csi remove(Please note that > > > > > > > controller has reboot and is available now). Now, issue SG > > > > > > > unlock. Amfnd crashes on > > > > > > > PL-3 and PL-4 at the same location[Please note that this > > > > > > > test case works with controller > > > > > > up]: > > > > > > > Syslog and Bt: > > > > > > > Feb 10 16:35:51 PM_PL-3 amf_demo[26623]: CSI Remove for all > > > > > > > CSIs Feb > > > > > > > 10 16:35:51 PM_PL-3 osafamfnd[26545]: NO Removed > > > > > > > 'safSi=AmfDemo,safApp=AmfDemo1' from > > > > > > > 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' > > > > > > > Feb 10 16:36:02 PM_PL-3 osafamfnd[26545]: NO Assigning > > > > > > > 'safSi=AmfDemo,safApp=AmfDemo1' ACTIVE to > > > > > > > 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' > > > > > > > Feb 10 16:36:02 PM_PL-3 amf_demo[26623]: CSI Set - add > > > > > > > 'safCsi=AmfDemo,safSi=AmfDemo,safApp=AmfDemo1' HAState > > > Active > > > > > > > Feb 10 16:36:02 PM_PL-3 amf_demo[26623]: name: abcdef, > > > value: > > > > > val1 > > > > > > > Feb 10 16:36:02 PM_PL-3 amf_demo[26623]: name: abcdef, > > > value: > > > > > val2 > > > > > > > Feb 10 16:36:02 PM_PL-3 osafamfnd[26545]: NO Assigned > > > > > > > 'safSi=AmfDemo,safApp=AmfDemo1' ACTIVE to > > > > > > > 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' > > > > > > > Feb 10 16:36:02 PM_PL-3 osafamfnd[26545]: di.cc:850: > > > > > > > avnd_di_susi_resp_send: Assertion 'si' failed. > > > > > > > Feb 10 16:36:02 PM_PL-3 osafclmna[26536]: AL AMF Node > > > > > > > Director is down, terminate this process > > > > > > > > > > > > > > Core was generated by `/usr/local/lib/opensaf/osafamfnd -- > > > > > > > tracemask=0xffffffff'. > > > > > > > Program terminated with signal 6, Aborted. > > > > > > > #0 0x00007f022ebd9b55 in raise () from /lib64/libc.so.6 > > > > > > > (gdb) bt > > > > > > > #0 0x00007f022ebd9b55 in raise () from /lib64/libc.so.6 > > > > > > > #1 0x00007f022ebdb131 in abort () from /lib64/libc.so.6 > > > > > > > #2 0x00007f023038331b in __osafassert_fail () from > > > > > > > /usr/local/lib/libopensaf_core.so.0 > > > > > > > #3 0x000000000041b399 in > > > > > > > avnd_di_susi_resp_send(avnd_cb_tag*, > > > > > > > avnd_su_tag*, avnd_su_si_rec*) () > > > > > > > #4 0x000000000042e9fa in > avnd_su_si_oper_done(avnd_cb_tag*, > > > > > > > avnd_su_tag*, avnd_su_si_rec*) () > > > > > > > #5 0x0000000000411622 in > > > > avnd_comp_csi_assign_done(avnd_cb_tag*, > > > > > > > avnd_comp_tag*, avnd_comp_csi_rec*) () > > > > > > > #6 0x0000000000407397 in > > > > > > > avnd_evt_ava_resp_evh(avnd_cb_tag*, > > > > > > > avnd_evt_tag*) () > > > > > > > #7 0x000000000042133f in avnd_main_process() () at > > > > > > > main.cc:667 > > > > > > > #8 0x0000000000405517 in main () at main.cc:186 > > > > > > > > > > > > > > TC #19. Same configuration as #12: Run Node lock > > and keep > > > > > > sleep of > > > > > > > 5 sec in amf_csi_set_callback and stop controller. Reject > > > > > > > quisced assignment in amf_csi_set_callback, Amfnd crashes. > > > > > > > Syslog and gdb is the same as in TC #17. > > > > > > > > > > > > > > TC #20. Same configuration as #12: Issue Node > > shutdown: > > > > > > and keep > > > > > > > sleep of 5 sec in amf_csi_set_callback before sending > > > > > > > saAmfResponse() and stop controller. Amfnd crashes: > > > > > > > Syslog: > > > > > > > > > > > > > > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO component with > > > > > > > QUIESCED/QUIESCING a > > > > > > > ssignment failed > > > > > > > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO recovery action > > > > 'comp > > > > > > > restart' esca lated > > > > > > > to 'comp failover' > > > > > > > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO SU failover > > > > > > > probation > > > > > > timer > > > > > > > started (timeout: > > > > > > > 1200000000000 ns) > > > > > > > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO Performing > > > > > > > failover of 'safSu=SU1,s > > > > > afSg=AmfDemo,safApp=AmfDemo1' > > > > > > > (SU failover count: 1) > > > > > > > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO > > > > > > > 'safComp=AmfDemo,safSu=SU1,safSg=Am > > > > > > > fDemo,safApp=AmfDemo1' recovery action escalated from > > > > > > > 'componentRestart' to 'com > > ponentFailover' > > > > > > > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO > > > > > > > 'safComp=AmfDemo,safSu=SU1,safSg=Am > > > > > > > fDemo,safApp=AmfDemo1' faulted due to 'csiSetcallbackFailed' : > > > > > Recovery > > > > > > is > > > > > > > 'comp onentFailover' > > > > > > > Feb 10 17:21:10 PM_PL-3 osafamfnd[29330]: NO > > > > > > > 'safSu=SU1,safSg=AmfDemo,safApp=Amf > > > > > Demo1' > > > > > > > Presence State INSTANTIATED => TERMINATING Feb 10 17:21:10 > > > > PM_PL- > > > > > 3 > > > > > > > osafamfnd[29330]: NO Removed > > > > > > > 'safSi=AmfDemo,safApp=AmfDe > > > > > > > mo1' > from > > > > > > > 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' > > > > > > > Feb 10 17:21:10 PM_PL-3 amf_demo[29519]: saAmfHAStateGet > > > FAILED > > > > - > > > > > 7 > > > > > > > Feb 10 17:21:10 PM_PL-3 osafimmnd[29561]: AL AMF Node > > Director > > > is > > > > > > > down, terminat e > > > > > > > this process > > > > > > > Feb 10 17:21:10 PM_PL-3 amf_demo[29519]: AL AMF Node > > > > > > > Director > > > is > > > > > > down, > > > > > > > terminate this > > > > > > > process > > > > > > > Feb 10 17:21:10 PM_PL-3 amf_demo[29519]: exiting (caught > > > > > > > term > > > > > > > signal) Feb 10 17:21:10 PM_PL-3 osafclmna[29321]: AL AMF > > > > > > > Node Director is > > > > > > down, > > > > > > > terminat e this > > > > > > > process > > > > > > > > > > > > > > Bt: > > > > > > > Core was generated by `/usr/local/lib/opensaf/osafamfnd -- > > > > > > > tracemask=0xffffffff'. > > > > > > > Program terminated with signal 11, Segmentation fault. > > > > > > > #0 0x00000000004117c9 in > > > > avnd_comp_csi_assign_done(avnd_cb_tag*, > > > > > > > avnd_comp_tag*, avnd_comp_csi_rec*) () > > > > > > > (gdb) bt > > > > > > > #0 0x00000000004117c9 in > > > > avnd_comp_csi_assign_done(avnd_cb_tag*, > > > > > > > avnd_comp_tag*, avnd_comp_csi_rec*) () > > > > > > > #1 0x0000000000406a3b in > > > > > > > avnd_evt_ava_csi_quiescing_compl_evh(avnd_cb_tag*, > > > > avnd_evt_tag*) > > > > > () > > > > > > > #2 0x000000000042133f in avnd_main_process() () at > > > > > > > main.cc:667 > > > > > > > #3 0x0000000000405517 in main () at main.cc:186 > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > -Nagu > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Nagendra Kumar > > > > > > > > Sent: 09 February 2016 21:39 > > > > > > > > To: minh chau; HYPERLINK > > > > > > > > "mailto:hans.nordeb...@ericsson.com" > > > > > > > > HYPERLINK "mailto:hans.nordeb...@ericsson.com" > > > > > > > > hans.nordeb...@ericsson.com; > > > > > > HYPERLINK "mailto:gary....@dektech.com.au" > > > > > > HYPERLINK "mailto:gary....@dektech.com.au" gary....@dektech.com.au; > > > > > > > > Praveen Malviya > > > > > > > > Cc: HYPERLINK > > > > > > > > "mailto:opensaf-devel@lists.sourceforge.net" > > > > > > > > HYPERLINK "mailto:opensaf-devel@lists.sourceforge.net" > > > > > > > > opensaf-devel@lists.sourceforge.net > > > > > > > > Subject: Re: [devel] FW: [PATCH 0 of 5] Review Request for amf: > > > > > > > > Add > > > > > > > support > > > > > > > > for cloud resilience [#1620] V2 > > > > > > > > > > > > > > > > 15. Same configuration as Test Case #12, SI lock. Keep gdb > > > > > > > > in both the SUs for csi remove and keep timeout as 100 sec. > > > > > > > > Slock SI and stop > > > > > > controller. > > > > > > > > Start controller and allow csi remove to timeout. > > > > > > > > Two things: > > > > > > > > SU2 has Standby assignment(which is wrong), SU1 has not > > > > > > assignment. > > > > > > > > Error at PL-4 : SU-SI record addition failed > > > > > > > > > > > > > > > > PM_SC-1:/home/nagu/views/staging # amf-state siass > > > > > > > > safSISU=safSu=PL- > > > > > > 4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF > > > > > > > > saAmfSISUHAState=ACTIVE(1) > > > > > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > safSISU=safSu=PL- > > > > > > > > > > 3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF > > > > > > > > saAmfSISUHAState=ACTIVE(1) > > > > > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s > > > > > > > > afApp=AmfDemo1 > > > > > > > > saAmfSISUHAState=STANDBY(2) > > > > > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > safSISU=safSu=SC- > > > > > > > > > > 1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF > > > > > > > > saAmfSISUHAState=ACTIVE(1) > > > > > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC- > > > > > > > > 2N,safApp=OpenSAF > > > > > > > > saAmfSISUHAState=ACTIVE(1) > > > > > > > > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > > > > > > > > > > > > > > > Syslog of PL-4: > > > > > > > > > > > > > > > > Feb 9 21:24:50 PM_PL-4 osafamfnd[7998]: NO > > > > > > > > 'safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1' component > > > restart > > > > > > > > probation timer started (timeout: 60000000000 ns) Feb 9 > > > > > > > > 21:24:50 > > > > > > > > PM_PL- > > > > > > > 4 > > > > > > > > osafamfnd[7998]: NO Restarting a component of > > > > > > > > 'safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1' (comp restart > > > > count: > > > > > 1) > > > > > > > Feb > > > > > > > > 9 21:24:50 PM_PL-4 osafamfnd[7998]: NO > > > > > > > > > > > > 'safComp=AmfDemo,safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1' > > > > > > > > faulted due to 'csiRemovecallbackTimeout' : Recovery is > > > > > > > 'componentRestart' > > > > > > > > Feb 9 21:24:55 PM_PL-4 amf_demo_script: killproc > > > > > > > > /opt/amf_demo/amf_demo failed Feb 9 21:24:55 PM_PL-4 > > > > > > > > amf_demo[8200]: > > > > > > > > > > > > 'safComp=AmfDemo,safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1' > > > > > > > > started Feb 9 21:24:55 PM_PL-4 osafamfnd[7998]: NO > > > > > > > > Removed 'safSi=AmfDemo1,safApp=AmfDemo1' from > > > > > > > > 'safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1' > > > > > > > > Feb 9 21:24:55 PM_PL-4 amf_demo[8200]: HC started with > > > > > > > > AMF Feb > > > > > > > > 9 > > > > > > > > 21:24:55 PM_PL-4 amf_demo[8200]: Registered with AMF Feb > > > > > > > > 9 > > > > > > 21:24:55 > > > > > > > > PM_PL-4 amf_demo[8200]: CSI Set - add > > > > > > > > 'safCsi=AmfDemo,safSi=AmfDemo,safApp=AmfDemo1' HAState > > > > > Standby > > > > > > > > Feb 9 21:24:55 PM_PL-4 amf_demo[8200]: name: abcdef, > > > value: > > > > > > val1 > > > > > > > > Feb 9 21:24:55 PM_PL-4 amf_demo[8200]: name: abcdef, > > > value: > > > > > > val2 > > > > > > > > Feb 9 21:24:55 PM_PL-4 osafamfnd[7998]: CR SU-SI record > > > > > > > > addition failed, SU= > > safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 > > > : > > > > > > > > SI=safSi=AmfDemo,safApp=AmfDemo1 Feb 9 21:24:55 PM_PL-4 > > > > > > > > amf_demo[8200]: Health check 1 Feb 9 21:25:50 PM_PL-4 > > > > > > > > osafamfnd[7998]: NO > > > > 'safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1' > > > > > > > > Component or SU restart probation timer expired > > > > > > > > > > > > > > > > Thanks > > > > > > > > -Nagu > > > > > > > > > > > > > > > > ---------------------------------------------------------- > > > > > > > > -- > > > > > > > > -- > > > > > > > > -- > > > > > > > > ---- > > > > > > > > ---------- > > > > > > > > Site24x7 APM Insight: Get Deep Visibility into Application > > > > > > > > Performance APM > > > > > > > + > > > > > > > > Mobile APM + RUM: Monitor 3 App instances at just > > > > > > > > $35/Month > > > > > Monitor > > > > > > > > end-to-end web transactions and take corrective actions > > > > > > > > now > > > > > > > Troubleshoot > > > > > > > > faster and improve end-user experience. Signup Now! > > > > > > > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > > > > > > > > _______________________________________________ > > > > > > > > Opensaf-devel mailing list HYPERLINK > > > > > > > > "mailto:Opensaf-devel@lists.sourceforge.net" > > > > > > > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > > > > > > > Opensaf-devel@lists.sourceforge.net > > > > > > > > https://lists.sourceforge.net/lists/listinfo/opensaf-devel > > > > > > > > > > > > > > ------------------------------------------------------------ > > > > > > > -- > > > > > > > -- > > > > > > > -- > > > > > > > ---- > > > > > > > -------- > > > > > > > Site24x7 APM Insight: Get Deep Visibility into Application > > > > > > > Performance APM + Mobile APM + RUM: Monitor 3 App instances > > > > > > > at just $35/Month Monitor end-to-end web transactions and > > > > > > > take corrective actions now Troubleshoot faster and improve > > > > > > > end-user > > > > experience. Signup Now! > > > > > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > > > > > > > _______________________________________________ > > > > > > > Opensaf-devel mailing list > > > > > > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > > > > > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > > > > > > Opensaf-devel@lists.sourceforge.net > > > > > > > https://lists.sourceforge.net/lists/listinfo/opensaf-devel > > > > > > > > > > > > -------------------------------------------------------------- > > > > > > -- > > > > > > -- > > > > > > -- > > > > > > ---------- > > > > > > Site24x7 APM Insight: Get Deep Visibility into Application > > > > > > Performance APM > > > > > + > > > > > > Mobile APM + RUM: Monitor 3 App instances at just $35/Month > > > > > > Monitor end-to-end web transactions and take corrective > > > > > > actions now > > > > > Troubleshoot > > > > > > faster and improve end-user experience. Signup Now! > > > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > > > > > > _______________________________________________ > > > > > > Opensaf-devel mailing list > > > > > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > > > > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > > > > > Opensaf-devel@lists.sourceforge.net > > > > > > https://lists.sourceforge.net/lists/listinfo/opensaf-devel > > > > > > > > > > ---------------------------------------------------------------- > > > > > -- > > > > > -- > > > > > -- > > > > > -------- > > > > > Site24x7 APM Insight: Get Deep Visibility into Application > > > > > Performance APM + Mobile APM + RUM: Monitor 3 App instances at > > > > > just $35/Month Monitor end-to-end web transactions and take > > > > > corrective actions now Troubleshoot faster and improve end-user > > experience. > > > Signup Now! > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > > > > > _______________________________________________ > > > > > Opensaf-devel mailing list > > > > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > > > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > > > > Opensaf-devel@lists.sourceforge.net > > > > > https://lists.sourceforge.net/lists/listinfo/opensaf-devel > > > > > > > > ------------------------------------------------------------------ > > > > -- > > > > -- > > > > -------- > > > > Site24x7 APM Insight: Get Deep Visibility into Application > > > > Performance APM + Mobile APM + RUM: Monitor 3 App instances at > > > > just $35/Month Monitor end-to-end web transactions and take > > > > corrective actions now Troubleshoot faster and improve end-user > experience. > > Signup Now! > > > > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > > > > _______________________________________________ > > > > Opensaf-devel mailing list > > > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > > > Opensaf-devel@lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/opensaf-devel > > > > > > -------------------------------------------------------------------- > > > -- > > > -------- > > > Site24x7 APM Insight: Get Deep Visibility into Application > > > Performance APM + Mobile APM + RUM: Monitor 3 App instances at just > > > $35/Month Monitor end-to-end web transactions and take corrective > > > actions now Troubleshoot faster and improve end-user experience. > Signup Now! > > > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > > > _______________________________________________ > > > Opensaf-devel mailing list > > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > > Opensaf-devel@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/opensaf-devel > > > > ---------------------------------------------------------------------- > > -------- > > Site24x7 APM Insight: Get Deep Visibility into Application Performance > > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > > Monitor end-to-end web transactions and take corrective actions now > > Troubleshoot faster and improve end-user experience. Signup Now! > > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > > _______________________________________________ > > Opensaf-devel mailing list > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > > Opensaf-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/opensaf-devel > ------------------------------------------------------------------------------ > Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + > Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor > end-to-end web transactions and take corrective actions now Troubleshoot > faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > _______________________________________________ > Opensaf-devel mailing list > HYPERLINK "mailto:Opensaf-devel@lists.sourceforge.net" > Opensaf-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/opensaf-devel ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ Opensaf-devel mailing list Opensaf-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/opensaf-devel