Hi Nagu, From your test description TC#1, it says SU2 hosted on SC-2. And after SC-1 comes back, SU2 on PL-4 gets assignment. This description is symptom of su mapping issue which is addressed in patch 11_1.
But now looking at the trace file, SU2 is being hosted in PL-4 actually (not as in description). And errors relating SU-SI "record addition failed", "avd_ckpt_siass: ... does not exist",... are because the patches set you are applying missing #5 #6 #7 #8 (delayed failover) or concept patch (reboot transient state) amfd after headless needs something to adjust the transient states or reboot the whole PL. How's about if you have #5 #6 #7 #8 applied? Thanks, Minh On 04/03/16 17:45, Nagendra Kumar wrote: > Hi, > I have conducted the same 9 test cases sent on Mar 2 (in review > response) with the patches #1-#4along with attached patches(#9-#13). > > The summary of the results: All the 9 test cases have failed except in TC #2, > in which stopping PL-4 has worked. > > ====================================== > TC #1: Configuration(Comp recovery is comp failover, saAmfSutDefSUFailover as > false) and logs attached(New TC 1) in the ticket. > 1. Start SC-1, PL-3 and PL-4. SU1 Act on PL-3 and SU2 Standby on SC-2. > 2. Stop SC-1 and kill demo. It goes for comp failover as configured. Ideally, > node should reboot. > 3. Start SC-1. After cluster timer expires, PL-4 got the following error > messages: > > Mar 4 10:10:15 PM_PL-4 osafamfnd[10290]: CR SU-SI record addition failed, > SU= safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 : SI=safSi=AmfDemo,safApp=AmfDemo1 > Mar 4 10:10:15 PM_PL-4 osafamfnd[10290]: CR SU-SI record addition failed, > SU= safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 : > SI=safSi=AmfDemo1,safApp=AmfDemo1 > > There is no assignment given for SU1. SU2 has Standby assignments: > safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF > saAmfSISUHAState=ACTIVE(1) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,safApp=AmfDemo1 > saAmfSISUHAState=STANDBY(2) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > safSISU=safSu=PL-3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF > saAmfSISUHAState=ACTIVE(1) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF > saAmfSISUHAState=ACTIVE(1) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > safSISU=safSu=SC-1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF > saAmfSISUHAState=ACTIVE(1) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1,safApp=AmfDemo1 > saAmfSISUHAState=STANDBY(2) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > Other problems: a.) Further command for locking SU1/SU2 fails in SG unstable > error. > b.) Immlist if SU2 gives the below result, > Standby assignment it prints as 4, which is wrong: > saAmfSUNumCurrStandbySIs SA_UINT32_T 4 (0x4) > saAmfSUNumCurrActiveSIs SA_UINT32_T 0 (0x0) > c.) Even if SC-2 joins, and you do > failover/switchover of SC-1, still same as above. > > TC #2: After execution of TC #1, stop PL-3. In worst case, SU2 assignment > should change to Act, which is not happening. SU2 still holds Standby > assignment: > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1,safApp=AmfDemo1 > saAmfSISUHAState=STANDBY(2) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,safApp=AmfDemo1 > saAmfSISUHAState=STANDBY(2) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > Failure message same as above TC #1: > Mar 4 10:40:18 PM_PL-4 osafamfnd[12749]: CR SU-SI record addition failed, > SU= safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 : SI=safSi=AmfDemo,safApp=AmfDemo1 > Mar 4 10:40:18 PM_PL-4 osafamfnd[12749]: CR SU-SI record addition failed, > SU= safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 : > SI=safSi=AmfDemo1,safApp=AmfDemo1 > > But after stopping of PL-4, Assignments are gone, which is good. I am able to > lock/unlock the SU1. > The configuration and logs attached(New TC 2). > > TC #3: After TC #2(before stopping PL-4), start PL-3 and start SC-2. > > SU1 is instantiated, but no assignment and the same problem > as above. > > When stop PL-4, SU1 gets Act assignments, the following logs > comes at SC-2: > > Mar 4 10:59:22 PM_SC-2 osafamfd[11449]: ER avd_ckpt_siass: > safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 safSi=AmfDemo,safApp=AmfDemo1 does > not exist > Mar 4 10:59:22 PM_SC-2 osafamfd[11449]: ER avd_ckpt_siass: > safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 safSi=AmfDemo1,safApp=AmfDemo1 does > not exist > > > Start PL-4, SU2 gets Standby assignments and everything works fine after > that. The configuration and logs attached(New TC 3) in the ticket. > > TC #4: Similar problems exist in the following test cases: > > a.) Configuration same as TC #1 except saAmfSutDefSUFailover as true. > After killing demo, PL-3 went for reboot. > But the problem is the same as shown in TC #1. > The configuration and logs attached(New TC 4.a) in the ticket. > > b.) Configuration same as TC #1 except with saAmfCtDefRecoveryOnError as > 2 and saAmfCtDefDisableRestart as 1. > But the problem is the same as shown in TC #1, TC #2 and TC > #3. > The configuration and logs attached(New TC 4.b) in the ticket. > > c.) I didn't run it, but as I guess it will have same problem as 4.a. > > TC #5: Configuration same as TC #1 except with saAmfCtDefRecoveryOnError as > 2. Configuration and logs(New TC 5) attached in ticket. > 1. Start SC-1, PL-3 and PL-4. SU1 Act on PL-3 and SU2 Standby on PL-4. > 2. Stop SC-1 and kill demo. It goes for comp restart as configured. > 3. Start SC-1. After SC-1 comes up and before cluster timer expires, stop > PL-3: > > Even if PL-3 is stopped(see below PL-3 is not available), SU1 is still having > Act assignment and SU2 is having Standby assignment: > PM_SC-1:/home/nagu/views/staging # date > Fri Mar 4 11:26:21 IST 2016 > PM_SC-1:/home/nagu/views/staging # amf-state siass > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1,safApp=AmfDemo1 > saAmfSISUHAState=ACTIVE(1) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF > saAmfSISUHAState=ACTIVE(1) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,safApp=AmfDemo1 > saAmfSISUHAState=STANDBY(2) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,safApp=AmfDemo1 > saAmfSISUHAState=ACTIVE(1) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1,safApp=AmfDemo1 > saAmfSISUHAState=STANDBY(2) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF > saAmfSISUHAState=ACTIVE(1) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > safSISU=safSu=SC-1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF > saAmfSISUHAState=ACTIVE(1) > saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) > > TC #6: After TC #5, start PL-3: > SU1 is not given any assignment (may be because it exists in Amfd db): > Mar 4 11:30:00 PM_PL-3 osafamfnd[29869]: NO > 'safSu=PL-3,safSg=NoRed,safApp=OpenSAF' Presence State INSTANTIATING => > INSTANTIATED > Mar 4 11:30:00 PM_PL-3 osafamfnd[29869]: NO Assigning > 'safSi=NoRed4,safApp=OpenSAF' ACTIVE to > 'safSu=PL-3,safSg=NoRed,safApp=OpenSAF' > Mar 4 11:30:00 PM_PL-3 osafamfnd[29869]: NO Assigned > 'safSi=NoRed4,safApp=OpenSAF' ACTIVE to > 'safSu=PL-3,safSg=NoRed,safApp=OpenSAF' > Mar 4 11:30:00 PM_PL-3 osafamfnd[29869]: NO > 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State UNINSTANTIATED => > INSTANTIATING > Mar 4 11:30:00 PM_PL-3 opensafd: OpenSAF(5.0.M0 - 7282:4fbffe857512:) > services successfully started > Mar 4 11:30:00 PM_PL-3 amf_demo[29947]: > 'safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' started > Mar 4 11:30:00 PM_PL-3 osafamfnd[29869]: NO > 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State INSTANTIATING => > INSTANTIATED > Mar 4 11:30:00 PM_PL-3 amf_demo[29947]: HC started with AMF > Mar 4 11:30:00 PM_PL-3 amf_demo[29947]: Registered with AMF > Mar 4 11:30:00 PM_PL-3 amf_demo[29947]: Health check 1 > > TC #7: After TC #6: > > Lock SU1: Amfnd of PL-3 throws error: > Mar 4 11:53:50 PM_PL-3 osafamfnd[31064]: ER susi_assign_evh: > 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' has no assignments > > This is obvious because, Amfnd doesn't have any assignment. > SU1 admin state is locked, but SUSI is being shown on SU1. > > TC #8: After TC #7: > > Lock SU1, it throws error: > Mar 4 11:59:51.406386 osafamfd [8859:su.cc:1146] >> su_admin_op_cb: > 60129542146, 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1', 2 > Mar 4 11:59:51.406401 osafamfd [8859:imm.cc:1998] >> report_admin_op_error: > inv:60129542146, res:6, Error String: 'SG state is not stable' > > TC #9: Same as TC #6 except Configure saAmfCtDefRecoveryOnError as Node > Switchover/Failover/Failfast. > The problem reported in TC #4 exists. > > Thanks > -Nagu > >> -----Original Message----- >> From: Nagendra Kumar >> Sent: 02 March 2016 20:42 >> To: Minh Hon Chau; hans.nordeb...@ericsson.com; >> gary....@dektech.com.au; Praveen Malviya >> Cc: opensaf-devel@lists.sourceforge.net >> Subject: Re: [devel] [PATCH 01 of 15] amfd: Add support for cloud resilience >> at common libs [#1620] >> >> #1 I have applied patches #1 to #4 only. With this patches(not having patch >> #6), I thought to have passed most of the following tests, but they got >> failed(Listed below). >> >> >> >> I could not test other scenarios (including alarms and notifications), >> because >> I haven't applied patch #6. I think there should be a simple patch replacing >> patch #6, which handles transient state as 'reboot the node' if Amf finds >> SUSI >> in transient state on that node. >> >> I am attaching a concept patch(assignment_recovery.patch), which pass >> some of the scenarios and we are testing and enhancing it. >> >> As Praveen has suggested that we need to reboot the node which is >> undergoing in transient state to make it simple. >> >> This patch reduces complexity and maintainability. >> >> >> >> So, ACK for patch #1-#4 along with the attached patch. >> >> Please note that the attached patch has been created on patch #6 of yours, >> so please apply #1 to #4 and then #6 and then the attached patch. >> >> Currently the patch is for 2N red model. We are working to make for Nway >> Act and No red model (and possibly for Nway and NpM), we will publish it >> tomorrow. >> >> >> >> TC #1: >> >> Configuration(Comp recovery is comp failover, saAmfSutDefSUFailover as >> false) and logs attached(TC 1) in the ticket. >> >> 1. Start SC-1, PL-3 and PL-4. SU1 Act on PL-3 and SU2 Standby on SC-2. >> >> 2. Stop SC-1 and kill demo. It goes for comp failover as configured. Ideally, >> node should reboot. >> >> 3. Start SC-1. After cluster timer expires, PL-4 got the following error >> messages: >> >> >> >> Mar 2 08:01:15 PM_PL-4 osafamfnd[20050]: CR SU-SI record addition failed, >> SU= safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 : >> SI=safSi=AmfDemo,safApp=AmfDemo1 >> >> Mar 2 08:01:15 PM_PL-4 osafamfnd[20050]: CR SU-SI record addition failed, >> SU= safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 : >> SI=safSi=AmfDemo1,safApp=AmfDemo1 >> >> >> >> There is no assignment given for SU1. SU2 has Standby assignments: >> >> safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s >> afApp=AmfDemo1 >> >> saAmfSISUHAState=STANDBY(2) >> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) >> >> safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 >> ,safApp=AmfDemo1 >> >> saAmfSISUHAState=STANDBY(2) >> >> >> >> Other problems: a.) Further command for locking SU1/SU2 fails in SG >> unstable error. >> >> b.) Immlist if SU2 gives the below result, >> Standby >> assignment it prints as 4, which is wrong: >> >> saAmfSUNumCurrStandbySIs >> SA_UINT32_T >> 4 (0x4) >> >> saAmfSUNumCurrActiveSIs >> SA_UINT32_T 0 >> (0x0) >> >> c.) Even if SC-2 joins, and you do >> failover/switchover of SC- >> 1, still same as above. >> >> >> >> TC #2: After execution of TC #1, stop PL-3. In worst case, SU2 assignment >> should change to Act, which is not happening. After stopping of PL-4 also, >> the same problems as TC #1. logs attached(TC 2). >> >> >> >> TC #3: After TC #2, start PL-3 and start SC-2. >> >> SU1 is instantiated, but no assignment and the same problem >> as >> above. >> >> When stop PL-4, SU1 gets assignments, the following logs >> comes at >> SC-2: >> >> >> >> Mar 2 09:06:18 PM_SC-2 osafamfd[8518]: ER avd_ckpt_siass: >> safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 >> safSi=AmfDemo,safApp=AmfDemo1 does not exist >> >> Mar 2 09:06:18 PM_SC-2 osafamfd[8518]: ER avd_ckpt_siass: >> safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 >> safSi=AmfDemo1,safApp=AmfDemo1 does not exist >> >> Mar 2 09:06:21 PM_SC-2 kernel: [ 3290.784933] tipc: Resetting link >> <1.1.2:eth0-1.1.4:eth0>, peer not responding >> >> Mar 2 09:06:21 PM_SC-2 kernel: [ 3290.784947] tipc: Lost link <1.1.2:eth0- >> 1.1.4:eth0> on network plane A >> >> Mar 2 09:06:21 PM_SC-2 kernel: [ 3290.784956] tipc: Lost contact with >> <1.1.4> >> >> >> >> Start PL-4, SU2 gets Standby assignments and everything works fine after >> that. >> >> >> >> TC #4: Similar problems exist in the following test cases: >> >> a.) Configuration same as TC #1 except saAmfSutDefSUFailover as true. >> >> After killing demo, PL-3 went for reboot. >> >> But the problem is the same as shown in TC #1, TC #2 and TC >> #3. >> >> >> >> b.) Configuration same as TC #1 except with saAmfCtDefRecoveryOnError >> as 2 and saAmfCtDefDisableRestart as 1. >> >> But the problem is the same as shown in TC #1, TC #2 and TC >> #3. >> >> >> >> c.) Configuration same as TC #1 except with saAmfCtDefRecoveryOnError >> as 2 and saAmfCtDefDisableRestart as 1 and saAmfSutDefSUFailover as 1. >> >> After killing demo, PL-3 went for reboot. >> >> But the problem is the same as shown in TC #1, TC #2 and TC >> #3. >> >> >> >> TC #5: Configuration same as TC #1 except with >> saAmfCtDefRecoveryOnError as 2. Configuration and logs(TC 5) attached in >> ticket. >> >> 1. Start SC-1, PL-3 and PL-4. SU1 Act on PL-3 and SU2 Standby on SC-2. >> >> 2. Stop SC-1 and kill demo. It goes for comp restart as configured. >> >> 3. Start SC-1. After SC-1 comes up and before cluster timer expires, stop PL- >> 3: >> >> >> >> Even if PL-3 is stopped(see below PL-3 is not available), SU1 is still >> having Act >> assignment and SU2 is having Standby assignment: >> >> >> >> PM_SC-1:/home/nagu/views/staging # amf-state siass >> >> safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 >> ,safApp=AmfDemo1 >> >> saAmfSISUHAState=STANDBY(2) >> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) >> >> safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s >> afApp=AmfDemo1 >> >> saAmfSISUHAState=STANDBY(2) >> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) >> >> safSISU=safSu=SC- >> 1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF >> >> saAmfSISUHAState=ACTIVE(1) >> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) >> >> safSISU=safSu=PL- >> 4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF >> >> saAmfSISUHAState=ACTIVE(1) >> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) >> >> safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,s >> afApp=AmfDemo1 >> >> saAmfSISUHAState=ACTIVE(1) >> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) >> >> safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC- >> 2N,safApp=OpenSAF >> >> saAmfSISUHAState=ACTIVE(1) >> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) >> >> safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1 >> ,safApp=AmfDemo1 >> >> saAmfSISUHAState=ACTIVE(1) >> >> saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1) >> >> >> >> TC #6: After TC #5, start PL-3: >> >> SU1 is not given any assignment (may be because it exists in Amfd db): >> >> Mar 2 14:22:06 PM_PL-3 osafamfwd[8318]: Started >> >> Mar 2 14:22:06 PM_PL-3 osafamfnd[8259]: NO 'safSu=PL- >> 3,safSg=NoRed,safApp=OpenSAF' Presence State INSTANTIATING => >> INSTANTIATED >> >> Mar 2 14:22:06 PM_PL-3 osafamfnd[8259]: NO Assigning >> 'safSi=NoRed2,safApp=OpenSAF' ACTIVE to 'safSu=PL- >> 3,safSg=NoRed,safApp=OpenSAF' >> >> Mar 2 14:22:06 PM_PL-3 osafamfnd[8259]: NO Assigned >> 'safSi=NoRed2,safApp=OpenSAF' ACTIVE to 'safSu=PL- >> 3,safSg=NoRed,safApp=OpenSAF' >> >> Mar 2 14:22:06 PM_PL-3 osafamfnd[8259]: NO >> 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State >> UNINSTANTIATED => INSTANTIATING >> >> Mar 2 14:22:06 PM_PL-3 opensafd: OpenSAF(5.0.M0 - 7282:4fbffe857512:) >> services successfully started >> >> Mar 2 14:22:06 PM_PL-3 amf_demo[8337]: >> 'safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' >> started >> >> Mar 2 14:22:06 PM_PL-3 osafamfnd[8259]: NO >> 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State >> INSTANTIATING => INSTANTIATED >> >> Mar 2 14:22:06 PM_PL-3 amf_demo[8337]: HC started with AMF >> >> >> >> TC #7: After TC #6: >> >> Lock SU1: Amfnd of PL-3 throws error: >> >> Mar 2 14:23:57 PM_PL-3 osafamfnd[8259]: ER susi_assign_evh: >> 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' has no assignments >> >> >> >> This is obvious because, Amfnd doesn't have any assignment. >> >> SU1 admin state is locked, but SUSI is being shown on SU1. >> >> >> >> TC #8: After TC #7: >> >> Lock SU1, it throws error: >> >> Admin operation is already going on >> (su'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 >> >> >> >> TC #9: Same as TC #6 except Configure saAmfCtDefRecoveryOnError as >> Node Switchover/Failover/Failfast. >> >> The problem reported in TC #4 exists. >> >> >> >> Thanks >> >> -Nagu >> >> >> >>> -----Original Message----- >>> From: Minh Hon Chau [mailto:minh.c...@dektech.com.au] >>> Sent: 25 February 2016 14:14 >>> To: hans.nordeb...@ericsson.com; gary....@dektech.com.au; Nagendra >>> Kumar; Praveen Malviya; minh.c...@dektech.com.au >>> Cc: opensaf-devel@lists.sourceforge.net >>> Subject: [PATCH 01 of 15] amfd: Add support for cloud resilience at >> common >> >>> libs [#1620] ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ Opensaf-devel mailing list Opensaf-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/opensaf-devel
