Hi Mahesh,
osafimmloadd/osafimmpbed are not a part of imm tools (immcfg, immadm, ...).
They are forked by immnd on SCs (before headless enhancement).
Currently, they are included in 'imm-director' package (which is installed on
SC only).
We just need to move those two to 'imm-nodedirector' package and the problem
will be fixed.
diff --git a/opensaf.spec.in b/opensaf.spec.in
--- a/opensaf.spec.in
+++ b/opensaf.spec.in
@@ -1115,8 +1115,6 @@ fi
%config(noreplace) %{_pkgsysconfdir}/immd.conf
%{_pkglibdir}/osafimmd
%{_pkgclcclidir}/osaf-immd
-%{_pkglibdir}/osafimmloadd
-%{_pkglibdir}/osafimmpbed
%if %is_immxml
%{_pkgimmxmldir}/baseimm.py*
%{_pkgimmxmldir}/README
@@ -1142,7 +1140,8 @@ fi
%config(noreplace) %{_pkgsysconfdir}/immnd.conf
%{_pkglibdir}/osafimmnd
%{_pkgclcclidir}/osaf-immnd
-
+%{_pkglibdir}/osafimmloadd
+%{_pkglibdir}/osafimmpbed
%if %is_ais_plm
BR,
Hung Nguyen - DEK Technologies
--------------------------------------------------------------------------------
From: Mahesh Valla mahesh.va...@oracle.com
Sent: Wednesday, February 17, 2016 3:07PM
To: Zoran Milinkovic, Opensaf-devel, Neelakanta Reddy
zoran.milinko...@ericsson.com, opensaf-devel@lists.sourceforge.net,
reddy.neelaka...@oracle.com
Cc:
Subject: Re: [devel] [PATCH 1 of 5] imm: add common library support for cloud
resilience feature [#1625]
Hi Zoran,
Thanks, just copying osafimmloadd/osafimmpbed manually to payload
resolved the issue, can you Please send the patch to separate
Imm tools in to a separate RPM and make part of Payload.
-AVM
On 2/17/2016 1:27 PM, Zoran Milinkovic wrote:
> Hi Mahesh,
>
> The problem is in executing osafimmloadd. Check if the file is
> corrupted or is missing, as Hung stated.
>
> BR,
>
> Zoran
>
> *From:*A V Mahesh [mailto:mahesh.va...@oracle.com]
> *Sent:* Wednesday, February 17, 2016 6:28 AM
> *To:* opensaf-devel@lists.sourceforge.net; Zoran Milinkovic;
> reddy.neelaka...@oracle.com
> *Subject:* Re: [devel] [PATCH 1 of 5] imm: add common library support
> for cloud resilience feature [#1625]
>
> Hi Zoran/Neel,
>
> I am still facing the problem , the only difference between other
> successful testing and my testing is ,
> i am using RPMS and others are using `make install`
>
>
> ==================================
> Feb 17 10:52:14 SC-1 osafimmd[9450]: NO ACT: New Epoch for IMMND
> process at node 2030f old epoch: 20 new epoch:21
> Feb 17 10:52:14 SC-1 osafimmd[9450]: NO Attached Nodes:4 Accepted
> nodes:4 KnownVeteran:0 doReply:0
> Feb 17 10:52:14 SC-1 osafimmnd[9461]: NO ABT REQUESTING SYNC
> Feb 17 10:52:14 SC-1 osafimmd[9450]: WA IMMND on controller (not
> currently coord) requests sync
> Feb 17 10:52:14 SC-1 osafimmd[9450]: NO Sc Absence Allowed is
> configured (900) => IMMND coord at payload node:2040f dest567416481644548
> Feb 17 10:52:14 SC-1 osafimmd[9450]: NO Node 2010f request sync
> sync-pid:9461 epoch:0
> Feb 17 10:52:14 SC-1 osafimmnd[9461]: NO SERVER STATE:
> IMM_SERVER_LOADING_PENDING --> IMM_SERVER_SYNC_PENDING
> Feb 17 10:52:14 SC-1 osafimmnd[9461]: NO NODE STATE-> IMM_NODE_ISOLATED
> Feb 17 10:52:14 SC-1 osafimmd[9450]: WA IMMND on controller (not
> currently coord) requests sync
> Feb 17 10:52:14 SC-1 osafimmd[9450]: NO Sc Absence Allowed is
> configured (900) => IMMND coord at payload node:2040f dest567416481644548
> Feb 17 10:52:14 SC-1 osafimmd[9450]: NO Node 2020f request sync
> sync-pid:8283 epoch:0
> Feb 17 10:52:18 SC-1 osafimmd[9450]: NO Successfully announced sync.
> New ruling epoch:22
> Feb 17 10:52:18 SC-1 osafimmnd[9461]: NO NODE STATE-> IMM_NODE_W_AVAILABLE
> Feb 17 10:52:18 SC-1 osafimmnd[9461]: NO SERVER STATE:
> IMM_SERVER_SYNC_PENDING --> IMM_SERVER_SYNC_CLIENT
> Feb 17 10:52:18 SC-1 osafimmd[9450]: NO ACT: New Epoch for IMMND
> process at node 2040f old epoch: 21 new epoch:22
> Feb 17 10:52:18 SC-1 osafimmd[9450]: NO Attached Nodes:4 Accepted
> nodes:4 KnownVeteran:0 doReply:0
> Feb 17 10:52:18 SC-1 osafimmd[9450]: WA Successfully aborted sync.
> Epoch:22
> Feb 17 10:52:18 SC-1 osafimmnd[9461]: WA Global ABORT SYNC received
> for epoch 22
> Feb 17 10:52:18 SC-1 osafimmnd[9461]: WA SERVER STATE:
> IMM_SERVER_SYNC_CLIENT --> IMM_SERVER_LOADING_PENDING (sync aborted)
> Feb 17 10:52:18 SC-1 osafimmnd[9461]: NO NODE STATE-> IMM_NODE_UNKNOW 2624
> Feb 17 10:52:18 SC-1 osafimmnd[9461]: NO Abort sync: Discarding synced
> objects
> Feb 17 10:52:18 SC-1 osafimmnd[9461]: NO Abort sync: Discarding synced
> classes
> Feb 17 10:52:18 SC-1 osafimmd[9450]: NO ACT: New Epoch for IMMND
> process at node 2030f old epoch: 21 new epoch:22
> Feb 17 10:52:18 SC-1 osafimmd[9450]: NO Attached Nodes:4 Accepted
> nodes:4 KnownVeteran:0 doReply:0
> Feb 17 10:52:18 SC-1 osafimmnd[9461]: NO ABT REQUESTING SYNC
> Feb 17 10:52:18 SC-1 osafimmnd[9461]: NO SERVER STATE:
> IMM_SERVER_LOADING_PENDING --> IMM_SERVER_SYNC_PENDING
> Feb 17 10:52:18 SC-1 osafimmd[9450]: WA IMMND on controller (not
> currently coord) requests sync
> Feb 17 10:52:18 SC-1 osafimmnd[9461]: NO NODE STATE-> IMM_NODE_ISOLATED
> Feb 17 10:52:18 SC-1 osafimmd[9450]: NO Sc Absence Allowed is
> configured (900) => IMMND coord at payload node:2040f dest567416481644548
> Feb 17 10:52:18 SC-1 osafimmd[9450]: NO Node 2010f request sync
> sync-pid:9461 epoch:0
> Feb 17 10:52:18 SC-1 osafimmd[9450]: WA IMMND on controller (not
> currently coord) requests sync
> Feb 17 10:52:18 SC-1 osafimmd[9450]: NO Sc Absence Allowed is
> configured (900) => IMMND coord at payload node:2040f dest567416481644548
> Feb 17 10:52:18 SC-1 osafimmd[9450]: NO Node 2020f request sync
> sync-pid:8283 epoch:0
>
> Feb 17 10:52:04 SC-2 osafimmnd[8283]: NO NODE STATE-> IMM_NODE_W_AVAILABLE
> Feb 17 10:52:04 SC-2 osafimmnd[8283]: NO SERVER STATE:
> IMM_SERVER_SYNC_PENDING --> IMM_SERVER_SYNC_CLIENT
> Feb 17 10:52:04 SC-2 osafimmd[8272]: NO SBY: New Epoch for IMMND
> process at node 2040f old epoch: 17 new epoch:18
> Feb 17 10:52:04 SC-2 osafimmd[8272]: NO IMMND coord at 2040f
> Feb 17 10:52:04 SC-2 osafimmd[8272]: NO IMMND coord at 2040f
> Feb 17 10:52:04 SC-2 osafimmnd[8283]: WA Global ABORT SYNC received
> for epoch 18
> Feb 17 10:52:04 SC-2 osafimmnd[8283]: WA SERVER STATE:
> IMM_SERVER_SYNC_CLIENT --> IMM_SERVER_LOADING_PENDING (sync aborted)
> Feb 17 10:52:04 SC-2 osafimmnd[8283]: NO NODE STATE-> IMM_NODE_UNKNOW 2624
> Feb 17 10:52:04 SC-2 osafimmnd[8283]: NO Abort sync: Discarding synced
> objects
> Feb 17 10:52:04 SC-2 osafimmnd[8283]: NO Abort sync: Discarding synced
> classes
> Feb 17 10:52:04 SC-2 osafimmd[8272]: NO SBY: New Epoch for IMMND
> process at node 2030f old epoch: 17 new epoch:18
> Feb 17 10:52:04 SC-2 osafimmnd[8283]: NO ABT REQUESTING SYNC
> Feb 17 10:52:04 SC-2 osafimmnd[8283]: NO SERVER STATE:
> IMM_SERVER_LOADING_PENDING --> IMM_SERVER_SYNC_PENDING
> Feb 17 10:52:04 SC-2 osafimmnd[8283]: NO NODE STATE-> IMM_NODE_ISOLATED
> Feb 17 10:52:07 SC-2 osafimmd[8272]: NO SBY: Ruling epoch noted as:19
> Feb 17 10:52:07 SC-2 osafimmd[8272]: NO IMMND coord at 2040f
> Feb 17 10:52:07 SC-2 osafimmnd[8283]: NO NODE STATE-> IMM_NODE_W_AVAILABLE
> Feb 17 10:52:07 SC-2 osafimmnd[8283]: NO SERVER STATE:
> IMM_SERVER_SYNC_PENDING --> IMM_SERVER_SYNC_CLIENT
> Feb 17 10:52:07 SC-2 osafimmd[8272]: NO SBY: New Epoch for IMMND
> process at node 2040f old epoch: 18 new epoch:19
> Feb 17 10:52:07 SC-2 osafimmd[8272]: NO IMMND coord at 2040f
> Feb 17 10:52:07 SC-2 osafimmd[8272]: NO IMMND coord at 2040f
> Feb 17 10:52:07 SC-2 osafimmnd[8283]: WA Global ABORT SYNC received
> for epoch 19
> Feb 17 10:52:07 SC-2 osafimmnd[8283]: WA SERVER STATE:
> IMM_SERVER_SYNC_CLIENT --> IMM_SERVER_LOADING_PENDING (sync aborted)
> Feb 17 10:52:07 SC-2 osafimmnd[8283]: NO NODE STATE-> IMM_NODE_UNKNOW 2624
> Feb 17 10:52:07 SC-2 osafimmnd[8283]: NO Abort sync: Discarding synced
> objects
> Feb 17 10:52:07 SC-2 osafimmnd[8283]: NO Abort sync: Discarding synced
> classes
> Feb 17 10:52:07 SC-2 osafimmd[8272]: NO SBY: New Epoch for IMMND
> process at node 2030f old epoch: 18 new epoch:19
> Feb 17 10:52:07 SC-2 osafimmnd[8283]: NO ABT REQUESTING SYNC
> Feb 17 10:52:07 SC-2 osafimmnd[8283]: NO SERVER STATE:
> IMM_SERVER_LOADING_PENDING --> IMM_SERVER_SYNC_PENDING
> Feb 17 10:52:07 SC-2 osafimmnd[8283]: NO NODE STATE-> IMM_NODE_ISOLATED
>
>
> Feb 17 10:51:12 PL-3 osafimmnd[5293]: NO NODE STATE->
> IMM_NODE_FULLY_AVAILABLE (2610)
> Feb 17 10:51:17 PL-3 osafimmnd[5293]: NO NODE STATE-> IMM_NODE_R_AVAILABLE
> Feb 17 10:51:17 PL-3 osafimmnd[5293]: WA Global ABORT SYNC received
> for epoch 7
> Feb 17 10:51:17 PL-3 osafimmnd[5293]: NO Epoch set to 7 in ImmModel
> Feb 17 10:51:17 PL-3 osafimmnd[5293]: NO NODE STATE->
> IMM_NODE_FULLY_AVAILABLE (2610)
> Feb 17 10:51:21 PL-3 osafimmnd[5293]: NO NODE STATE-> IMM_NODE_R_AVAILABLE
> Feb 17 10:51:21 PL-3 osafimmnd[5293]: WA Global ABORT SYNC received
> for epoch 8
> Feb 17 10:51:21 PL-3 osafimmnd[5293]: NO Epoch set to 8 in ImmModel
> Feb 17 10:51:21 PL-3 osafimmnd[5293]: NO NODE STATE->
> IMM_NODE_FULLY_AVAILABLE (2610)
> Feb 17 10:51:25 PL-3 osafimmnd[5293]: NO NODE STATE-> IMM_NODE_R_AVAILABLE
> Feb 17 10:51:25 PL-3 osafimmnd[5293]: WA Global ABORT SYNC received
> for epoch 9
> Feb 17 10:51:25 PL-3 osafimmnd[5293]: NO Epoch set to 9 in ImmModel
> Feb 17 10:51:25 PL-3 osafimmnd[5293]: NO NODE STATE->
> IMM_NODE_FULLY_AVAILABLE (2610)
> Feb 17 10:51:30 PL-3 osafimmnd[5293]: NO NODE STATE-> IMM_NODE_R_AVAILABLE
> Feb 17 10:51:30 PL-3 osafimmnd[5293]: WA Global ABORT SYNC received
> for epoch 10
> Feb 17 10:51:30 PL-3 osafimmnd[5293]: NO Epoch set to 10 in ImmModel
> Feb 17 10:51:30 PL-3 osafimmnd[5293]: NO NODE STATE->
> IMM_NODE_FULLY_AVAILABLE (2610)
> Feb 17 10:51:34 PL-3 osafimmnd[5293]: NO NODE STATE-> IMM_NODE_R_AVAILABLE
> Feb 17 10:51:34 PL-3 osafimmnd[5293]: WA Global ABORT SYNC received
> for epoch 11
> Feb 17 10:51:34 PL-3 osafimmnd[5293]: NO Epoch set to 11 in ImmModel
> Feb 17 10:51:34 PL-3 osafimmnd[5293]: NO NODE STATE->
> IMM_NODE_FULLY_AVAILABLE (2610)
> Feb 17 10:51:38 PL-3 osafimmnd[5293]: NO NODE STATE-> IMM_NODE_R_AVAILABLE
> Feb 17 10:51:38 PL-3 osafimmnd[5293]: WA Global ABORT SYNC received
> for epoch 12
> Feb 17 10:51:38 PL-3 osafimmnd[5293]: NO Epoch set to 12 in ImmModel
> Feb 17 10:51:38 PL-3 osafimmnd[5293]: NO NODE STATE->
> IMM_NODE_FULLY_AVAILABLE (2610)
> Feb 17 10:51:43 PL-3 osafimmnd[5293]: NO NODE STATE-> IMM_NODE_R_AVAILABLE
>
>
>
> Feb 17 10:51:20 PL-4 osafimmnd[5276]: NO NODE STATE-> IMM_NODE_R_AVAILABLE
> Feb 17 10:51:20 PL-4 osafimmnd[5381]: ER osafimmnd failed to exec
> sync, error 2, exiting
> Feb 17 10:51:20 PL-4 osafimmnd[5276]: ER SYNC APPARENTLY FAILED status:1
> Feb 17 10:51:20 PL-4 osafimmnd[5276]: NO -SERVER STATE:
> IMM_SERVER_SYNC_SERVER --> IMM_SERVER_READY
> Feb 17 10:51:20 PL-4 osafimmnd[5276]: NO NODE STATE->
> IMM_NODE_FULLY_AVAILABLE (2610)
> Feb 17 10:51:20 PL-4 osafimmnd[5276]: NO Epoch set to 8 in ImmModel
> Feb 17 10:51:20 PL-4 osafimmnd[5276]: NO Coord broadcasting
> ABORT_SYNC, epoch:8
> Feb 17 10:51:20 PL-4 osafimmnd[5276]: WA Global ABORT SYNC received
> for epoch 8
> Feb 17 10:51:24 PL-4 osafimmnd[5276]: NO Announce sync, epoch:9
> Feb 17 10:51:24 PL-4 osafimmnd[5276]: NO SERVER STATE:
> IMM_SERVER_READY --> IMM_SERVER_SYNC_SERVER
> Feb 17 10:51:24 PL-4 osafimmnd[5276]: NO NODE STATE-> IMM_NODE_R_AVAILABLE
> Feb 17 10:51:24 PL-4 osafimmnd[5386]: ER osafimmnd failed to exec
> sync, error 2, exiting
> Feb 17 10:51:25 PL-4 osafimmnd[5276]: ER SYNC APPARENTLY FAILED status:1
> Feb 17 10:51:25 PL-4 osafimmnd[5276]: NO -SERVER STATE:
> IMM_SERVER_SYNC_SERVER --> IMM_SERVER_READY
> Feb 17 10:51:25 PL-4 osafimmnd[5276]: NO NODE STATE->
> IMM_NODE_FULLY_AVAILABLE (2610)
> Feb 17 10:51:25 PL-4 osafimmnd[5276]: NO Epoch set to 9 in ImmModel
> Feb 17 10:51:25 PL-4 osafimmnd[5276]: NO Coord broadcasting
> ABORT_SYNC, epoch:9
> Feb 17 10:51:25 PL-4 osafimmnd[5276]: WA Global ABORT SYNC received
> for epoch 9
> Feb 17 10:51:29 PL-4 osafimmnd[5276]: NO Announce sync, epoch:10
> Feb 17 10:51:29 PL-4 osafimmnd[5276]: NO SERVER STATE:
> IMM_SERVER_READY --> IMM_SERVER_SYNC_SERVER
> Feb 17 10:51:29 PL-4 osafimmnd[5276]: NO NODE STATE-> IMM_NODE_R_AVAILABLE
> Feb 17 10:51:29 PL-4 osafimmnd[5387]: ER osafimmnd failed to exec
> sync, error 2, exiting
> Feb 17 10:51:29 PL-4 osafimmnd[5276]: ER SYNC APPARENTLY FAILED status:1
> Feb 17 10:51:29 PL-4 osafimmnd[5276]: NO -SERVER STATE:
> IMM_SERVER_SYNC_SERVER --> IMM_SERVER_READY
> Feb 17 10:51:29 PL-4 osafimmnd[5276]: NO NODE STATE->
> IMM_NODE_FULLY_AVAILABLE (2610)
> Feb 17 10:51:29 PL-4 osafimmnd[5276]: NO Epoch set to 10 in ImmModel
> Feb 17 10:51:29 PL-4 osafimmnd[5276]: NO Coord broadcasting
> ABORT_SYNC, epoch:10
> Feb 17 10:51:29 PL-4 osafimmnd[5276]: WA Global ABORT SYNC received
> for epoch 10
> ==============================================
>
> On 2/12/2016 10:53 AM, A V Mahesh wrote:
>
> Hi Zoran/Neel,
>
> As part of reviewing/testing CPSV headless state V2 [#1621] , we
> bring down the both SC`s and
> bring up both SC`s in reverse order ( old active as new standby ,
> old standby as new active)
> which is piratical case where you cannot control heads joining
> sequence,
> as a result heads never joined the cluster and cluster went for
> reboot.
>
> I s this supported or not in headless state ?
>
> ==============================================
> Feb 12 10:41:13 SC-2 osafimmnd[30074]: NO SERVER STATE:
> IMM_SERVER_SYNC_PENDING --> IMM_SERVER_SYNC_CLIENT
> Feb 12 10:41:13 SC-2 osafimmd[29890]: NO SBY: New Epoch for IMMND
> process at node 2040f old epoch: 198 new epoch:199
> Feb 12 10:41:13 SC-2 osafimmd[29890]: NO IMMND coord at 2040f
> Feb 12 10:41:13 SC-2 osafimmd[29890]: NO IMMND coord at 2040f
> Feb 12 10:41:13 SC-2 osafimmnd[30074]: WA Global ABORT SYNC
> received for epoch 199
> Feb 12 10:41:13 SC-2 osafimmnd[30074]: WA SERVER STATE:
> IMM_SERVER_SYNC_CLIENT --> IMM_SERVER_LOADING_PENDING (sync aborted)
> Feb 12 10:41:13 SC-2 osafimmnd[30074]: NO NODE STATE->
> IMM_NODE_UNKNOW 2621
> Feb 12 10:41:13 SC-2 osafimmnd[30074]: NO Abort sync: Discarding
> synced objects
> Feb 12 10:41:13 SC-2 osafimmnd[30074]: NO Abort sync: Discarding
> synced classes
> Feb 12 10:41:13 SC-2 osafimmd[29890]: NO SBY: New Epoch for IMMND
> process at node 2030f old epoch: 198 new epoch:199
> Feb 12 10:41:13 SC-2 osafimmnd[30074]: NO ABT REQUESTING SYNC
> Feb 12 10:41:13 SC-2 osafimmnd[30074]: NO SERVER STATE:
> IMM_SERVER_LOADING_PENDING --> IMM_SERVER_SYNC_PENDING
> Feb 12 10:41:13 SC-2 osafimmnd[30074]: NO NODE STATE->
> IMM_NODE_ISOLATED
> Feb 12 10:41:17 SC-2 kernel: [71593.180556] tipc: Resetting link
> <1.1.2:eth2-1.1.3:eth1>, changeover initiated by peer
> Feb 12 10:41:17 SC-2 kernel: [71593.180562] tipc: Lost link
> <1.1.2:eth2-1.1.3:eth1> on network plane B
> Feb 12 10:41:17 SC-2 kernel: [71593.192851] tipc: Resetting link
> <1.1.2:eth2-1.1.4:eth2>, changeover initiated by peer
> Feb 12 10:41:17 SC-2 kernel: [71593.192857] tipc: Lost link
> <1.1.2:eth2-1.1.4:eth2> on network plane B
> Feb 12 10:41:19 SC-2 osafimmd[29890]: WA Message count:11144 + 1
> != 11144
> Feb 12 10:41:19 SC-2 kernel: [71595.365041] tipc: Resetting link
> <1.1.2:eth3-1.1.3:eth4>, peer not responding
> Feb 12 10:41:19 SC-2 kernel: [71595.365047] tipc: Lost link
> <1.1.2:eth3-1.1.3:eth4> on network plane A
> Feb 12 10:41:19 SC-2 kernel: [71595.365051] tipc: Lost contact
> with <1.1.3>
> Feb 12 10:41:19 SC-2 kernel: [71595.372092] tipc: Resetting link
> <1.1.2:eth3-1.1.4:eth0>, peer not responding
> Feb 12 10:41:19 SC-2 kernel: [71595.372100] tipc: Lost link
> <1.1.2:eth3-1.1.4:eth0> on network plane A
> Feb 12 10:41:19 SC-2 kernel: [71595.372105] tipc: Lost contact
> with <1.1.4>
> Feb 12 10:41:19 SC-2 osafimmd[29890]: ER Standby IMMD recieved
> reset message. All IMMNDs will restart.
> Feb 12 10:41:19 SC-2 osafimmd[29890]: ER IMM RELOAD with NO
> persistent back end => ensure cluster restart by IMMD exit at both
> SCs, exiting
> Feb 12 10:41:20 SC-2 osafimmnd[30074]: WA SC Absence IS
> allowed:900 IMMD service is DOWN
> Feb 12 10:41:20 SC-2 osafimmnd[30074]: WA IMMND was being synced
> or loaded (6), has to restart
> Feb 12 10:41:20 SC-2 opensafd[29839]: ER Could Not RESPAWN IMMND
> Feb 12 10:41:20 SC-2 opensafd[29839]: ER Failed DESC:IMMND
> Feb 12 10:41:20 SC-2 opensafd[29839]: ER Trying To RESPAWN
> /usr/lib64/opensaf/clc-cli/osaf-immnd attempt #2
> Feb 12 10:41:20 SC-2 opensafd[29839]: ER Sending SIGABRT to IMMND,
> pid=30074, (origin parent pid=30068)
> Feb 12 10:41:35 SC-2 osafimmnd[30187]: Started
> ==============================================
>
>
>
> On 2/3/2016 5:05 PM, Hung Nguyen wrote:
>
> Hi Zoran,
>
> Please find my comments inline.
>
>
> BR,
>
> Hung Nguyen - DEK Technologies
>
>
>
> --------------------------------------------------------------------------------
>
> From: Zoran Milinkovic zoran.milinko...@ericsson.com
> <mailto:zoran.milinko...@ericsson.com>
> Sent: Tuesday, December 22, 2015 9:14PM
> To: Neelakanta Reddy
> reddy.neelaka...@oracle.com <mailto:reddy.neelaka...@oracle.com>
> Cc: Opensaf-devel
> opensaf-devel@lists.sourceforge.net
> <mailto:opensaf-devel@lists.sourceforge.net>
> Subject: [devel] [PATCH 1 of 5] imm: add common library support
> for cloud resilience feature [#1625]
>
>
> osaf/libs/common/immsv/immpbe_dump.cc | 378
> +++++++++++++++++++++++++-
> osaf/libs/common/immsv/immsv_evt.c | 43 ++
> osaf/libs/common/immsv/include/immpbe_dump.hh | 2 +-
> osaf/libs/common/immsv/include/immsv_api.h | 1 +
> osaf/libs/common/immsv/include/immsv_evt.h | 17 +-
> 5 files changed, 430 insertions(+), 11 deletions(-)
>
>
> The patch contains IMM common library code for cloud resilience
> feature
>
> diff --git a/osaf/libs/common/immsv/immpbe_dump.cc
> b/osaf/libs/common/immsv/immpbe_dump.cc
> --- a/osaf/libs/common/immsv/immpbe_dump.cc
> +++ b/osaf/libs/common/immsv/immpbe_dump.cc
> @@ -1185,6 +1185,359 @@ bailout:
> exit(1);
> }
> +static bool verifyAttributeValuePBE(const char *attrVal,
> + SaImmAttrValueT value, SaImmValueTypeT attrType) {
> +
> + if(!attrVal && !value) {
> + return true;
> + }
> + if(!attrVal || !value) {
> + return false;
> + }
> +
> + switch(attrType) {
> + case SA_IMM_ATTR_SADOUBLET: {
> + char val[50];
> + snprintf(val, 50, "%.15g", *(double *)value);
> + return strtod(attrVal, NULL) == strtod(val, NULL);
> + }
> +
> + case SA_IMM_ATTR_SAFLOATT:
> + return strtof(attrVal, NULL) == *(float *)value;
> +
> + case SA_IMM_ATTR_SAUINT32T:
> + return strtoul(attrVal, NULL, 10) == *(SaUint32T
> *)value;
> +
> + case SA_IMM_ATTR_SAINT32T:
> + return strtol(attrVal, NULL, 10) == *(SaInt32T *)value;
> +
> + case SA_IMM_ATTR_SAUINT64T:
> + return strtoull(attrVal, NULL, 10) == *(SaUint64T
> *)value;
> +
> + case SA_IMM_ATTR_SATIMET:
> + case SA_IMM_ATTR_SAINT64T:
> + return strtoll(attrVal, NULL, 10) == *(SaInt64T *)value;
> +
> + case SA_IMM_ATTR_SASTRINGT:
> + return strcmp(attrVal, *(SaStringT *)value) == 0;
> +
> + case SA_IMM_ATTR_SANAMET:
> + return strcmp(attrVal,
> osaf_extended_name_borrow((SaNameT *)value)) == 0;
> +
> + case SA_IMM_ATTR_SAANYT: {
> + std::ostringstream ost;
> + SaAnyT *any = (SaAnyT *)value;
> +
> + for (unsigned int i = 0; i < any->bufferSize; i++) {
> + ost << std::hex
> + << (((int)any->bufferAddr[i] < 0x10)?
> "0":"")
> + << (int)any->bufferAddr[i];
> + }
> +
> + return strcmp(ost.str().c_str(), attrVal) == 0;
> + }
> +
> + default:
> + LOG_ER("UNKNOWN ATTRIBUTE TYPE: %d", attrType);
> + abort();
> + }
> +
> + return false;
> +}
> +
> +static bool verifyObjectAttributePBE(const char *attrVal,
> SaImmAttrValuesT_2 *attr) {
> + unsigned int i;
> +
> + if(!attrVal && attr->attrValuesNumber == 0) {
> + return true;
> + }
> +
> + for(i=0; i<attr->attrValuesNumber; i++) {
> + if(verifyAttributeValuePBE(attrVal, attr->attrValues[i],
> attr->attrValueType)) {
> + return true;
> + }
> + }
> +
> + return false;
> +}
> +
> +static bool verifyClassObjectsPBE(std::string classNameString,
> + SaImmHandleT immHandle,
> + void* db_handle)
> +{
> + sqlite3* dbHandle = (sqlite3 *) db_handle;
> + SaImmSearchHandleT searchHandle = 0;
> + SaImmSearchParametersT_2 searchParam;
> + const char *className = classNameString.c_str();
> + SaAisErrorT err;
> + std::string sql;
> + char **result = NULL;
> + int nrow;
> + int ncol;
> + char *errmsg;
> + int n, c, r, i;
> + int attrPos;
> + int objectCounter = 0;
> + std::map<std::string, int> rdnMap; // <RDN, row in the
> result>
> + std::map<std::string, int>::iterator rmi;
> + SaStringT objectName;
> + SaImmAttrValuesT_2 **attributes;
> + SaImmAttrDefinitionT_2 **attrDefinitions = NULL;
> + SaImmClassCategoryT classCategory;
> + int rdnAttrDefIndex = -1;
> + bool ok = true;
> + int rc;
> +
> + TRACE_ENTER();
> +
> + err = saImmOmClassDescriptionGet_2(immHandle,
> (SaImmClassNameT)className, &classCategory, &attrDefinitions);
> + if(err != SA_AIS_OK) {
> + LOG_ER("Cannot fetch class description for class '%s':
> %d", className, err);
> + TRACE_LEAVE();
> + return false;
> + }
> +
> + // Skip runtime classes
> + if(classCategory == SA_IMM_CLASS_RUNTIME) {
> + saImmOmClassDescriptionMemoryFree_2(immHandle,
> attrDefinitions);
> + TRACE_LEAVE();
> + return true;
> + }
> +
> + for(i=0; attrDefinitions[i]; i++) {
> + if(attrDefinitions[i]->attrFlags & SA_IMM_ATTR_RDN) {
> + rdnAttrDefIndex = i;
> + }
> + }
> +
> + if(attrDefinitions[i]) {
> + // This should never happen
> + LOG_ER("Cannot find RDN attribute in search result for
> class '%s'", className);
> + abort();
> + }
> +
> + searchParam.searchOneAttr.attrName =
> (SaImmAttrNameT)"SaImmAttrClassName";
> + searchParam.searchOneAttr.attrValue = &className;
> + searchParam.searchOneAttr.attrValueType = SA_IMM_ATTR_SASTRINGT;
> +
> + err = saImmOmSearchInitialize_2(immHandle, NULL, SA_IMM_SUBTREE,
> + SA_IMM_SEARCH_ONE_ATTR | SA_IMM_SEARCH_GET_ALL_ATTR |
> SA_IMM_SEARCH_PERSISTENT_ATTRS,
> + &searchParam, NULL, &searchHandle);
> + if(err != SA_AIS_OK) {
> + LOG_ER("Cannot fetch objects of class '%s' from IMM: %d",
> className, err);
> + goto bailout;
> + }
> +
> + sql = "SELECT o.dn, c.* FROM objects o, ";
> + sql.append(className);
> + sql.append(" c WHERE o.obj_id = c.obj_id");
> + rc = sqlite3_get_table(dbHandle, sql.c_str(), &result, &nrow,
> &ncol, &errmsg);
> + if(rc) {
> + LOG_ER("SQL statement ('%s') failed. Reason: %s",
> sql.c_str(), errmsg);
> + goto bailout;
> + }
> +
> + // Map object DNs for fast search
> + for(n=1; n<=nrow; n++) {
> + rdnMap[result[n * ncol]] = n;
> + }
> +
> + // Iterate through IMM search result and check attributes
> + while((err = saImmOmSearchNext_o3(searchHandle, &objectName,
> &attributes)) == SA_AIS_OK) {
> + // Increase object counter
> + objectCounter++;
> +
> + // Skip checking IMM object
> + if(!strcmp(objectName,
> "opensafImm=opensafImm,safApp=safImmService") ||
> + !strcmp(objectName,
> "safRdn=immManagement,safApp=safImmService")) {
> + continue;
> + }
> +
> + // Find object index in the fetched result
> + rmi = rdnMap.find(objectName);
> + if(rmi == rdnMap.end()) {
> + LOG_ER("IMM object '%s' cannot be found in PBE
> database", objectName);
> + goto bailout;
> + }
> +
> + // Check each attribute
> + for(i=0; attrDefinitions[i]; i++) {
> + // Don't check RDN attribute
> + if(i == rdnAttrDefIndex) {
> + continue;
> + }
> +
> + /* Skip persistent attributes.
> + * TODO: Should persistent runtime data be checked ???
> + */
> + if(attrDefinitions[i]->attrFlags &
> SA_IMM_ATTR_RUNTIME) {
> + continue;
> + }
> +
> + // Skip default attributes
> + if(!strcmp(attrDefinitions[i]->attrName,
> "SaImmAttrImplementerName") ||
> + !strcmp(attrDefinitions[i]->attrName, "SaImmAttrClassName") ||
> + !strcmp(attrDefinitions[i]->attrName, "SaImmAttrAdminOwnerName")) {
> + continue;
> + }
> +
> + // Find attribute from the search result
> + for(attrPos=0; attributes[attrPos]; attrPos++) {
> + if(!strcmp(attributes[attrPos]->attrName,
> attrDefinitions[i]->attrName)) {
> + break;
> + }
> + }
> + if(!attributes[attrPos]) {
> + LOG_ER("Cannot find attribute '%s' in object '%s'
> in the search result",
> + attrDefinitions[i]->attrName, objectName);
> + goto bailout;
> + }
> +
> + if(attrDefinitions[i]->attrFlags &
> SA_IMM_ATTR_MULTI_VALUE) {
> + // Check multi value attribute
> + char **res;
> + int nr, nc; // number of rows and columns
> +
> + // Create SQL statements to fetch multi values
> + switch(attrDefinitions[i]->attrValueType) {
> + case SA_IMM_ATTR_SADOUBLET:
> + case SA_IMM_ATTR_SAFLOATT:
> + sql = "SELECT real_val FROM
> objects_real_multi WHERE obj_id = ";
> + sql.append(result[rmi->second * ncol + 1]);
> + sql.append(" AND attr_name = '");
> + sql.append(attrDefinitions[i]->attrName);
> + sql.append("'");
> + break;
> +
> + case SA_IMM_ATTR_SAINT32T:
> + case SA_IMM_ATTR_SAINT64T:
> + case SA_IMM_ATTR_SAUINT32T:
> + case SA_IMM_ATTR_SAUINT64T:
> + case SA_IMM_ATTR_SATIMET:
> + sql = "SELECT int_val FROM
> objects_int_multi WHERE obj_id = ";
> + sql.append(result[rmi->second * ncol + 1]);
> + sql.append(" AND attr_name = '");
> + sql.append(attrDefinitions[i]->attrName);
> + sql.append("'");
> + break;
> +
> + case SA_IMM_ATTR_SASTRINGT:
> + case SA_IMM_ATTR_SANAMET:
> + sql = "SELECT text_val FROM
> objects_text_multi WHERE obj_id = ";
> + sql.append(result[rmi->second * ncol + 1]);
> + sql.append(" AND attr_name = '");
> + sql.append(attrDefinitions[i]->attrName);
> + sql.append("'");
> + break;
> +
> + case SA_IMM_ATTR_SAANYT:
> + sql = "SELECT real_val FROM
> objects_real_multi WHERE obj_id = ";
> + sql.append(result[rmi->second * ncol + 1]);
> + sql.append(" AND attr_name = '");
> + sql.append(attrDefinitions[i]->attrName);
> + sql.append("'");
> + break;
> +
> + default:
> + LOG_ER("UNKOWN attribute type: %d",
> attrDefinitions[i]->attrValueType);
> + abort();
> + }
> +
> + // Execute SQL
> + if(sqlite3_get_table(dbHandle, sql.c_str(), &res,
> &nr, &nc, &errmsg) != SQLITE_OK) {
> + LOG_ER("Cannot fetch values for attribute
> '%s' in class '%s'. Reason: %s",
> + attrDefinitions[i]->attrName, className, errmsg);
> + goto bailout;
> + }
> +
> + // First multi value check. Check number of values
> + if(nr !=
> (int)attributes[attrPos]->attrValuesNumber) {
> + LOG_ER("Mismatch in number of multi value
> attributes in attribute '%s' in object '%s'",
> + attributes[attrPos]->attrName, objectName);
> + ok = false;
> + goto finish;
> + }
> +
> + for(r=1; r<=nr; r++) {
> + // First loop. res vs. attributes
> + if(!verifyObjectAttributePBE(res[r],
> attributes[attrPos])) {
> + LOG_ER("Cannot match value for multi
> value attribute '%s' in object '%s'",
> + attributes[attrPos]->attrName, objectName);
> + // Values do not match
> + ok = false;
> + goto finish;
> + }
> + // Second loop. attributes vs. res
> + for(n=0; n<nr; n++) {
> + if(verifyAttributeValuePBE(res[r],
> attributes[attrPos]->attrValues[n],
> attributes[attrPos]->attrValueType)) {
> + // Found value
> + break;
> + }
> + }
> + if(n == nr) {
> + LOG_ER("Cannot match value for multi
> value attribute '%s' in object '%s'.",
> + attributes[attrPos]->attrName, objectName);
> + // Values didn't match in the second loop
> + ok = false;
> + goto finish;
> + }
> + }
> +
> +finish:
> + sqlite3_free_table(res);
> + if(!ok) {
> + goto bailout;
> + }
> + } else { // Check non-multi value attribute
> + // Find column in 'result'
> + for(c=2; c<ncol; c++) {
> + if(!strcmp(attrDefinitions[i]->attrName, result[c])) {
> + break;
> + }
> + }
> +
> + // Verify fetched value against value from the
> search result
> + if(!verifyObjectAttributePBE(result[rmi->second * ncol + c],
> attributes[attrPos])) {
> + LOG_ER("Cannot match attribute value for
> attribute '%s' in object '%s'",
> + attributes[attrPos]->attrName, objectName);
> + goto bailout;
> + }
> + }
> + }
> + }
> +
> + saImmOmSearchFinalize(searchHandle);
> + sqlite3_free_table(result);
> + saImmOmClassDescriptionMemoryFree_2(immHandle, attrDefinitions);
> +
> + if(objectCounter != nrow) {
> + LOG_ER("Mismatch in number of objects between IMM
> database and PBE database");
> + TRACE_LEAVE();
> + return false;
> + }
> +
> + TRACE_LEAVE();
> +
> + return true;
> +
> +bailout:
> + if(result) {
> + sqlite3_free_table(result);
> + }
> +
> + if(searchHandle) {
> + saImmOmSearchFinalize(searchHandle);
> + }
> +
> + if(attrDefinitions) {
> + saImmOmClassDescriptionMemoryFree_2(immHandle,
> attrDefinitions);
> + }
> +
> + TRACE_LEAVE();
> +
> + return false;
> +}
> +
> static ClassInfo* verifyClassPBE(std::string classNameString,
> SaImmHandleT immHandle,
> void* db_handle)
> @@ -1199,11 +1552,9 @@ static ClassInfo* verifyClassPBE(std::st
> SaImmAttrDefinitionT_2 **attrDefinitions;
> SaAisErrorT errorCode;
> /*
> - This function does very little verification right now. It
> should do more.
> - Reason is we are re-attaching to a DB file after a gap
> during which we do not know
> - what has happened to that file. We should at least verify
> that the number of classes
> - and number of objects match. Each object should really have
> a checksum.
> - Verification of objects should be a separate function
> verifyObjectPBE.
> + We are re-attaching to a DB file after a gap during which
> we do not know
> + what has happened to that file. We should at least verify
> that the number
> + of classes and number of objects match.
> */
> TRACE_ENTER();
> @@ -2509,7 +2860,7 @@ bool dumpClassesToPbe(SaImmHandleT immHa
> return false;
> }
> -int verifyPbeState(SaImmHandleT immHandle, ClassMap
> *classIdMap, void* db_handle)
> +int verifyPbeState(SaImmHandleT immHandle, ClassMap *classIdMap,
> void* db_handle, bool pbeVerifyObjects)
> {
> /* Function used only when re-connecting to an already
> existing DB file. */
> std::list<std::string> classNameList;
> @@ -2524,6 +2875,8 @@ int verifyPbeState(SaImmHandleT immHandl
> int nrows=0;
> int ncols=0;
> bool badfile=false;
> + bool verified;
> +
> TRACE_ENTER();
> classNameList = getClassNames(immHandle);
> @@ -2542,11 +2895,20 @@ int verifyPbeState(SaImmHandleT immHandl
> ClassInfo* cl_info = verifyClassPBE((*it), immHandle,
> dbHandle);
> if(cl_info) {
> (*classIdMap)[(*it)] = cl_info;
> - it++;
> } else {
> badfile=true;
> goto bailout;
> }
> +
> + if(pbeVerifyObjects) {
> + verified = verifyClassObjectsPBE((*it), immHandle,
> dbHandle);
> + if(!verified) {
> + badfile=true;
> + goto bailout;
> + }
> + }
> +
> + it++;
> }
> rc = sqlite3_get_table(dbHandle, sqlQ.c_str(), &result,
> &nrows, &ncols, &qErr);
> @@ -3816,7 +4178,7 @@ SaAisErrorT getCcbOutcomeFromPbe(void* d
> return SA_AIS_ERR_LIBRARY;
> }
> -int verifyPbeState(SaImmHandleT immHandle, ClassMap
> *classIdMap, void* db_handle)
> +int verifyPbeState(SaImmHandleT immHandle, ClassMap *classIdMap,
> void* db_handle, bool pbeVerifyObjects)
> {
> abort();
> return 0;
> diff --git a/osaf/libs/common/immsv/immsv_evt.c
> b/osaf/libs/common/immsv/immsv_evt.c
> --- a/osaf/libs/common/immsv/immsv_evt.c
> +++ b/osaf/libs/common/immsv/immsv_evt.c
> @@ -2908,6 +2908,27 @@ static uint32_t immsv_evt_enc_toplevel(I
> ncs_enc_claim_space(o_ub, 4);
> /* immdevt->info.ctrl_msg.pbeFile.buf encoded
> by sublevel */
> }
> +
> + if((immdevt->info.ctrl_msg.refresh == 2) &&
> + (immdevt->type == IMMD_EVT_ND2D_INTRO)) { /*
> Intro after IMMD restart. */
> +
> + IMMSV_RSRV_SPACE_ASSERT(p8, o_ub, 8);
> + ncs_encode_64bit(&p8,
> immdevt->info.ctrl_msg.fevs_count);
> + ncs_enc_claim_space(o_ub, 8);
> +
> + IMMSV_RSRV_SPACE_ASSERT(p8, o_ub, 4);
> + ncs_encode_32bit(&p8,
> immdevt->info.ctrl_msg.admo_id_count);
> + ncs_enc_claim_space(o_ub, 4);
> +
> + IMMSV_RSRV_SPACE_ASSERT(p8, o_ub, 4);
> + ncs_encode_32bit(&p8,
> immdevt->info.ctrl_msg.ccb_id_count);
> + ncs_enc_claim_space(o_ub, 4);
> +
> + IMMSV_RSRV_SPACE_ASSERT(p8, o_ub, 4);
> + ncs_encode_32bit(&p8,
> immdevt->info.ctrl_msg.impl_count);
> + ncs_enc_claim_space(o_ub, 4);
> + }
> +
> break;
> case IMMD_EVT_ND2D_SYNC_FEVS_BASE:
> @@ -4283,6 +4304,28 @@ static uint32_t immsv_evt_dec_toplevel(N
> ncs_dec_skip_space(i_ub, 4);
> /* immdevt->info.ctrl_msg.pbeFile.buf decoded
> by sublevel */
> }
> +
> +
> + if((immdevt->info.ctrl_msg.refresh == 2) &&
> + (immdevt->type == IMMD_EVT_ND2D_INTRO)) { /*
> Intro after IMMD restart. */
> +
> + IMMSV_FLTN_SPACE_ASSERT(p8, local_data, i_ub, 8);
> + immdevt->info.ctrl_msg.fevs_count =
> ncs_decode_64bit(&p8);
> + ncs_dec_skip_space(i_ub, 8);
> +
> + IMMSV_FLTN_SPACE_ASSERT(p8, local_data, i_ub, 4);
> + immdevt->info.ctrl_msg.admo_id_count =
> ncs_decode_32bit(&p8);
> + ncs_dec_skip_space(i_ub, 4);
> +
> + IMMSV_FLTN_SPACE_ASSERT(p8, local_data, i_ub, 4);
> + immdevt->info.ctrl_msg.ccb_id_count =
> ncs_decode_32bit(&p8);
> + ncs_dec_skip_space(i_ub, 4);
> +
> + IMMSV_FLTN_SPACE_ASSERT(p8, local_data, i_ub, 4);
> + immdevt->info.ctrl_msg.impl_count =
> ncs_decode_32bit(&p8);
> + ncs_dec_skip_space(i_ub, 4);
> + }
> +
> break;
> case IMMD_EVT_ND2D_SYNC_FEVS_BASE:
> diff --git a/osaf/libs/common/immsv/include/immpbe_dump.hh
> b/osaf/libs/common/immsv/include/immpbe_dump.hh
> --- a/osaf/libs/common/immsv/include/immpbe_dump.hh
> +++ b/osaf/libs/common/immsv/include/immpbe_dump.hh
> @@ -81,7 +81,7 @@ void deleteClassToPBE(std::string classN
> ClassInfo* theClass);
> int verifyPbeState(SaImmHandleT immHandle, ClassMap
> *classIdMap,
> - void* db_handle);
> + void* db_handle, bool pbeVerifyObjects);
> int dumpObjectsToPbe(SaImmHandleT immHandle, ClassMap*
> classIdMap,
> void* db_handle);
> diff --git a/osaf/libs/common/immsv/include/immsv_api.h
> b/osaf/libs/common/immsv/include/immsv_api.h
> --- a/osaf/libs/common/immsv/include/immsv_api.h
> +++ b/osaf/libs/common/immsv/include/immsv_api.h
> @@ -61,6 +61,7 @@ extern "C" {
> #define OPENSAF_IMM_LONG_DNS_ALLOWED "longDnsAllowed"
> #define OPENSAF_IMM_ACCESS_CONTROL_MODE "accessControlMode"
> #define OPENSAF_IMM_AUTHORIZED_GROUP "authorizedGroup"
> +#define OPENSAF_IMM_SC_ABSENCE_ALLOWED "scAbsenceAllowed"
> typedef enum {
> ACCESS_CONTROL_DISABLED = 0,
> diff --git a/osaf/libs/common/immsv/include/immsv_evt.h
> b/osaf/libs/common/immsv/include/immsv_evt.h
> --- a/osaf/libs/common/immsv/include/immsv_evt.h
> +++ b/osaf/libs/common/immsv/include/immsv_evt.h
> @@ -412,7 +412,9 @@ typedef struct immsv_d2nd_control {
> IMMSV_OCTET_STRING dir;
> IMMSV_OCTET_STRING xmlFile;
> - IMMSV_OCTET_STRING pbeFile;
> + IMMSV_OCTET_STRING pbeFile;
> +
> +
> } IMMSV_D2ND_CONTROL;
>
> /****************************************************************************
> @@ -422,7 +424,7 @@ typedef struct immsv_d2nd_control {
> typedef struct immsv_nd2d_control {
> SaUint32T ndExecPid;
> SaUint32T epoch;
> - uint8_t refresh; //TRUE=> This is a refresh of epoch.
> + uint8_t refresh; /* refresh>0 => Refresh of epoch; refresh==2
> => IMMD restart intro.*/
> uint8_t pbeEnabled;/* OpenSaf4.4:
> 2:not-enabled-not-configured can be
> convertred to 0 in immd.
> 3:not-enabled-configured
> @@ -435,6 +437,17 @@ typedef struct immsv_nd2d_control {
> IMMSV_OCTET_STRING dir;
> IMMSV_OCTET_STRING xmlFile;
> IMMSV_OCTET_STRING pbeFile;
> +
> + /* OpenSaf4.6: At restart of IMMD sv-id, IMMND gives
> feeedback on:
> [Hung] Not OpenSaf4.6 :)
>
> + - fevs_count
> + - admo_id_count
> + - ccb_id_count
> + - impl_count
> + */
> + SaUint64T fevs_count; //Max received at IMMND for Fevs Count
> + SaUint32T admo_id_count;//Max received at IMMND for
> AdminOwner ID
> + SaUint32T ccb_id_count; //Max received at IMMND for CCB ID
> + SaUint32T impl_count; //Max received at IMMND for
> Implementer ID
> } IMMSV_ND2D_CONTROL;
> typedef struct immsv_nd2d_2_pbe {
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Opensaf-devel mailing list
> Opensaf-devel@lists.sourceforge.net
> <mailto:Opensaf-devel@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/opensaf-devel
>
>
>
> ------------------------------------------------------------------------------
>
> Site24x7 APM Insight: Get Deep Visibility into Application
> Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
> _______________________________________________
> Opensaf-devel mailing list
> Opensaf-devel@lists.sourceforge.net
> <mailto:Opensaf-devel@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/opensaf-devel
>
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel
