- **status**: unassigned --> duplicate
- **assigned_to**: Praveen
- **Part**: - --> d
- **Milestone**: future --> never
- **Comment**:
Analysis and Reproduction of the issue:
1)Analysis from the provided amfd internal state and syslog:
Assignment status from the attached AMFD internal state.
SC-1:
dn: safSi=NoRed1,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=PL-3,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
dn: safSi=NoRed2,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=PL-4,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
dn: safSi=NoRed3,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=SC-2,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
dn: safSi=NoRed4,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=SC-1,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
>From SC-2:
dn: safSi=NoRed1,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=PL-3,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
dn: safSi=NoRed2,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=PL-4,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
dn: safSi=NoRed3,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=SC-2,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
dn: safSi=NoRed4,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=SC-1,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
>From syslog nodes assignment status are like this :
Apr 2 15:42:35 SLES-64BIT-SLOT1 osafamfnd[3884]: NO Assigned
'safSi=NoRed4,safApp=OpenSAF' ACTIVE to 'safSu=SC-
1,safSg=NoRed,safApp=OpenSAF'
Apr 2 15:46:19 SLES-32BIT-SLOT2 osafamfnd[2653]: NO Assigned
'safSi=NoRed3,safApp=OpenSAF' ACTIVE to 'safSu=SC-
2,safSg=NoRed,safApp=OpenSAF'
Note that AMF proccessId is same as in the names of amf internal state files.
Apr 2 15:46:03 SLES-32BIT-SLOT4 osafamfnd[7316]: NO Assigned
'safSi=NoRed2,safApp=OpenSAF' ACTIVE to 'safSu=PL-
4,safSg=NoRed,safApp=OpenSAF'
Apr 2 15:41:43 SLES-64BIT-SLOT3 osafamfnd[2480]: NO Assigned
'safSi=NoRed1,safApp=OpenSAF' ACTIVE to 'safSu=PL-
3,safSg=NoRed,safApp=OpenSAF'
There are consistent with the AMF internal state.
Why extra SUSI exists for PL-4 ?
Because of multiple role changes of controller and reboot of payload in
between, AMF could not
delete the SUSI in IMM after deleting it in AMF database.As shown above, in
AMF database such a SUSI relationship
does not exists.SUSI relationship could not be deleted in IMM because AMF
pushes all IMM jobs for clearing
it later when it completes other important activites in this case node
failover. If role of controller changes
before this queue is emptied then some updates may get missed. Such a problem
is already reported in #1141.
2)Reproduction:
After reproducing this issue, please find attached traces and amfd internal
state in 1301_reproduce.tgz.
Below are the steps to reproduce and analysis using AMFD internal states and
traces.
steps to reproduce:
1)Bring up SC-1.
2)Bring up PL-3.
3)Bring up SC-2.
4)Bring down SC-1.
5)Bring up PL-4
6)Bring up SC-1
After this assignments would look like for middleware SGs:
safSISU=safSu=SC-2\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed4,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=PL-3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-2\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=STANDBY(2)
7)On PL-4, run pkill -9 saf.
8)On SC-2, stop opensaf and then restart it.
steps 7) and 8) should be peroformed very fast.
9)Start opensaf on PL-4.
After this assignments would look like for middleware SGs:
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=PL-3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-2\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-2\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=STANDBY(2)
safSISU=safSu=SC-1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed4,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
So there are two SUSIs shown for PL-4 one for safSi=NoRed4,safApp=OpenSAF
and another for safSi=NoRed4,safApp=OpenSAF.
SUSI
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF
does not
exist in AMF database.It is verfied by dumping the amf internal states. The
reason it is
shown still listed is because SC-2 could not delete it in IMM after deleting it
in AMF. SC-2 pushed
it in IMM job queue maintained in AMF. After completion of PL-4 down handling.
SC-2 started clearing
IMM jobs, but SC-2 was stopped before completing it. When SC-1 took the active
role, it wil not update
IMM for old jobs because only active controller maintains the iMM job queue.
Also this susi relationship
does not exist in SC-1 AMF database because in the checkpointing it deleted it.
Sc-1 delete this SUSI relationship:
Apr 7 12:42:37.263251 osafamfd [8532:ckpt_dec.cc:0629] >> dec_siass: i_action
'2'
Apr 7 12:42:37.263820 osafamfd [8532:ckpt_updt.cc:0392] >> avd_ckpt_siass:
'safSi=NoRed2,safApp=OpenSAF' 'safSu=PL-
4,safSg=NoRed,safApp=OpenSAF'
Apr 7 12:42:37.263854 osafamfd [8532:csi.cc:1209] >> avd_compcsi_delete
Apr 7 12:42:37.263863 osafamfd [8532:csi.cc:1244] << avd_compcsi_delete
Apr 7 12:42:37.263868 osafamfd [8532:siass.cc:0351] >> avd_susi_delete:
safSu=PL-4,safSg=NoRed,safApp=OpenSAF
safSi=NoRed2,safApp=OpenSAF
Apr 7 12:42:37.263876 osafamfd [8532:siass.cc:0422] << avd_susi_delete
Apr 7 12:42:37.263901 osafamfd [8532:ckpt_updt.cc:0503] << avd_ckpt_siass:
status '1'
SUSI relationship from SC-1 AMF internal state:
No assignment of safSi=NoRed2,safApp=OpenSAF to
'safSu=PL-4,safSg=NoRed,safApp=OpenSAF'.
dn: safSi=NoRed1,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=PL-3,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
dn: safSi=NoRed2,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=SC-2,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
dn: safSi=NoRed3,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=SC-1,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
dn: safSi=NoRed4,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=PL-4,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
SUSI relationship from SC-2 AMF internal state:
No assignment of safSi=NoRed2,safApp=OpenSAF to
'safSu=PL-4,safSg=NoRed,safApp=OpenSAF'.
dn: safSi=NoRed1,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=PL-3,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
dn: safSi=NoRed2,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=SC-2,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
dn: safSi=NoRed3,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 0
assigned_to_sus:
dn: safSu=SC-1,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
dn: safSi=NoRed4,safApp=OpenSAF
saAmfSIProtectedbySG: safSg=NoRed,safApp=OpenSAF
saAmfSIAdminState: UNLOCKED
saAmfSIAssignmentState: FULLY_ASSIGNED
saAmfSINumCurrActiveAssignments: 1
saAmfSINumCurrStandbyAssignments: 0
si_switch: 0
si_dep_state: 1
num_dependents: 0
alarm_sent: 1
assigned_to_sus:
dn: safSu=PL-4,safSg=NoRed,safApp=OpenSAF
hastate: ACTIVE
fsm: 3
\#1141 also reports the same problem, marking this ticket as duplicate of that.
---
** [tickets:#1301] Middleware no redundancy SI is assigned to the controller
and to the payload.**
**Status:** duplicate
**Milestone:** never
**Created:** Thu Apr 02, 2015 02:49 PM UTC by Srikanth R
**Last Updated:** Fri Apr 03, 2015 06:23 AM UTC
**Owner:** Praveen
*Setup*
Version : 4.6 FC
Setup is enabled with single PBE and no AMF application configured.
*Issues*
1) IMMD on active controller faulted due to healthCheckcallbackTimeout ( Might
be the issue mentioned in #1291
2) Middleware no redundancy SI is assigned to the controller and to the payload.
*Steps performed *
-> Performed cluster stop of all the nodes by running the command
"/etc/init.d/opensafd stop" on all the nodes from payload to controller
-> Started opensaf on all the nodes SC-1, SC-2, PL-3 and PL-4. Opensafd came
up successfully.
Apr 2 15:34:24 SLES-64BIT-SLOT1 opensafd: OpenSAF(4.6.FC - ) services
successfully starte
Apr 2 15:34:25 SLES-32BIT-SLOT4 opensafd: OpenSAF(4.6.FC - ) services
successfully started
Apr 2 15:34:28 SLES-64BIT-SLOT3 opensafd: OpenSAF(4.6.FC - ) services
successfully started
-> Ran the command "/etc/init.d/opensafd restart" on both the payloads.
Because of this, PL-4 could not join the cluster, although PL-3 joined the
cluster.
* Opensafd start on PL-3 succeeded with the following syslog.
Apr 2 15:34:38 SLES-64BIT-SLOT1 kernel: [ 6867.812494] TIPC: Resetting link
<1.1.1:eth0-1.1.3:eth0>, requested by peer while probing
Apr 2 15:34:38 SLES-64BIT-SLOT1 kernel: [ 6867.812667] TIPC: Established link
<1.1.1:eth0-1.1.3:eth0> on network plane A
* Opensafd start on PL-4 failed with following syslog
Apr 2 15:34:43 SLES-32BIT-SLOT4 kernel: [258083.946896] TIPC: Established link
<1.1.4:eth0-1.1.1:eth0> on network plane A
Apr 2 15:34:43 SLES-32BIT-SLOT4 osafimmnd[6997]: NO SERVER STATE:
IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING
Apr 2 15:34:48 SLES-32BIT-SLOT4 osafimmnd[6997]: WA Resending introduce-me -
problems with MDS ? 5
Apr 2 15:36:40 SLES-32BIT-SLOT4 osafimmnd[7032]: ER Failed to load/sync.
Giving up after 51 seconds, restarting..
Apr 2 15:36:40 SLES-32BIT-SLOT4 opensafd[6965]: ER Could Not RESPAWN IMMND
Apr 2 15:36:40 SLES-32BIT-SLOT4 opensafd[6965]: ER Failed DESC:IMMND
Apr 2 15:36:40 SLES-32BIT-SLOT4 opensafd[6965]: ER Trying To RESPAWN
/usr/lib/opensaf/clc-cli/osaf-immnd attempt #2
Apr 2 15:36:40 SLES-32BIT-SLOT4 opensafd[6965]: ER Sending SIGKILL to IMMND,
pid=7027
Apr 2 15:36:40 SLES-32BIT-SLOT4 osafimmnd[7032]: ER IMMND - Periodic server
job failed
Apr 2 15:36:40 SLES-32BIT-SLOT4 osafimmnd[7032]: ER Failed, exiting...
Apr 2 15:36:55 SLES-32BIT-SLOT4 osafimmnd[7068]: Started
Apr 2 15:37:46 SLES-32BIT-SLOT4 osafimmnd[7068]: ER Failed, exiting...
Apr 2 15:37:46 SLES-32BIT-SLOT4 opensafd: Starting OpenSAF fail
Corresponding logs on SC-1 :
Apr 2 15:34:44 SLES-64BIT-SLOT1 kernel: [ 6873.737376] TIPC: Lost contact with
<1.1.4>
Apr 2 15:34:44 SLES-64BIT-SLOT1 kernel: [ 6873.737983] TIPC: Established link
<1.1.1:eth0-1.1.4:eth0> on network plane A
Apr 2 15:36:23 SLES-64BIT-SLOT1 osafimmloadd: ER Too many TRY_AGAIN on
saImmOmSearchNext - aborting
Apr 2 15:36:23 SLES-64BIT-SLOT1 osafimmnd[5772]: ER SYNC APPARENTLY FAILED
status:1
Finally SC-1 went for reboot, as IMMD health checkout timedout.
Apr 2 15:41:22 SLES-64BIT-SLOT1 osafimmnd[5772]: NO Coord broadcasting
PBE_PRTO_PURGE_MUTATIONS, epoch:220
Apr 2 15:41:23 SLES-64BIT-SLOT1 osafimmnd[5772]: NO ImmModel::getPbeOi reports
missing PbeOi locally => unsafe
Apr 2 15:41:23 SLES-64BIT-SLOT1 osafimmnd[5772]: NO Coord broadcasting
PBE_PRTO_PURGE_MUTATIONS, epoch:220
Apr 2 15:41:24 SLES-64BIT-SLOT1 osafamfnd[5853]: NO SU failover probation
timer started (timeout: 1200000000000 ns)
Apr 2 15:41:24 SLES-64BIT-SLOT1 osafamfnd[5853]: NO Performing failover of
'safSu=SC-1,safSg=2N,safApp=OpenSAF' (SU failover count: 1)
Apr 2 15:41:24 SLES-64BIT-SLOT1 osafamfnd[5853]: NO
'safComp=IMMD,safSu=SC-1,safSg=2N,safApp=OpenSAF' recovery action escalated
from 'componentFailover' to 'suFailover'
-> Induced a manual failver on SC-2 by killing osaflogd on SC-2.
-> Now SC-1 is the active controller.
-> When opensafd is started on PL-4, middleware si Nored4 is assigned to two
middleware SUs
Below is the opensafd status
safSISU=safSu=SC-1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed4,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=PL-3\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed4,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed2,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-2\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
safSISU=safSu=SC-2\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=STANDBY(2)
Also the SUSI objects are created for Nored4 SI for both SC-1 SU and PL-4 SU.
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
