Hi, Praveen:
I checked the lbm on PL4 log and found that it was restarted on 6/16, but no SI
assignment was sent from opensaf. So the question is why it is started and left
in unassigned state? I thought it is not started if it is not assigned with any
SI.
Thanks.
Kang-sen
PL4 lbm
root@ruby:/shared/webcache/Bharti/BLR/node-1-4/movik_logs/BHA-IND-WHF-KK-CAE-4/var/movik/log#
grep ====== lbm*
lbm:NTC:1:(0){06/16/16 - 20:07:50:079402}:=========== DefaultThread started for
MovikApp =============
NTC:1:(0){06/16/16 - 20:08:10:802336}:HaAppStateProvisioning.cpp:93:configDone
Deferring configDone action until HA state assignment
lbm.6:NTC:1:(0){06/08/16 - 00:45:39:161432}:=========== DefaultThread started
for MovikApp =============
NTC:1:(0){06/08/16 - 00:45:39:289960}:AmfBase.cpp:836:csiSetCB
csiDescriptor.csiName =
safCsi=LbmCSI_SI-2,safSi=loadBalancerSI-2,safApp=loadBalancerApp
lbm.7:NTC:1:(0){05/04/16 - 03:05:16:100494}:=========== DefaultThread started
for MovikApp =============
NTC:1:(0){05/04/16 - 03:05:16:190950}:AmfBase.cpp:836:csiSetCB
csiDescriptor.csiName =
safCsi=LbmCSI_SI-1,safSi=loadBalancerSI-1,safApp=loadBalancerApp
-----Original Message-----
From: praveen malviya [mailto:[email protected]]
Sent: Monday, July 04, 2016 8:27 AM
To: [email protected]
Subject: Re: [users] question about 'N WAY ACTIVE' redundancy model
Including user list.
On 04-Jul-16 12:04 PM, praveen malviya wrote:
>
>
> On 01-Jul-16 5:54 PM, Kang-Sen Lu wrote:
>> Hi, Praveen:
>>
>> As I said before, the problem is that we did not save old syslog
>> (older than 1 week), and we found the problem over 1 week later. If I
>> had old syslog, it is easy to tell when did opensaf start. I wonder
>> from safLog/saLog*, are there any log to indicate the opensaf restart?
>>
>> I can still get safLog from all slots. So I should be able to get
>> when
>> PL4/PL6/PL8 start/stop. I will gather those info and send them to you
>> later.
>>
>> I am curious how does 'immfind' print the object list? Also how does
>> 'immlist' get the attributes of any object? Are they both coming from
>> the same database?
>>
> Immfind will list all the objects that are present in its database.
> For immlist <object name>, values of cached attributes will be listed
> from IMM database. But for non-cached attributes, IMM will give a
> callback to the object implementer and in this way IMM will get the
> latest value and it will list this latest value.
> So in the present case "immlist
> safSu=loadBalancerSU_PL-4,safSg=loadBalancerSG,safApp=loadBalancerApp "
> will give a callback to AMF for "saAmfSUNumCurrStandbySIs,
> saAmfSUNumCurrActiveSIs etc". Since there value is listed as 0, it
> means
> PL4 is not assigned SI1 when this list operation was carried out.
> Now the question, Why immfind lists assignment of PL4-SI2?
> Creation/modificatiion/deleteion of runtime object from IMM is the
> responsibility of OI, in this case AMF. As I have mentioned earlier,
> that there has been an issue in AMF when it failed to delete a runtime
> object from IMM.
> How to check it whether AMF is really assigning SI2 or it could not
> delete it from IMM when PL4 reboots?
> Based on some clues it can be known:
> -As you pointed out that it was found 1 week later. So in this case
> did you see application behavior change. In case of real assignment,
> same service will be served from two nodes and this will be easily
> visible as service impact.
> -Before this extra assignment was not there and when it appeared, Is
> there any controller role change.
>
>
> Thanks
> Praveen
>> If so, then the immfind showed both PL4 and PL6 assigned with SI2.
>> But immlist showed PL4 was not assigned with any SI at all. That is
>> just internally contradictory.
>>
>> It seems from safLog, this kind of log text would tell us when a PL
>> to SI assignment occurred, correct?
>>
>> saLogSystem_20160512_233228.log: 1660 23:34:00 05/12/2016 NO
>> safApp=safAmfService "HA State ACTIVE of
>> safSu=loadBalancerSU_PL-8,safSg=loadBalancerSG,safApp=loadBalancerApp
>> for safSi=loadBalancerSI-1,safApp=loadBalancerApp
>> saLogSystem_20160512_233228.log: 1668 23:34:01 05/12/2016 NO
>> safApp=safAmfService "HA State ACTIVE of
>> safSu=loadBalancerSU_PL-6,safSg=loadBalancerSG,safApp=loadBalancerApp
>> for safSi=loadBalancerSI-2,safApp=loadBalancerApp
>> saLogSystem_20160512_233228.log: 1921 23:31:16 06/01/2016 NO
>> safApp=safAmfService "HA State ACTIVE of
>> safSu=loadBalancerSU_PL-4,safSg=loadBalancerSG,safApp=loadBalancerApp
>> for safSi=loadBalancerSI-2,safApp=loadBalancerApp
>> saLogSystem_20160512_233228.log: 2021 00:40:29 06/08/2016 NO
>> safApp=safAmfService "HA State ACTIVE of
>> safSu=loadBalancerSU_PL-6,safSg=loadBalancerSG,safApp=loadBalancerApp
>> for safSi=loadBalancerSI-2,safApp=loadBalancerApp
>> saLogSystem_20160512_233228.log: 4430 20:02:59 06/16/2016 NO
>> safApp=safAmfService "HA State ACTIVE of
>> safSu=loadBalancerSU_PL-6,safSg=loadBalancerSG,safApp=loadBalancerApp
>> for safSi=loadBalancerSI-2,safApp=loadBalancerApp
>>
>>
>> Thanks.
>>
>> Kang-sen
>>
>>
>> -----Original Message-----
>> From: praveen malviya [mailto:[email protected]]
>> Sent: Friday, July 01, 2016 3:06 AM
>> To: Kang-Sen Lu <[email protected]>
>> Cc: [email protected]
>> Subject: Re: [users] question about 'N WAY ACTIVE' redundancy model
>>
>>
>>
>> On 30-Jun-16 6:08 PM, Kang-Sen Lu wrote:
>>> Hi, Praveen:
>>>
>>> Thanks for your response.
>>>
>>> You asked "... for the configured value of
>>> saAmfSIPrefActiveAssignments for each SI". I am sure it is
>>> explicitly set to 1 by our application code. It is also shown in the
>>> immlist output for Sis in the original post.
>>>
>>> I have done some experiment by first configure all 3 Sus. Then
>>> configure SI-1. And I saw only one of the SU was assigned with the
>>> SI-1. Then I configured SI-2, and only 1 SU was assigned to SI-2.
>>>
>>> So I think in normal conditions, the amf is doing the correct thing.
>>>
>>> I suspect when nodes are crashing and restarting, may be the amf
>>> database was not sync'ed up correctly. Because the immfind and
>>> immlist was showing inconsistency, as far as PL-4 is assigned or not.
>>>
>>> Unfortunately, the safLog did not show anything useful. I also lost
>>> the syslog because we did not detect this problem quick enough.
>>>
>>> I was hoping you have some insight about how could immfind and
>>> immlist display inconsistent fact. I think immlist is displaying the
>>> current amf database. But immfind is also displaying amf database.
>>>
>> So before crash, situation is like this:
>> Model: N-Way Active,2 SIs: SI1 and SI2, 3 SUs: SU1 on Node1, SU2 on
>> Node2 and SU3 on Node3.
>> SU1 assigned SI1 on Node1, SU2 assigned SI2 on Node2 and SU3 spare is
>> not assigned on Node3.
>> Now suppose Node1 crashes/restarts then AMF will assigned SI1 on
>> spare
>> Su3 and assignments, thus, will look like this in ideal case:
>> SU1 spare (after Node1 comes up), SU2 is assigned SI2 on Node2(no
>> change) and SU3 is assigned SI1( fresh assignment after Node1 lefts).
>>
>> But I think what is being observed is SU1 assigned to SI1 (old
>> assignment) is still listed by immlist/immfind. If this is the case
>> Could you please check that after restart/reboot of Node1, SU1 was
>> instantiated and was given callbacks for SI1?
>> If after fresh assignment of SI1 in SU3, AMF is still assigning it to
>> SU1 on Node1 then it is a bug and a ticket can be written for it.
>> But we have had issues in the past when assignments were not present
>> in AMF database but still shown by IMM. This could be because
>> controller failover/switchover happens after Node1 restart and before
>> AMF deletes assignments from IMM.
>> Is the node that is restarting/crashing is a controller node? or If
>> it is a payload node, is the crash of this node is followed by
>> controller role change?
>>
>> Thanks,
>> Praveen
>>
>>> Thanks again.
>>>
>>> Kang-sen
>>>
>>> -----Original Message-----
>>> From: praveen malviya [mailto:[email protected]]
>>> Sent: Thursday, June 30, 2016 2:02 AM
>>> To: Kang-Sen Lu <[email protected]>;
>>> [email protected]
>>> Subject: Re: [users] question about 'N WAY ACTIVE' redundancy model
>>>
>>> Please find response below with [Praveen].
>>>
>>> Thanks,
>>> Praveen
>>>
>>> On 29-Jun-16 8:04 PM, Kang-Sen Lu wrote:
>>>> We are running opensaf 4.4.0.
>>>>
>>>> We have a service group containing 2 SIs (SI1, SI2), and 3 SUs
>>>> (PL4, PL6, PL8).
>>>>
>>>> We are selecting N WAY ACTIVE redundancy model, and expected 1 SI
>>>> assigned to 1 SU, with the 3rd SU as spare.
>>>>
>>>> >From immfind output, we saw PL4 and PL6 are assigned with SI2, PL8
>>>> was assigned with SI2.
>>>>
>>>> >From immlist output, we saw PL4 was not assigned with any SI.
>>> [Praveen] PL4 is assigned saAmfSGMaxActiveSIsperSU as per immfind
>>> output below.
>>>
>>> Anyways, primary reason could be that the configured value of
>>> saAmfSGMaxActiveSIsperSU=1 in SG. Since all the SUs got one SI
>>> assigned, because of this criteria now there will be no more
>>> assignments. So in this case SI1 has only one assignments in SG.
>>>
>>> But still I have a doubt, why SI2 got two assignments when in the
>>> below mentioned "immlist safSi=loadBalancerSI-2,safApp=loadBalancerApp"
>>> output, value of saAmfSIPrefActiveAssignments is 1.
>>> Could you please check the configuration for the configured value of
>>> saAmfSIPrefActiveAssignments for each SI.
>>>
>>>
>>>
>>>>
>>>> That is not consistent.
>>>>
>>>> Any idea why this happened?
>>>>
>>>> Here is the immfind output:
>>>>
>>>> root@BHA-IND-WHF-KK-CAE-5:~# immfind | grep -i
>>>> "safCSIComp=safComp=Lbm_PL-"
>>>> safCSIComp=safComp=Lbm_PL-4\,safSu=loadBalancerSU_PL-4\,safSg=loadB
>>>> ala
>>>> ncerSG\,safApp=loadBalancerApp,safCsi=LbmCSI_SI-2,safSi=loadBalance
>>>> rSI
>>>> -2,safApp=loadBalancerApp
>>>> safCSIComp=safComp=Lbm_PL-6\,safSu=loadBalancerSU_PL-6\,safSg=loadB
>>>> ala
>>>> ncerSG\,safApp=loadBalancerApp,safCsi=LbmCSI_SI-2,safSi=loadBalance
>>>> rSI
>>>> -2,safApp=loadBalancerApp
>>>> safCSIComp=safComp=Lbm_PL-8\,safSu=loadBalancerSU_PL-8\,safSg=loadB
>>>> ala
>>>> ncerSG\,safApp=loadBalancerApp,safCsi=LbmCSI_SI-1,safSi=loadBalance
>>>> rSI
>>>> -1,safApp=loadBalancerApp
>>>> root@BHA-IND-WHF-KK-CAE-5:~#
>>>>
>>>> Here is the immlist output:
>>>>
>>>> root@BHA-IND-WHF-KK-CAE-5:~# immlist
>>>> safSu=loadBalancerSU_PL-4,safSg=loadBalancerSG,safApp=loadBalancerApp
>>>> Name Type
>>>> Value(s)
>>>> ===================================================================
>>>> =====
>>>>
>>>> safSu SA_STRING_T
>>>> safSu=loadBalancerSU_PL-4
>>>> saAmfSUType SA_NAME_T
>>>> safVersion=4.0.0,safSuType=MovikSUType (38)
>>>> saAmfSURestartCount SA_UINT32_T 0 (0x0)
>>>> saAmfSUReadinessState SA_UINT32_T 2 (0x2)
>>>> saAmfSURank SA_UINT32_T 1 (0x1)
>>>> saAmfSUPresenceState SA_UINT32_T 3 (0x3)
>>>> saAmfSUPreInstantiable SA_UINT32_T 1 (0x1)
>>>> saAmfSUOperState SA_UINT32_T 1 (0x1)
>>>> saAmfSUNumCurrStandbySIs SA_UINT32_T 0 (0x0)
>>>> saAmfSUNumCurrActiveSIs SA_UINT32_T 0 (0x0)
>>>> saAmfSUMaintenanceCampaign SA_NAME_T <Empty>
>>>> saAmfSUHostedByNode SA_NAME_T
>>>> safAmfNode=PL-4,safAmfCluster=myAmfCluster (42)
>>>> saAmfSUHostNodeOrNodeGroup SA_NAME_T
>>>> safAmfNode=PL-4,safAmfCluster=myAmfCluster (42)
>>>> saAmfSUFailover SA_UINT32_T <Empty>
>>>> saAmfSUAssignedSIs SA_NAME_T <Empty>
>>>> saAmfSUAdminState SA_UINT32_T 1 (0x1)
>>>> SaImmAttrImplementerName SA_STRING_T
>>>> safAmfService
>>>> SaImmAttrClassName SA_STRING_T SaAmfSU
>>>> SaImmAttrAdminOwnerName SA_STRING_T <Empty>
>>>> MovikServiceMode SA_STRING_T <Empty>
>>>>
>>>> root@BHA-IND-WHF-KK-CAE-5:~# immlist
>>>> safSu=loadBalancerSU_PL-6,safSg=loadBalancerSG,safApp=loadBalancerApp
>>>> Name Type
>>>> Value(s)
>>>> ===================================================================
>>>> =====
>>>>
>>>> safSu SA_STRING_T
>>>> safSu=loadBalancerSU_PL-6
>>>> saAmfSUType SA_NAME_T
>>>> safVersion=4.0.0,safSuType=MovikSUType (38)
>>>> saAmfSURestartCount SA_UINT32_T 0 (0x0)
>>>> saAmfSUReadinessState SA_UINT32_T 2 (0x2)
>>>> saAmfSURank SA_UINT32_T 1 (0x1)
>>>> saAmfSUPresenceState SA_UINT32_T 3 (0x3)
>>>> saAmfSUPreInstantiable SA_UINT32_T 1 (0x1)
>>>> saAmfSUOperState SA_UINT32_T 1 (0x1)
>>>> saAmfSUNumCurrStandbySIs SA_UINT32_T 0 (0x0)
>>>> saAmfSUNumCurrActiveSIs SA_UINT32_T 1 (0x1)
>>>> saAmfSUMaintenanceCampaign SA_NAME_T <Empty>
>>>> saAmfSUHostedByNode SA_NAME_T
>>>> safAmfNode=PL-6,safAmfCluster=myAmfCluster (42)
>>>> saAmfSUHostNodeOrNodeGroup SA_NAME_T
>>>> safAmfNode=PL-6,safAmfCluster=myAmfCluster (42)
>>>> saAmfSUFailover SA_UINT32_T <Empty>
>>>> saAmfSUAssignedSIs SA_NAME_T <Empty>
>>>> saAmfSUAdminState SA_UINT32_T 1 (0x1)
>>>> SaImmAttrImplementerName SA_STRING_T
>>>> safAmfService
>>>> SaImmAttrClassName SA_STRING_T SaAmfSU
>>>> SaImmAttrAdminOwnerName SA_STRING_T <Empty>
>>>> MovikServiceMode SA_STRING_T <Empty>
>>>>
>>>> root@BHA-IND-WHF-KK-CAE-5:~# immlist
>>>> safSu=loadBalancerSU_PL-8,safSg=loadBalancerSG,safApp=loadBalancerApp
>>>> Name Type
>>>> Value(s)
>>>> ===================================================================
>>>> =====
>>>>
>>>> safSu SA_STRING_T
>>>> safSu=loadBalancerSU_PL-8
>>>> saAmfSUType SA_NAME_T
>>>> safVersion=4.0.0,safSuType=MovikSUType (38)
>>>> saAmfSURestartCount SA_UINT32_T 0 (0x0)
>>>> saAmfSUReadinessState SA_UINT32_T 2 (0x2)
>>>> saAmfSURank SA_UINT32_T 1 (0x1)
>>>> saAmfSUPresenceState SA_UINT32_T 3 (0x3)
>>>> saAmfSUPreInstantiable SA_UINT32_T 1 (0x1)
>>>> saAmfSUOperState SA_UINT32_T 1 (0x1)
>>>> saAmfSUNumCurrStandbySIs SA_UINT32_T 0 (0x0)
>>>> saAmfSUNumCurrActiveSIs SA_UINT32_T 1 (0x1)
>>>> saAmfSUMaintenanceCampaign SA_NAME_T <Empty>
>>>> saAmfSUHostedByNode SA_NAME_T
>>>> safAmfNode=PL-8,safAmfCluster=myAmfCluster (42)
>>>> saAmfSUHostNodeOrNodeGroup SA_NAME_T
>>>> safAmfNode=PL-8,safAmfCluster=myAmfCluster (42)
>>>> saAmfSUFailover SA_UINT32_T <Empty>
>>>> saAmfSUAssignedSIs SA_NAME_T <Empty>
>>>> saAmfSUAdminState SA_UINT32_T 1 (0x1)
>>>> SaImmAttrImplementerName SA_STRING_T
>>>> safAmfService
>>>> SaImmAttrClassName SA_STRING_T SaAmfSU
>>>> SaImmAttrAdminOwnerName SA_STRING_T <Empty>
>>>> MovikServiceMode SA_STRING_T <Empty>
>>>>
>>>> root@BHA-IND-WHF-KK-CAE-5:~#
>>>>
>>>>
>>>>
>>>> root@BHA-IND-WHF-KK-CAE-5:~# immlist
>>>> safSi=loadBalancerSI-1,safApp=loadBalancerApp
>>>> Name Type
>>>> Value(s)
>>>> ===================================================================
>>>> =====
>>>>
>>>> safSi SA_STRING_T
>>>> safSi=loadBalancerSI-1
>>>> saAmfSvcType SA_NAME_T
>>>> safVersion=4.0.0,safSvcType=MovikSvcType (40)
>>>> saAmfSIStandbyWeight SA_STRING_T <Empty>
>>>> saAmfSIRank SA_UINT32_T 0 (0x0)
>>>> saAmfSIProtectedbySG SA_NAME_T
>>>> safSg=loadBalancerSG,safApp=loadBalancerApp (43)
>>>> saAmfSIPrefStandbyAssignments SA_UINT32_T 1 (0x1)
>>>> saAmfSIPrefActiveAssignments SA_UINT32_T 1 (0x1)
>>>> saAmfSINumCurrStandbyAssignments SA_UINT32_T 0 (0x0)
>>>> saAmfSINumCurrActiveAssignments SA_UINT32_T 1 (0x1)
>>>> saAmfSIAssignmentState SA_UINT32_T 2 (0x2)
>>>> saAmfSIAdminState SA_UINT32_T 1 (0x1)
>>>> saAmfSIActiveWeight SA_STRING_T <Empty>
>>>> SaImmAttrImplementerName SA_STRING_T
>>>> safAmfService
>>>> SaImmAttrClassName SA_STRING_T SaAmfSI
>>>> SaImmAttrAdminOwnerName SA_STRING_T <Empty>
>>>>
>>>> root@BHA-IND-WHF-KK-CAE-5:~# immlist
>>>> safSi=loadBalancerSI-2,safApp=loadBalancerApp
>>>> Name Type
>>>> Value(s)
>>>> ===================================================================
>>>> =====
>>>>
>>>> safSi SA_STRING_T
>>>> safSi=loadBalancerSI-2
>>>> saAmfSvcType SA_NAME_T
>>>> safVersion=4.0.0,safSvcType=MovikSvcType (40)
>>>> saAmfSIStandbyWeight SA_STRING_T <Empty>
>>>> saAmfSIRank SA_UINT32_T 0 (0x0)
>>>> saAmfSIProtectedbySG SA_NAME_T
>>>> safSg=loadBalancerSG,safApp=loadBalancerApp (43)
>>>> saAmfSIPrefStandbyAssignments SA_UINT32_T 1 (0x1)
>>>> saAmfSIPrefActiveAssignments SA_UINT32_T 1 (0x1)
>>>> saAmfSINumCurrStandbyAssignments SA_UINT32_T 0 (0x0)
>>>> saAmfSINumCurrActiveAssignments SA_UINT32_T 1 (0x1)
>>>> saAmfSIAssignmentState SA_UINT32_T 2 (0x2)
>>>> saAmfSIAdminState SA_UINT32_T 1 (0x1)
>>>> saAmfSIActiveWeight SA_STRING_T <Empty>
>>>> SaImmAttrImplementerName SA_STRING_T
>>>> safAmfService
>>>> SaImmAttrClassName SA_STRING_T SaAmfSI
>>>> SaImmAttrAdminOwnerName SA_STRING_T <Empty>
>>>>
>>>> root@BHA-IND-WHF-KK-CAE-5:~#
>>>>
>>>> root@BHA-IND-WHF-KK-CAE-5:~# immlist
>>>> safSg=loadBalancerSG,safApp=loadBalancerApp
>>>> Name Type
>>>> Value(s)
>>>> ===================================================================
>>>> =====
>>>>
>>>> safSg SA_STRING_T
>>>> safSg=loadBalancerSG
>>>> saAmfSGType SA_NAME_T
>>>> safVersion=4.0.0,safSgType=MovikSGTypeNWayActive (48)
>>>> saAmfSGSuRestartProb SA_TIME_T <Empty>
>>>> saAmfSGSuRestartMax SA_UINT32_T <Empty>
>>>> saAmfSGSuHostNodeGroup SA_NAME_T <Empty>
>>>> saAmfSGNumPrefStandbySUs SA_UINT32_T 0 (0x0)
>>>> saAmfSGNumPrefInserviceSUs SA_UINT32_T 100
>>>> (0x64)
>>>> saAmfSGNumPrefAssignedSUs SA_UINT32_T 100
>>>> (0x64)
>>>> saAmfSGNumPrefActiveSUs SA_UINT32_T 100
>>>> (0x64)
>>>> saAmfSGNumCurrNonInstantiatedSpareSUs SA_UINT32_T 0 (0x0)
>>>> saAmfSGNumCurrInstantiatedSpareSUs SA_UINT32_T 1 (0x1)
>>>> saAmfSGNumCurrAssignedSUs SA_UINT32_T 2 (0x2)
>>>> saAmfSGMaxStandbySIsperSU SA_UINT32_T <Empty>
>>>> saAmfSGMaxActiveSIsperSU SA_UINT32_T 1 (0x1)
>>>> saAmfSGCompRestartProb SA_TIME_T <Empty>
>>>> saAmfSGCompRestartMax SA_UINT32_T <Empty>
>>>> saAmfSGAutoRepair SA_UINT32_T 1 (0x1)
>>>> saAmfSGAutoAdjustProb SA_TIME_T <Empty>
>>>> saAmfSGAutoAdjust SA_UINT32_T 0 (0x0)
>>>> saAmfSGAdminState SA_UINT32_T 1 (0x1)
>>>> SaImmAttrImplementerName SA_STRING_T
>>>> safAmfService
>>>> SaImmAttrClassName SA_STRING_T SaAmfSG
>>>> SaImmAttrAdminOwnerName SA_STRING_T <Empty>
>>>>
>>>> Thanks.
>>>>
>>>> Kang-sen
>>>>
>>>> -------------------------------------------------------------------
>>>> ---
>>>> -------- Attend Shape: An AT&T Tech Expo July 15-16. Meet us at
>>>> AT&T Park in San Francisco, CA to explore cutting-edge tech and
>>>> listen to tech luminaries present their vision of the future. This
>>>> family event has something for everyone, including kids. Get more
>>>> information and register today.
>>>> http://sdm.link/attshape
>>>> _______________________________________________
>>>> Opensaf-users mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/opensaf-users
>>>>
------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Opensaf-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-users
------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Opensaf-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-users
