Including user list.
On 04-Jul-16 12:04 PM, praveen malviya wrote: > > > On 01-Jul-16 5:54 PM, Kang-Sen Lu wrote: >> Hi, Praveen: >> >> As I said before, the problem is that we did not save old syslog >> (older than 1 week), and we found the problem over 1 week later. If I >> had old syslog, it is easy to tell when did opensaf start. I wonder >> from safLog/saLog*, are there any log to indicate the opensaf restart? >> >> I can still get safLog from all slots. So I should be able to get when >> PL4/PL6/PL8 start/stop. I will gather those info and send them to you >> later. >> >> I am curious how does 'immfind' print the object list? Also how does >> 'immlist' get the attributes of any object? Are they both coming from >> the same database? >> > Immfind will list all the objects that are present in its database. > For immlist <object name>, values of cached attributes will be listed > from IMM database. But for non-cached attributes, IMM will give a > callback to the object implementer and in this way IMM will get the > latest value and it will list this latest value. > So in the present case "immlist > safSu=loadBalancerSU_PL-4,safSg=loadBalancerSG,safApp=loadBalancerApp " > will give a callback to AMF for "saAmfSUNumCurrStandbySIs, > saAmfSUNumCurrActiveSIs etc". Since there value is listed as 0, it means > PL4 is not assigned SI1 when this list operation was carried out. > Now the question, Why immfind lists assignment of PL4-SI2? > Creation/modificatiion/deleteion of runtime object from IMM is the > responsibility of OI, in this case AMF. As I have mentioned earlier, > that there has been an issue in AMF when it failed to delete a runtime > object from IMM. > How to check it whether AMF is really assigning SI2 or it could not > delete it from IMM when PL4 reboots? > Based on some clues it can be known: > -As you pointed out that it was found 1 week later. So in this case did > you see application behavior change. In case of real assignment, same > service will be served from two nodes and this will be easily visible as > service impact. > -Before this extra assignment was not there and when it appeared, Is > there any controller role change. > > > Thanks > Praveen >> If so, then the immfind showed both PL4 and PL6 assigned with SI2. But >> immlist showed PL4 was not assigned with any SI at all. That is just >> internally contradictory. >> >> It seems from safLog, this kind of log text would tell us when a PL to >> SI assignment occurred, correct? >> >> saLogSystem_20160512_233228.log: 1660 23:34:00 05/12/2016 NO >> safApp=safAmfService "HA State ACTIVE of >> safSu=loadBalancerSU_PL-8,safSg=loadBalancerSG,safApp=loadBalancerApp >> for safSi=loadBalancerSI-1,safApp=loadBalancerApp >> saLogSystem_20160512_233228.log: 1668 23:34:01 05/12/2016 NO >> safApp=safAmfService "HA State ACTIVE of >> safSu=loadBalancerSU_PL-6,safSg=loadBalancerSG,safApp=loadBalancerApp >> for safSi=loadBalancerSI-2,safApp=loadBalancerApp >> saLogSystem_20160512_233228.log: 1921 23:31:16 06/01/2016 NO >> safApp=safAmfService "HA State ACTIVE of >> safSu=loadBalancerSU_PL-4,safSg=loadBalancerSG,safApp=loadBalancerApp >> for safSi=loadBalancerSI-2,safApp=loadBalancerApp >> saLogSystem_20160512_233228.log: 2021 00:40:29 06/08/2016 NO >> safApp=safAmfService "HA State ACTIVE of >> safSu=loadBalancerSU_PL-6,safSg=loadBalancerSG,safApp=loadBalancerApp >> for safSi=loadBalancerSI-2,safApp=loadBalancerApp >> saLogSystem_20160512_233228.log: 4430 20:02:59 06/16/2016 NO >> safApp=safAmfService "HA State ACTIVE of >> safSu=loadBalancerSU_PL-6,safSg=loadBalancerSG,safApp=loadBalancerApp >> for safSi=loadBalancerSI-2,safApp=loadBalancerApp >> >> >> Thanks. >> >> Kang-sen >> >> >> -----Original Message----- >> From: praveen malviya [mailto:[email protected]] >> Sent: Friday, July 01, 2016 3:06 AM >> To: Kang-Sen Lu <[email protected]> >> Cc: [email protected] >> Subject: Re: [users] question about 'N WAY ACTIVE' redundancy model >> >> >> >> On 30-Jun-16 6:08 PM, Kang-Sen Lu wrote: >>> Hi, Praveen: >>> >>> Thanks for your response. >>> >>> You asked "... for the configured value of >>> saAmfSIPrefActiveAssignments for each SI". I am sure it is >>> explicitly set to 1 by our application code. It is also shown in the >>> immlist output for Sis in the original post. >>> >>> I have done some experiment by first configure all 3 Sus. Then >>> configure SI-1. And I saw only one of the SU was assigned with the >>> SI-1. Then I configured SI-2, and only 1 SU was assigned to SI-2. >>> >>> So I think in normal conditions, the amf is doing the correct thing. >>> >>> I suspect when nodes are crashing and restarting, may be the amf >>> database was not sync'ed up correctly. Because the immfind and >>> immlist was showing inconsistency, as far as PL-4 is assigned or not. >>> >>> Unfortunately, the safLog did not show anything useful. I also lost >>> the syslog because we did not detect this problem quick enough. >>> >>> I was hoping you have some insight about how could immfind and >>> immlist display inconsistent fact. I think immlist is displaying the >>> current amf database. But immfind is also displaying amf database. >>> >> So before crash, situation is like this: >> Model: N-Way Active,2 SIs: SI1 and SI2, 3 SUs: SU1 on Node1, SU2 on >> Node2 and SU3 on Node3. >> SU1 assigned SI1 on Node1, SU2 assigned SI2 on Node2 and SU3 spare is >> not assigned on Node3. >> Now suppose Node1 crashes/restarts then AMF will assigned SI1 on spare >> Su3 and assignments, thus, will look like this in ideal case: >> SU1 spare (after Node1 comes up), SU2 is assigned SI2 on Node2(no >> change) and SU3 is assigned SI1( fresh assignment after Node1 lefts). >> >> But I think what is being observed is SU1 assigned to SI1 (old >> assignment) is still listed by immlist/immfind. If this is the case >> Could you please check that after restart/reboot of Node1, SU1 was >> instantiated and was given callbacks for SI1? >> If after fresh assignment of SI1 in SU3, AMF is still assigning it to >> SU1 on Node1 then it is a bug and a ticket can be written for it. >> But we have had issues in the past when assignments were not present >> in AMF database but still shown by IMM. This could be because >> controller failover/switchover happens after Node1 restart and before >> AMF deletes assignments from IMM. >> Is the node that is restarting/crashing is a controller node? or If it >> is a payload node, is the crash of this node is followed by controller >> role change? >> >> Thanks, >> Praveen >> >>> Thanks again. >>> >>> Kang-sen >>> >>> -----Original Message----- >>> From: praveen malviya [mailto:[email protected]] >>> Sent: Thursday, June 30, 2016 2:02 AM >>> To: Kang-Sen Lu <[email protected]>; [email protected] >>> Subject: Re: [users] question about 'N WAY ACTIVE' redundancy model >>> >>> Please find response below with [Praveen]. >>> >>> Thanks, >>> Praveen >>> >>> On 29-Jun-16 8:04 PM, Kang-Sen Lu wrote: >>>> We are running opensaf 4.4.0. >>>> >>>> We have a service group containing 2 SIs (SI1, SI2), and 3 SUs (PL4, >>>> PL6, PL8). >>>> >>>> We are selecting N WAY ACTIVE redundancy model, and expected 1 SI >>>> assigned to 1 SU, with the 3rd SU as spare. >>>> >>>> >From immfind output, we saw PL4 and PL6 are assigned with SI2, PL8 >>>> was assigned with SI2. >>>> >>>> >From immlist output, we saw PL4 was not assigned with any SI. >>> [Praveen] PL4 is assigned saAmfSGMaxActiveSIsperSU as per immfind >>> output below. >>> >>> Anyways, primary reason could be that the configured value of >>> saAmfSGMaxActiveSIsperSU=1 in SG. Since all the SUs got one SI >>> assigned, because of this criteria now there will be no more >>> assignments. So in this case SI1 has only one assignments in SG. >>> >>> But still I have a doubt, why SI2 got two assignments when in the >>> below mentioned "immlist safSi=loadBalancerSI-2,safApp=loadBalancerApp" >>> output, value of saAmfSIPrefActiveAssignments is 1. >>> Could you please check the configuration for the configured value of >>> saAmfSIPrefActiveAssignments for each SI. >>> >>> >>> >>>> >>>> That is not consistent. >>>> >>>> Any idea why this happened? >>>> >>>> Here is the immfind output: >>>> >>>> root@BHA-IND-WHF-KK-CAE-5:~# immfind | grep -i >>>> "safCSIComp=safComp=Lbm_PL-" >>>> safCSIComp=safComp=Lbm_PL-4\,safSu=loadBalancerSU_PL-4\,safSg=loadBala >>>> ncerSG\,safApp=loadBalancerApp,safCsi=LbmCSI_SI-2,safSi=loadBalancerSI >>>> -2,safApp=loadBalancerApp >>>> safCSIComp=safComp=Lbm_PL-6\,safSu=loadBalancerSU_PL-6\,safSg=loadBala >>>> ncerSG\,safApp=loadBalancerApp,safCsi=LbmCSI_SI-2,safSi=loadBalancerSI >>>> -2,safApp=loadBalancerApp >>>> safCSIComp=safComp=Lbm_PL-8\,safSu=loadBalancerSU_PL-8\,safSg=loadBala >>>> ncerSG\,safApp=loadBalancerApp,safCsi=LbmCSI_SI-1,safSi=loadBalancerSI >>>> -1,safApp=loadBalancerApp >>>> root@BHA-IND-WHF-KK-CAE-5:~# >>>> >>>> Here is the immlist output: >>>> >>>> root@BHA-IND-WHF-KK-CAE-5:~# immlist >>>> safSu=loadBalancerSU_PL-4,safSg=loadBalancerSG,safApp=loadBalancerApp >>>> Name Type >>>> Value(s) >>>> ======================================================================== >>>> >>>> safSu SA_STRING_T >>>> safSu=loadBalancerSU_PL-4 >>>> saAmfSUType SA_NAME_T >>>> safVersion=4.0.0,safSuType=MovikSUType (38) >>>> saAmfSURestartCount SA_UINT32_T 0 (0x0) >>>> saAmfSUReadinessState SA_UINT32_T 2 (0x2) >>>> saAmfSURank SA_UINT32_T 1 (0x1) >>>> saAmfSUPresenceState SA_UINT32_T 3 (0x3) >>>> saAmfSUPreInstantiable SA_UINT32_T 1 (0x1) >>>> saAmfSUOperState SA_UINT32_T 1 (0x1) >>>> saAmfSUNumCurrStandbySIs SA_UINT32_T 0 (0x0) >>>> saAmfSUNumCurrActiveSIs SA_UINT32_T 0 (0x0) >>>> saAmfSUMaintenanceCampaign SA_NAME_T <Empty> >>>> saAmfSUHostedByNode SA_NAME_T >>>> safAmfNode=PL-4,safAmfCluster=myAmfCluster (42) >>>> saAmfSUHostNodeOrNodeGroup SA_NAME_T >>>> safAmfNode=PL-4,safAmfCluster=myAmfCluster (42) >>>> saAmfSUFailover SA_UINT32_T <Empty> >>>> saAmfSUAssignedSIs SA_NAME_T <Empty> >>>> saAmfSUAdminState SA_UINT32_T 1 (0x1) >>>> SaImmAttrImplementerName SA_STRING_T >>>> safAmfService >>>> SaImmAttrClassName SA_STRING_T SaAmfSU >>>> SaImmAttrAdminOwnerName SA_STRING_T <Empty> >>>> MovikServiceMode SA_STRING_T <Empty> >>>> >>>> root@BHA-IND-WHF-KK-CAE-5:~# immlist >>>> safSu=loadBalancerSU_PL-6,safSg=loadBalancerSG,safApp=loadBalancerApp >>>> Name Type >>>> Value(s) >>>> ======================================================================== >>>> >>>> safSu SA_STRING_T >>>> safSu=loadBalancerSU_PL-6 >>>> saAmfSUType SA_NAME_T >>>> safVersion=4.0.0,safSuType=MovikSUType (38) >>>> saAmfSURestartCount SA_UINT32_T 0 (0x0) >>>> saAmfSUReadinessState SA_UINT32_T 2 (0x2) >>>> saAmfSURank SA_UINT32_T 1 (0x1) >>>> saAmfSUPresenceState SA_UINT32_T 3 (0x3) >>>> saAmfSUPreInstantiable SA_UINT32_T 1 (0x1) >>>> saAmfSUOperState SA_UINT32_T 1 (0x1) >>>> saAmfSUNumCurrStandbySIs SA_UINT32_T 0 (0x0) >>>> saAmfSUNumCurrActiveSIs SA_UINT32_T 1 (0x1) >>>> saAmfSUMaintenanceCampaign SA_NAME_T <Empty> >>>> saAmfSUHostedByNode SA_NAME_T >>>> safAmfNode=PL-6,safAmfCluster=myAmfCluster (42) >>>> saAmfSUHostNodeOrNodeGroup SA_NAME_T >>>> safAmfNode=PL-6,safAmfCluster=myAmfCluster (42) >>>> saAmfSUFailover SA_UINT32_T <Empty> >>>> saAmfSUAssignedSIs SA_NAME_T <Empty> >>>> saAmfSUAdminState SA_UINT32_T 1 (0x1) >>>> SaImmAttrImplementerName SA_STRING_T >>>> safAmfService >>>> SaImmAttrClassName SA_STRING_T SaAmfSU >>>> SaImmAttrAdminOwnerName SA_STRING_T <Empty> >>>> MovikServiceMode SA_STRING_T <Empty> >>>> >>>> root@BHA-IND-WHF-KK-CAE-5:~# immlist >>>> safSu=loadBalancerSU_PL-8,safSg=loadBalancerSG,safApp=loadBalancerApp >>>> Name Type >>>> Value(s) >>>> ======================================================================== >>>> >>>> safSu SA_STRING_T >>>> safSu=loadBalancerSU_PL-8 >>>> saAmfSUType SA_NAME_T >>>> safVersion=4.0.0,safSuType=MovikSUType (38) >>>> saAmfSURestartCount SA_UINT32_T 0 (0x0) >>>> saAmfSUReadinessState SA_UINT32_T 2 (0x2) >>>> saAmfSURank SA_UINT32_T 1 (0x1) >>>> saAmfSUPresenceState SA_UINT32_T 3 (0x3) >>>> saAmfSUPreInstantiable SA_UINT32_T 1 (0x1) >>>> saAmfSUOperState SA_UINT32_T 1 (0x1) >>>> saAmfSUNumCurrStandbySIs SA_UINT32_T 0 (0x0) >>>> saAmfSUNumCurrActiveSIs SA_UINT32_T 1 (0x1) >>>> saAmfSUMaintenanceCampaign SA_NAME_T <Empty> >>>> saAmfSUHostedByNode SA_NAME_T >>>> safAmfNode=PL-8,safAmfCluster=myAmfCluster (42) >>>> saAmfSUHostNodeOrNodeGroup SA_NAME_T >>>> safAmfNode=PL-8,safAmfCluster=myAmfCluster (42) >>>> saAmfSUFailover SA_UINT32_T <Empty> >>>> saAmfSUAssignedSIs SA_NAME_T <Empty> >>>> saAmfSUAdminState SA_UINT32_T 1 (0x1) >>>> SaImmAttrImplementerName SA_STRING_T >>>> safAmfService >>>> SaImmAttrClassName SA_STRING_T SaAmfSU >>>> SaImmAttrAdminOwnerName SA_STRING_T <Empty> >>>> MovikServiceMode SA_STRING_T <Empty> >>>> >>>> root@BHA-IND-WHF-KK-CAE-5:~# >>>> >>>> >>>> >>>> root@BHA-IND-WHF-KK-CAE-5:~# immlist >>>> safSi=loadBalancerSI-1,safApp=loadBalancerApp >>>> Name Type >>>> Value(s) >>>> ======================================================================== >>>> >>>> safSi SA_STRING_T >>>> safSi=loadBalancerSI-1 >>>> saAmfSvcType SA_NAME_T >>>> safVersion=4.0.0,safSvcType=MovikSvcType (40) >>>> saAmfSIStandbyWeight SA_STRING_T <Empty> >>>> saAmfSIRank SA_UINT32_T 0 (0x0) >>>> saAmfSIProtectedbySG SA_NAME_T >>>> safSg=loadBalancerSG,safApp=loadBalancerApp (43) >>>> saAmfSIPrefStandbyAssignments SA_UINT32_T 1 (0x1) >>>> saAmfSIPrefActiveAssignments SA_UINT32_T 1 (0x1) >>>> saAmfSINumCurrStandbyAssignments SA_UINT32_T 0 (0x0) >>>> saAmfSINumCurrActiveAssignments SA_UINT32_T 1 (0x1) >>>> saAmfSIAssignmentState SA_UINT32_T 2 (0x2) >>>> saAmfSIAdminState SA_UINT32_T 1 (0x1) >>>> saAmfSIActiveWeight SA_STRING_T <Empty> >>>> SaImmAttrImplementerName SA_STRING_T >>>> safAmfService >>>> SaImmAttrClassName SA_STRING_T SaAmfSI >>>> SaImmAttrAdminOwnerName SA_STRING_T <Empty> >>>> >>>> root@BHA-IND-WHF-KK-CAE-5:~# immlist >>>> safSi=loadBalancerSI-2,safApp=loadBalancerApp >>>> Name Type >>>> Value(s) >>>> ======================================================================== >>>> >>>> safSi SA_STRING_T >>>> safSi=loadBalancerSI-2 >>>> saAmfSvcType SA_NAME_T >>>> safVersion=4.0.0,safSvcType=MovikSvcType (40) >>>> saAmfSIStandbyWeight SA_STRING_T <Empty> >>>> saAmfSIRank SA_UINT32_T 0 (0x0) >>>> saAmfSIProtectedbySG SA_NAME_T >>>> safSg=loadBalancerSG,safApp=loadBalancerApp (43) >>>> saAmfSIPrefStandbyAssignments SA_UINT32_T 1 (0x1) >>>> saAmfSIPrefActiveAssignments SA_UINT32_T 1 (0x1) >>>> saAmfSINumCurrStandbyAssignments SA_UINT32_T 0 (0x0) >>>> saAmfSINumCurrActiveAssignments SA_UINT32_T 1 (0x1) >>>> saAmfSIAssignmentState SA_UINT32_T 2 (0x2) >>>> saAmfSIAdminState SA_UINT32_T 1 (0x1) >>>> saAmfSIActiveWeight SA_STRING_T <Empty> >>>> SaImmAttrImplementerName SA_STRING_T >>>> safAmfService >>>> SaImmAttrClassName SA_STRING_T SaAmfSI >>>> SaImmAttrAdminOwnerName SA_STRING_T <Empty> >>>> >>>> root@BHA-IND-WHF-KK-CAE-5:~# >>>> >>>> root@BHA-IND-WHF-KK-CAE-5:~# immlist >>>> safSg=loadBalancerSG,safApp=loadBalancerApp >>>> Name Type >>>> Value(s) >>>> ======================================================================== >>>> >>>> safSg SA_STRING_T >>>> safSg=loadBalancerSG >>>> saAmfSGType SA_NAME_T >>>> safVersion=4.0.0,safSgType=MovikSGTypeNWayActive (48) >>>> saAmfSGSuRestartProb SA_TIME_T <Empty> >>>> saAmfSGSuRestartMax SA_UINT32_T <Empty> >>>> saAmfSGSuHostNodeGroup SA_NAME_T <Empty> >>>> saAmfSGNumPrefStandbySUs SA_UINT32_T 0 (0x0) >>>> saAmfSGNumPrefInserviceSUs SA_UINT32_T 100 >>>> (0x64) >>>> saAmfSGNumPrefAssignedSUs SA_UINT32_T 100 >>>> (0x64) >>>> saAmfSGNumPrefActiveSUs SA_UINT32_T 100 >>>> (0x64) >>>> saAmfSGNumCurrNonInstantiatedSpareSUs SA_UINT32_T 0 (0x0) >>>> saAmfSGNumCurrInstantiatedSpareSUs SA_UINT32_T 1 (0x1) >>>> saAmfSGNumCurrAssignedSUs SA_UINT32_T 2 (0x2) >>>> saAmfSGMaxStandbySIsperSU SA_UINT32_T <Empty> >>>> saAmfSGMaxActiveSIsperSU SA_UINT32_T 1 (0x1) >>>> saAmfSGCompRestartProb SA_TIME_T <Empty> >>>> saAmfSGCompRestartMax SA_UINT32_T <Empty> >>>> saAmfSGAutoRepair SA_UINT32_T 1 (0x1) >>>> saAmfSGAutoAdjustProb SA_TIME_T <Empty> >>>> saAmfSGAutoAdjust SA_UINT32_T 0 (0x0) >>>> saAmfSGAdminState SA_UINT32_T 1 (0x1) >>>> SaImmAttrImplementerName SA_STRING_T >>>> safAmfService >>>> SaImmAttrClassName SA_STRING_T SaAmfSG >>>> SaImmAttrAdminOwnerName SA_STRING_T <Empty> >>>> >>>> Thanks. >>>> >>>> Kang-sen >>>> >>>> ---------------------------------------------------------------------- >>>> -------- Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T >>>> Park in San Francisco, CA to explore cutting-edge tech and listen to >>>> tech luminaries present their vision of the future. This family event >>>> has something for everyone, including kids. Get more information and >>>> register today. >>>> http://sdm.link/attshape >>>> _______________________________________________ >>>> Opensaf-users mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/opensaf-users >>>> ------------------------------------------------------------------------------ Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San Francisco, CA to explore cutting-edge tech and listen to tech luminaries present their vision of the future. This family event has something for everyone, including kids. Get more information and register today. http://sdm.link/attshape _______________________________________________ Opensaf-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/opensaf-users
