On Mon, 2006-11-06 at 13:13, Wang, Feiyi wrote: > Hal - > > Please see the output for active port 1 (although there are two ports on > this HCA, the second one is disabled now). > > #smpquery portinfo 8 1 > # Port info: Lid 8 port 1 > Mkey:............................0x0000000000000000 > GidPrefix:.......................0xfe80000000000000 > Lid:.............................0x0008 > SMLid:...........................0x0001 > CapMask:.........................0x2510a68 > IsTrapSupported > IsAutomaticMigrationSupported > IsSLMappingSupported > IsLedInfoSupported > IsSystemImageGUIDsupported > IsCommunicatonManagementSupported > IsVendorClassSupported > IsCapabilityMaskNoticeSupported > IsClientRegistrationSupported > DiagCode:........................0x0000 > MkeyLeasePeriod:.................0 > LocalPort:.......................1 > LinkWidthEnabled:................1X or 4X > LinkWidthSupported:..............1X or 4X > LinkWidthActive:.................4X > LinkSpeedSupported:..............2.5 or 5.0 Gbps > LinkState:.......................Active > PhysLinkState:...................LinkUp > LinkDownDefState:................Polling > ProtectBits:.....................0 > LMC:.............................0 > LinkSpeedActive:.................2.5 Gbps > LinkSpeedEnabled:................2.5 or 5.0 Gbps > NeighborMTU:.....................2048 > SMSL:............................0 > VLCap:...........................VL0-7 > InitType:........................0x00 > VLHighLimit:.....................255
OK; this is pretty conclusive. > VLArbHighCap:....................8 > VLArbLowCap:.....................8 > InitReply:.......................0x00 > MtuCap:..........................2048 > VLStallCount:....................7 > HoqLife:.........................31 > OperVLs:.........................VL0-7 > PartEnforceInb:..................0 > PartEnforceOutb:.................0 > FilterRawInb:....................0 > FilterRawOutb:...................0 > MkeyViolations:..................0 > PkeyViolations:..................0 > QkeyViolations:..................0 > GuidCap:.........................32 > ClientReregister:................0 > SubnetTimeout:...................18 > RespTimeVal:.....................16 > LocalPhysErr:....................8 > OverrunErr:......................8 > MaxCreditHint:...................0 > RoundTrip:.......................0 Do you have an IB analyzer ? -- Hal > Feiyi > > -----Original Message----- > From: Hal Rosenstock [mailto:[EMAIL PROTECTED] > Sent: Friday, November 03, 2006 3:58 PM > To: Wang, Feiyi > Cc: [email protected] > Subject: RE: [openib-general] question on QoS support > > On Fri, 2006-11-03 at 15:56, Wang, Feiyi wrote: > > 255 > > > > I think I tested with default 0 before, that is send at most one > packet > > before give low priority table the chance according to IBA. It doesn't > > seem to make a difference though. > > I was hoping you would say 0 as that means 1 packet before looking at > low priority. > > 255 means unbounded packets on high priority. Can you send me the > results of smpquery portinfo on that port to ensure that it is being set > properly ? > > -- Hal > > > Feiyi > > > > > > -----Original Message----- > > From: Hal Rosenstock [mailto:[EMAIL PROTECTED] > > Sent: Friday, November 03, 2006 3:51 PM > > To: Wang, Feiyi > > Cc: [email protected] > > Subject: RE: [openib-general] question on QoS support > > > > On Fri, 2006-11-03 at 15:43, Wang, Feiyi wrote: > > > The test is done on two hosts, say A and B. A has 4x SDR (run > > ib_rdam_bw > > > as server), B has 4x DDR (run more than one thread of ib_rdma_bw as > > > clients). The sl2vl table read as: > > > > > > smpquery sl2vl 7 > > > # SL2VL table: Lid 7 > > > # SL: | 0| 1| 2| 3| 4| 5| 6| 7| 8| > > 9|10|11|12|13|14|15| > > > ports: in 0, out 0: | 0| 1| 2| 3| 4| 5| 6| 7| 0| 1| 2| 3| 4| 5| 6| > > 7| > > > > > > smpquery vlarb 7 > > > # VLArbitration tables: Lid 7 port 0 LowCap 8 HighCap 8 > > > # Low priority VL Arbitration Table: > > > VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x6 |0x7 | > > > WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 | > > > # High priority VL Arbitration Table: > > > VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x6 |0x7 | > > > WEIGHT: |0x1 |0x0 |0x8 |0x0 |0x0 |0x0 |0x0 |0x0 | > > > > > > Low priority table entries are all zero to skip. > > > High priority table give VL 0 and VL 2 different weight. > > > > > > The SL is specified on command line, one thread with SL 0, the other > > > thread with SL 2. > > > > > > Thanks for looking into this, and let me know if more info is > needed. > > > > What's the limit of high priority ? > > > > -- Hal > > > > > Feiyi > > > > > > > > > > > > -----Original Message----- > > > From: Hal Rosenstock [mailto:[EMAIL PROTECTED] > > > Sent: Friday, November 03, 2006 3:27 PM > > > To: Wang, Feiyi > > > Cc: [email protected] > > > Subject: Re: [openib-general] question on QoS support > > > > > > On Fri, 2006-11-03 at 15:12, Feiyi Wang wrote: > > > > In our test at the ORNL - it appears you can "turn off" the > traffic > > by > > > > giving every VL weight 0. > > > > > > A weight of 0 indicates to skip that entry. > > > > > > > As soon as you assign non-zero VL weight, > > > > the traffic starts to flow, however, VL with more weight doesn't > > have > > > > expected preference treatment. In other words, traffic shaping > > didn't > > > > take place. smpquery vlarb verified the mapping table was there. > > > > > > correctly ? > > > > > > Is it high or low priority or both ? > > > > > > What about SL2VLMapping table ? Is it setup correctly ? > > > > > > What's your topology for this ? > > > > > > Can you send your SL2VLMapping and VLarbitration configuration ? > > > > > > > I believe the scenario described below 'should' be able to > generate > > > > congestion point ... but it would be helpful if someone can > > elaborate > > > > a way to "look into" how/if scheduling/arbitration take place. > > > > > > The only ways I know would be to look at either the packets on the > > wire > > > or what you are doing with multiple streams which seems valid to me. > > > > > > Have you read section 7.6.9.2 (p. 189-190) in IBA 1.2 volume 1 to > > > understand how to configure this ? > > > > > > -- Hal > > > > > > > Best, > > > > > > > > Feiyi > > > > > > > > > > > > On 02 Nov 2006 10:49:04 -0500, Hal Rosenstock <[EMAIL PROTECTED]> > > > wrote: > > > > > Hi Oliver, > > > > > > > > > > On Thu, 2006-11-02 at 10:20, Oliver wrote: > > > > > > Hi, Hal - > > > > > > > > > > > > > How is this being observed/measured ? > > > > > > > > > > > > Host A, B, with 4x DDR both connected to Flextronic switch. > > > > > > A single process of ibv_read_bw gives about 1415MB /s average > > > > > > bandwidth. Two concurrent process report 714.45 MB/s each, > dead > > > even. > > > > > > Now if I bump up one process with a different SL, then I > expect > > to > > > see > > > > > > shaping to take place. Please let me if the scenario makes > > sense. > > > > > > > > > > It makes sense. However, if the higher priority traffic does not > > > fill > > > > > the scheduling, the low priority can take up the slack so I'm > not > > > sure > > > > > if this is what you are seeing or something else. > > > > > > > > > > It might be interesting to try the same thing at SDR speeds. > > > > > > > > > > -- Hal > > > > > > > > > > > > Yes, 8 VLs should be supported in your subnet. You can > verify > > > this with > > > > > > > smpquery portinfo on the HCA port and examine OperVLs > assuming > > > the port > > > > > > > is ACTIVE. > > > > > > > > > > > > yes, I verified the data VL support, it is 8. I will poke for > > more > > > > > > info with suggested commands by Sasha. > > > > > > > > > > > > > > A related question is, if I modify qos setting in SM, do I > > > need to > > > > > > > > restart SA on each hosts for it to see the changes? (I am > > > hoping not, > > > > > > > > as I tried in the test, it doesn't seem to make a > > difference) > > > > > > > > > > > > > > Not sure what you mean. SA is tightly coupled with the > OpenSM. > > > Do you > > > > > > > mean SA client ? The client hosts don't need restarting but > > did > > > you > > > > > > > restart OpenSM with your QoS configuration ? > > > > > > > > > > > > I mean client SA. yes, I understand OpenSM needs to be > > restarted. > > > > > > > > > > > > > BTW, which OpenSM are you running ? > > > > > > > > > > > > OFED 1.1 based. > > > > > > > > > > > > thanks > > > > > > > > > > > > - Oliver > > > > > > > > > > > > > > > > _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
