Bob, On Fri, May 14, 2010 at 3:55 PM, Bob Ciotti <[email protected]> wrote: > We are chasing down some issues related to fabric discovery and SM failover > and occasionally we see a significant number of vl15 drops and are not sure > if this is a problem or not. Anyone have a reference on what controls are > available to reduce the number of drops? We would like to keep maxsmps set to > at > least 16 or even raise it. Spec says at least one vl15 buffer, but are there > controls for buffer allocation on the switches wrt vl15, or other vl15 > specific controls (in SM or switches or?) that might reduce drops?
maxsmps exceeding the number of VL15 buffers available in the devices can cause such drops. Switches may have controls over configuring such buffering but it would be vendor proprietary. Does reducing maxsmps reduce the VL15 dropping ? > Its a collection of InfiniscaleIII and IV. > > Here is a small subset of the VL15 counters on a few on the switches. > Each line corresponds to a 24 port infiniscaleIII switch. All the d1 > thru d11 (hypercube dimensions) connections are to other switches (not HCA). > > r53i3/sw0/port 24 is where the SM is connected showing 9678 drops... > > > -- cb1 ib0 sw0 . . . . d1 d2 d3 d4 > d5 d6 d7 d8 d9 d10 d11 d12 io > == cb1 ib0 sw0 . . . SWPORT 9 13 14 15 > 16 17 18 19 20 21 22 23 24 > r49i0 cb1 ib0 sw0 SwLid 7842 port-1 VL15Dropped: . . 1 3 > 8 13 11 . . . . . . > r49i1 cb1 ib0 sw0 SwLid 8349 port-1 VL15Dropped: . . 17 8 > 20 16 2 2 . . . . . > r49i2 cb1 ib0 sw0 SwLid 8554 port-1 VL15Dropped: . 4 . 12 > 23 2 2 . 2 . . . . > r49i3 cb1 ib0 sw0 SwLid 7330 port-1 VL15Dropped: 3 1 1 25 > 71 57 7 8 1 . . . . > r50i0 cb1 ib0 sw0 SwLid 8682 port-1 VL15Dropped: . 2 61 3 > 6 7 . 5 . . . . . > r50i1 cb1 ib0 sw0 SwLid 8132 port-1 VL15Dropped: . 3 58 33 > 20 44 1 9 4 . . . . > r50i2 cb1 ib0 sw0 SwLid 6435 port-1 VL15Dropped: 4 122 14 8 > 13 36 2 . . . . . . > r50i3 cb1 ib0 sw0 SwLid 8027 port-1 VL15Dropped: 22 171 167 49 > 113 57 . 2 . . . . . > r51i0 cb1 ib0 sw0 SwLid 7756 port-1 VL15Dropped: . . 3 24 > 16 11 . 2 . . . . . > r51i1 cb1 ib0 sw0 SwLid 6678 port-1 VL15Dropped: . 5 8 28 > 7 27 2 . 4 . . . . > r51i2 cb1 ib0 sw0 SwLid 7933 port-1 VL15Dropped: 1 34 29 3 > 7 5 2 . . . . . . > r51i3 cb1 ib0 sw0 SwLid 7426 port-1 VL15Dropped: 14 31 145 683 > 18 5 2 . 1 . . . . > r52i0 cb1 ib0 sw0 SwLid 6990 port-1 VL15Dropped: 1 26 65 23 > 132 49 1 1 1 . . . . > r52i1 cb1 ib0 sw0 SwLid 7465 port-1 VL15Dropped: 33 66 388 48 > 18 4 4 . 2 . . . . > r52i2 cb1 ib0 sw0 SwLid 6914 port-1 VL15Dropped: 17 320 39 14 > 15 12 4 5 4 . . . . > r52i3 cb1 ib0 sw0 SwLid 7895 port-1 VL15Dropped: 1770 614 1449 189 > 299 197 35 124 56 . 55 . 9678 > r53i0 cb1 ib0 sw0 SwLid 8171 port-1 VL15Dropped: . . . . > 1 5 . . . . . . . > r53i1 cb1 ib0 sw0 SwLid 8471 port-1 VL15Dropped: . . 1 . > . 15 3 1 5 . . . . > r53i2 cb1 ib0 sw0 SwLid 6485 port-1 VL15Dropped: . . . . > 8 17 2 . 2 . . . . > r53i3 cb1 ib0 sw0 SwLid 5949 port-1 VL15Dropped: . . 1 3 > 3 1 8 . . . . . . > r54i0 cb1 ib0 sw0 SwLid 7809 port-1 VL15Dropped: . . . . > 2 . 2 . . . . . . > r54i1 cb1 ib0 sw0 SwLid 7890 port-1 VL15Dropped: . . . 4 > 16 1 . . 2 . . . . > r54i2 cb1 ib0 sw0 SwLid 6913 port-1 VL15Dropped: . 1 . 1 > 21 4 4 . 3 . . . . > r54i3 cb1 ib0 sw0 SwLid 7283 port-1 VL15Dropped: 1 11 1 7 > 34 4 18 . 2 . . . . > r55i0 cb1 ib0 sw0 SwLid 8350 port-1 VL15Dropped: . . . 2 > . 8 7 . . . . . . > r55i1 cb1 ib0 sw0 SwLid 7516 port-1 VL15Dropped: . . 3 24 > 13 3 4 . 1 . . . . > r55i2 cb1 ib0 sw0 SwLid 8518 port-1 VL15Dropped: . 6 . 16 > . 1 4 1 . . . . . > r55i3 cb1 ib0 sw0 SwLid 7666 port-1 VL15Dropped: . . 9 12 > 4 4 20 1 . . . . . > r56i0 cb1 ib0 sw0 SwLid 8219 port-1 VL15Dropped: . 3 29 12 > . 6 . . 1 . . . . > r56i1 cb1 ib0 sw0 SwLid 5912 port-1 VL15Dropped: . 3 11 13 > 4 13 . . 2 . . . . > r56i2 cb1 ib0 sw0 SwLid 7329 port-1 VL15Dropped: 1 105 39 27 > 1 1 1 . . . . . . > r56i3 cb1 ib0 sw0 SwLid 7424 port-1 VL15Dropped: 14 6 11 77 > 10 46 4 2 . . . Are these on both IS-3s and IS-4s ? Any idea on "when" the drops occur ? Are there any messages in the OpenSM log which might relate to this (e.g. duplicated traps received frequently) ? -- Hal . . > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to [email protected] > More majordomo info at http://vger.kernel.org/majordomo-info.html >
