Attached is a text file (unix file format) with the port mappings. As
you surmised, the leaf chips are in slots 9, 10, 13, 14, and the spines
are in 11, 12. Our nomenclature is that the leaf cards are "Node" cards
(Node 1 = Slot 9, Node 2 = Slot 10, etc.) and the spine cards are "Core"
cards (Core 1 = slot 11, Core 2 = slot 12). I apologize for the
somewhat goofy naming; it came from the early hardware design.
If you're running the Cisco SM, we have recently released a pretty
general-purpose network debugging/maintenance tool called FACT which may
help you out
(http://tools.cisco.com/support/downloads/go/Redirect.x?mdfid=279966112
-- look under "Server Fabric Software"). It requires a cisco.com login,
but not a support contract.
-Ted.
P.S. I don't generally keep up with ofa-general (my apologies), so
please cc' me on any replies, flames, etc.
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Charles
Taylor
Sent: Thursday, July 10, 2008 4:15 PM
To: general-list
Subject: Re: [ofa-general] Cisco 7008 Internal/External Port Mappings
Nope. Looks pretty much the same as ibnetdiscover without the "-
g". It would be really useful if it *did* work. :)
Thanks for the suggestion though. I think we just may be stuck
mapping it empirically. I'm sure the pattern will become obvious
quickly enough.
Charlie Taylor
UF HPC Center
On Jul 10, 2008, at 5:36 PM, Hal Rosenstock wrote:
> On Thu, 2008-07-10 at 17:27 -0400, Charles Taylor wrote:
>> We've got a couple of Cisco 7008/Topspin 270 core switches. Much
>> of
>> the diagnostic and topology information we get from the ib diag tools
>> in OFED refer to the internal ports rather than the actual external
>> slot/port.
>
> Does ibnetdiscover -g work ? There was some work done to do this
> mapping for Cisco but it may be out of date.
>
> -- Hal
>
>> We could deduce the mappings, I suppose, by tracing cables and
>> using ibtracert between nodes but that seems like a real
>> pain. I was wondering if anyone on this list knows how that
>> mapping
>> works. The chassis itself consists of 8 "ib port" blades in the
>> front labelled as slots 1 - 8. There are then six "fabric
>> controller" slots in the back of the chassis labelled 9 - 14.
>> Slots 11 and 12 are "special" as they hold the "master" and "standby
>> master" fabric controllers.
>>
>> Seems like the FC's in slots 11 and 12 connect the the FCs in slots
>> 9,10,13,14 which in turn connect to the external port blades in slots
>> 1 - 8.
>>
>> Again, all of the diagnostic information refers to the internal FC
>> ports and mapping those to the external ports (to which we actually
>> connect leaf switches) seems non-trivial. I think there was even a
>> document somewhere that described the mapping. We just can't find
>> it
>> now.
>>
>> Thanks,
>>
>> Charlie Taylor
>> UF HPC Center
>> _______________________________________________
>> general mailing list
>> [email protected]
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit
>> http://openib.org/mailman/listinfo/openib-general
>
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
1.1.1 Switch Fabric Port Mapping
==================================
Table 1: External IB Port Mapping
==================================
4x Port 12x Port Node / Switch # / Port #
------- -------- ------------------------
LIM 1, port 1 LIM 1, port 1 Node 1, Switch 2, Port 18 (1)
LIM 1, port 2 Node 1, Switch 2, Port 17 (1)
LIM 1, port 3 Node 1, Switch 2, Port 16 (1)
LIM 1, port 4 LIM 1, port 2 Node 1, Switch 2, Port 15 (1)
LIM 1, port 5 Node 1, Switch 2, Port 14 (1)
LIM 1, port 6 Node 1, Switch 2, Port 13 (1)
LIM 1, port 7 LIM 1, port 3 Node 1, Switch 1, Port 22
LIM 1, port 8 Node 1, Switch 1, Port 23
LIM 1, port 9 Node 1, Switch 1, Port 24
LIM 1, port 10 LIM 1, port 4 Node 1, Switch 1, Port 19
LIM 1, port 11 Node 1, Switch 1, Port 20
LIM 1, port 12 Node 1, Switch 1, Port 21
LIM 2, port 1 LIM 2, port 1 Node 1, Switch 2, Port 22 (2)
LIM 2, port 2 Node 1, Switch 2, Port 23 (2)
LIM 2, port 3 Node 1, Switch 2, Port 24 (2)
LIM 2, port 4 LIM 2, port 2 Node 1, Switch 2, Port 19 (2)
LIM 2, port 5 Node 1, Switch 2, Port 20 (2)
LIM 2, port 6 Node 1, Switch 2, Port 21 (2)
LIM 2, port 7 LIM 2, port 3 Node 1, Switch 1, Port 18 (1) (2)
LIM 2, port 8 Node 1, Switch 1, Port 17 (1) (2)
LIM 2, port 9 Node 1, Switch 1, Port 16 (1) (2)
LIM 2, port 10 LIM 2, port 4 Node 1, Switch 1, Port 15 (1) (2)
LIM 2, port 11 Node 1, Switch 1, Port 14 (1) (2)
LIM 2, port 12 Node 1, Switch 1, Port 13 (1) (2)
LIM 3, port 1 LIM 3, port 1 Node 2, Switch 2, Port 22 (2)
LIM 3, port 2 Node 2, Switch 2, Port 23 (2)
LIM 3, port 3 Node 2, Switch 2, Port 24 (2)
LIM 3, port 4 LIM 3, port 2 Node 2, Switch 2, Port 19 (2)
LIM 3, port 5 Node 2, Switch 2, Port 20 (2)
LIM 3, port 6 Node 2, Switch 2, Port 19 (2)
LIM 3, port 7 LIM 3, port 3 Node 2, Switch 1, Port 18 (1) (2)
LIM 3, port 8 Node 2, Switch 1, Port 17 (1) (2)
LIM 3, port 9 Node 2, Switch 1, Port 16 (1) (2)
LIM 3, port 10 LIM 3, port 4 Node 2, Switch 1, Port 15 (1) (2)
LIM 3, port 11 Node 2, Switch 1, Port 14 (1) (2)
LIM 3, port 12 Node 2, Switch 1, Port 13 (1) (2)
LIM 4, port 1 LIM 4, port 1 Node 2, Switch 2, Port 18 (1)
LIM 4, port 2 Node 2, Switch 2, Port 17 (1)
LIM 4, port 3 Node 2, Switch 2, Port 16 (1)
LIM 4, port 4 LIM 4, port 2 Node 2, Switch 2, Port 15 (1)
LIM 4, port 5 Node 2, Switch 2, Port 14 (1)
LIM 4, port 6 Node 2, Switch 2, Port 13 (1)
LIM 4, port 7 LIM 4, port 3 Node 2, Switch 1, Port 22
LIM 4, port 8 Node 2, Switch 1, Port 23
LIM 4, port 9 Node 2, Switch 1, Port 24
LIM 4, port 10 LIM 4, port 4 Node 2, Switch 1, Port 19
LIM 4, port 11 Node 2, Switch 1, Port 20
LIM 4, port 12 Node 2, Switch 1, Port 21
LIM 5, port 1 LIM 5, port 1 Node 3, Switch 2, Port 18 (1)
LIM 5, port 2 Node 3, Switch 2, Port 17 (1)
LIM 5, port 3 Node 3, Switch 2, Port 16 (1)
LIM 5, port 4 LIM 5, port 2 Node 3, Switch 2, Port 15 (1)
LIM 5, port 5 Node 3, Switch 2, Port 14 (1)
LIM 5, port 6 Node 3, Switch 2, Port 13 (1)
LIM 5, port 7 LIM 5, port 3 Node 3, Switch 1, Port 22
LIM 5, port 8 Node 3, Switch 1, Port 23
LIM 5, port 9 Node 3, Switch 1, Port 24
LIM 5, port 10 LIM 5, port 4 Node 3, Switch 1, Port 19
LIM 5, port 11 Node 3, Switch 1, Port 20
LIM 5, port 12 Node 3, Switch 1, Port 21
LIM 6, port 1 LIM 6, port 1 Node 3, Switch 2, Port 22 (2)
LIM 6, port 2 Node 3, Switch 2, Port 23 (2)
LIM 6, port 3 Node 3, Switch 2, Port 24 (2)
LIM 6, port 4 LIM 6, port 2 Node 3, Switch 2, Port 19 (2)
LIM 6, port 5 Node 3, Switch 2, Port 20 (2)
LIM 6, port 6 Node 3, Switch 2, Port 21 (2)
LIM 6, port 7 LIM 6, port 3 Node 3, Switch 1, Port 18 (1) (2)
LIM 6, port 8 Node 3, Switch 1, Port 17 (1) (2)
LIM 6, port 9 Node 3, Switch 1, Port 16 (1) (2)
LIM 6, port 10 LIM 6, port 4 Node 3, Switch 1, Port 15 (1) (2)
LIM 6, port 11 Node 3, Switch 1, Port 14 (1) (2)
LIM 6, port 12 Node 3, Switch 1, Port 13 (1) (2)
LIM 7, port 1 LIM 7, port 1 Node 4, Switch 2, Port 22 (2)
LIM 7, port 2 Node 4, Switch 2, Port 23 (2)
LIM 7, port 3 Node 4, Switch 2, Port 24 (2)
LIM 7, port 4 LIM 7, port 2 Node 4, Switch 2, Port 19 (2)
LIM 7, port 5 Node 4, Switch 2, Port 20 (2)
LIM 7, port 6 Node 4, Switch 2, Port 21 (2)
LIM 7, port 7 LIM 7, port 3 Node 4, Switch 1, Port 18 (1) (2)
LIM 7, port 8 Node 4, Switch 1, Port 17 (1) (2)
LIM 7, port 9 Node 4, Switch 1, Port 16 (1) (2)
LIM 7, port 10 LIM 7, port 4 Node 4, Switch 1, Port 15 (1) (2)
LIM 7, port 11 Node 4, Switch 1, Port 14 (1) (2)
LIM 7, port 12 Node 4, Switch 1, Port 13 (1) (2)
LIM 8, port 1 LIM 8, port 1 Node 4, Switch 2, Port 18 (1)
LIM 8, port 2 Node 4, Switch 2, Port 17 (1)
LIM 8, port 3 Node 4, Switch 2, Port 16 (1)
LIM 8, port 4 LIM 8, port 2 Node 4, Switch 2, Port 15 (1)
LIM 8, port 5 Node 4, Switch 2, Port 14 (1)
LIM 8, port 6 Node 4, Switch 2, Port 13 (1)
LIM 8, port 7 LIM 8, port 3 Node 4, Switch 1, Port 22
LIM 8, port 8 Node 4, Switch 1, Port 23
LIM 8, port 9 Node 4, Switch 1, Port 24
LIM 8, port 10 LIM 8, port 4 Node 4, Switch 1, Port 19
LIM 8, port 11 Node 4, Switch 1, Port 20
LIM 8, port 12 Node 4, Switch 1, Port 21
Note
(1): These ports are routed with lane reversal and require the
switch TX reversal on
(2): These ports are routed with polarity inversion and require the switch
TX pol, port inversion on
====================================
Table 2: Core to Node Port Mapping
(Internal IB Port Mapping)
====================================
Node / Switch Port # Core / Switch Port #
-------------------- --------------------
Node 1, Switch 1, Ports 1,2,3 Core 2, Switch 2, Ports 13,14,15
Node 1, Switch 1, Ports 4,5,6 Core 1, Switch 2, Ports 13,14,15
Node 1, Switch 1, Ports 7,8,9 Core 2, Switch 1, Ports 15,14,13 (3)(4)
Node 1, Switch 1, Ports 10,11,12 Core 1, Switch 1, Ports 15,14,13 (3)(4)
Node 1, Switch 2, Ports 1,2,3 Core 2, Switch 2, Ports 21,20,19 (3)(4)
Node 1, Switch 2, Ports 4,5,6 Core 1, Switch 2, Ports 21,20,19 (3)(4)
Node 1, Switch 2, Ports 7,8,9 Core 2, Switch 1, Ports 19,20,21
Node 1, Switch 2, Ports 10,11,12 Core 1, Switch 1, Ports 19,20,21
Node 2, Switch 1, Ports 1,2,3 Core 2, Switch 2, Ports 12,11,10 (4)
Node 2, Switch 1, Ports 4,5,6 Core 1, Switch 2, Ports 12,11,10 (4)
Node 2, Switch 1, Ports 7,8,9 Core 2, Switch 1, Ports 10,11,12
Node 2, Switch 1, Ports 10,11,12 Core 1, Switch 1, Ports 10,11,12
Node 2, Switch 2, Ports 1,2,3 Core 2, Switch 2, Ports 4,5,6
Node 2, Switch 2, Ports 4,5,6 Core 1, Switch 2, Ports 4,5,6
Node 2, Switch 2, Ports 7,8,9 Core 2, Switch 1, Ports 6,5,4 (3)(4)
Node 2, Switch 2, Ports 10,11,12 Core 1, Switch 1, Ports 6,5,4 (3)(4)
Node 3, Switch 1, Ports 1,2,3 Core 2, Switch 2, Ports 9,8,7 (4)
Node 3, Switch 1, Ports 4,5,6 Core 1, Switch 2, Ports 9,8,7 (4)
Node 3, Switch 1, Ports 7,8,9 Core 2, Switch 1, Ports 7,8,9
Node 3, Switch 1, Ports 10,11,12 Core 1, Switch 1, Ports 7,8,9
Node 3, Switch 2, Ports 1,2,3 Core 2, Switch 2, Ports 1,2,3
Node 3, Switch 2, Ports 4,5,6 Core 1, Switch 2, Ports 1,2,3
Node 3, Switch 2, Ports 7,8,9 Core 2, Switch 1, Ports 3,2,1 (3)(4)
Node 3, Switch 2, Ports 10,11,12 Core 1, Switch 1, Ports 3,2,1 (3)(4)
Node 4, Switch 1, Ports 1,2,3 Core 2, Switch 2, Ports 16,17,18 (3)
Node 4, Switch 1, Ports 4,5,6 Core 1, Switch 2, Ports 16,17,18 (3)
Node 4, Switch 1, Ports 7,8,9 Core 2, Switch 1, Ports 18,17,16 (3)(4)
Node 4, Switch 1, Ports 10,11,12 Core 1, Switch 1, Ports 18,17,16 (3)(4)
Node 4, Switch 2, Ports 1,2,3 Core 2, Switch 2, Ports 24,23,22 (3)(4)
Node 4, Switch 2, Ports 4,5,6 Core 1, Switch 2, Ports 24,23,22 (3)(4)
Node 4, Switch 2, Ports 7,8,9 Core 2, Switch 1, Ports 22,23,24
Node 4, Switch 2, Ports 10,11,12 Core 1, Switch 1, Ports 22,23,24
Note
(3): These ports are routed with polarity reversal
(4): These ports are routed with lane reversal
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
