On 10-May-12 3:42 PM, Yevgeny Kliteynik wrote:
>
> Here are the ActiveSpeed values:
>
> 1: SDR - 2.5 Gb/s * (8/10 coding)
> 2: DDR - 5 Gb/s * (8/10 coding)
> 4: QDR - 10 Gb/s * (8/10 coding)
> 8: FDR10 - 10 Gb/s * (64/66 coding)
> 16: FDR - 14 Gbps * (64/66 coding)
> 32: EDR - 25 Gbs * (64/66 coding)
>
> EDR is not out yet, so we can ignore it for the sake of this patch.
> It is also true that *most* of IB cables are 4x.
> There are 12x clusters out there (like TACC's Ranger), and sometimes
> there are 1x links (mostly because the cable is not perfect or too long).
>
> So cases of 4 and 8 are not the same for us, because MPI needs actual
> *data* rate instead of *bit* rate, which is different in these cases.
>
> I'll send a patch for review in a minute...
>
How about this patch? If it looks OK, I'll commit to trunk and open CMR.
ndex: btl_openib_component.c
===================================================================
--- btl_openib_component.c (revision 26420)
+++ btl_openib_component.c (working copy)
@@ -810,8 +810,10 @@
we have to look up the values corresponding to
port->active_speed and port->active_width. These
are enums corresponding to the IB spec. Overall
- forumula is 80% of the reported speed (to get the
- true link speed) times the number of links. */
+ forumula to get the true link speed is 8/10 or
+ 64/66 of the reported speed (depends on the coding
+ that is being used for the particular speed) times
+ the number of links. */
switch (ib_port_attr->active_speed) {
case 1:
/* 2.5Gbps * 0.8, in megabits */
@@ -826,9 +828,13 @@
openib_btl->super.btl_bandwidth = 8000;
break;
case 8:
- /* 40.0Gbps * 0.8, in megabits */
- openib_btl->super.btl_bandwidth = 32000;
+ /* FDR10: 10.0Gbps * 64/66, in megabits */
+ openib_btl->super.btl_bandwidth = 9700;
break;
+ case 16:
+ /* FDR: 14.0Gbps * 64/66, in megabits */
+ openib_btl->super.btl_bandwidth = 13600;
+ break;
default:
/* Who knows? Declare this port unreachable (do
*not* return ERR_VALUE_OF_OUT_OF_BOUNDS; that
