Re: Dell Poweedge 750 Mellanox ConnectX-6 LX with 1G SFP SX

2024-03-14 Thread Jonathan Matthew via misc
On Thu, Mar 14, 2024 at 10:13:52AM +0100, Joerg Streckfuss wrote:
> 
> Hi misc,
> 
> I am trying to connect a 1GE SFP of type 1G SFP SX (Flexoptix S.8512.02.D)
> to a Power Edge R750 with a Connect Mellanox ConnectX-6 Lx.
> 
> The mellanox driver supports the corresponding mode. I think it should be
> "1000base-SGMII":
> 
> 
> mcx5: flags=8843 mtu 1500
> lladdr a0:88:c2:33:d1:b7
> index 8 priority 0 llprio 3
> media: Ethernet autoselect
> status: no carrier
> supported media:
> media 1000base-SGMII
> media 1000base-KX
> media 10GbaseKR
> media 10GSFP+Cu
> media 10GbaseSR
> media 10GbaseLR
> media 25GbaseCR
> media 25GbaseKR
> media 25GbaseSR
> media autoselect
> 
> 
> The SFP is recognized correctly:
> 
> 
> fw# ifconfig mcx5 transceiver
> mcx5: flags=8843 mtu 1500
> lladdr a0:88:c2:33:d1:b7
> index 8 priority 0 llprio 3
> media: Ethernet autoselect
> status: no carrier
> transceiver: SFP LC, 850 nm, 270m OM1, 550m OM2
> model: FLEXOPTIX S.8512.02.D rev A
> serial: F7AM3CB, date: 2023-06-13
> voltage: 3.30 V, bias current: 9.70 mA
> temp: 40.60 C (low -10.00 C, high 85.00 C)
> tx: -6.30 dBm (low -12.00 dBm, high -1.00 dBm)
> rx: -8.40 dBm (low -20.00 dBm, high 1.00 dBm)
> 
> 
> However, the status remains in state "no carrier". It is interesting to note
> that the interface can obviously receive network packets:

There are some extra media status bits that the driver wasn't checking,
so it didn't see that the link was up, which in turn stops it from sending
packets.

This diff adds the missing media status bits, which should get it working
for you.

diff --git sys/dev/pci/if_mcx.c sys/dev/pci/if_mcx.c
index 9108423827d..6b156ed3210 100644
--- sys/dev/pci/if_mcx.c
+++ sys/dev/pci/if_mcx.c
@@ -199,6 +199,19 @@ CTASSERT(MCX_MAX_QUEUES * MCX_WQ_DOORBELL_STRIDE <
 #define MCX_ETHER_CAP_50G_CR2  30
 #define MCX_ETHER_CAP_50G_KR2  31
 
+#define MCX_ETHER_EXT_CAP_SGMII_1000
+#define MCX_ETHER_EXT_CAP_1000_X   1
+#define MCX_ETHER_EXT_CAP_5G_R 3
+#define MCX_ETHER_EXT_CAP_XAUI 4
+#define MCX_ETHER_EXT_CAP_XLAUI5
+#define MCX_ETHER_EXT_CAP_25G_AUI1 6
+#define MCX_ETHER_EXT_CAP_50G_AUI2 7
+#define MCX_ETHER_EXT_CAP_50G_AUI1 8
+#define MCX_ETHER_EXT_CAP_CAUI49
+#define MCX_ETHER_EXT_CAP_100G_AUI210
+#define MCX_ETHER_EXT_CAP_200G_AUI412
+#define MCX_ETHER_EXT_CAP_400G_AUI815
+
 #define MCX_MAX_CQE32
 
 #define MCX_CMD_QUERY_HCA_CAP  0x100
@@ -406,11 +419,14 @@ struct mcx_reg_ptys {
uint8_t rp_reserved2;
uint8_t rp_proto_mask;
 #define MCX_REG_PTYS_PROTO_MASK_ETH(1 << 2)
-   uint8_t rp_reserved3[8];
+   uint8_t rp_reserved3[4];
+   uint32_trp_ext_eth_proto_cap;
uint32_trp_eth_proto_cap;
-   uint8_t rp_reserved4[8];
+   uint8_t rp_reserved4[4];
+   uint32_trp_ext_eth_proto_admin;
uint32_trp_eth_proto_admin;
-   uint8_t rp_reserved5[8];
+   uint8_t rp_reserved5[4];
+   uint32_trp_ext_eth_proto_oper;
uint32_trp_eth_proto_oper;
uint8_t rp_reserved6[24];
 } __packed __aligned(4);
@@ -2691,6 +2707,21 @@ static const struct mcx_eth_proto_capability 
mcx_eth_cap_map[] = {
[MCX_ETHER_CAP_50G_KR2] = { IFM_50G_KR2,IF_Gbps(50) },
 };
 
+static const struct mcx_eth_proto_capability mcx_ext_eth_cap_map[] = {
+   [MCX_ETHER_EXT_CAP_SGMII_100]   = { IFM_100_FX, IF_Mbps(100) },
+   [MCX_ETHER_EXT_CAP_1000_X]  = { IFM_1000_SX,IF_Gbps(1) },
+   [MCX_ETHER_EXT_CAP_5G_R]= { IFM_5000_T, IF_Gbps(5) },
+   [MCX_ETHER_EXT_CAP_XAUI]= { IFM_10G_SFI,IF_Gbps(10) },
+   [MCX_ETHER_EXT_CAP_XLAUI]   = { IFM_40G_XLPPI,  IF_Gbps(40) },
+   [MCX_ETHER_EXT_CAP_25G_AUI1]= { 0 /*IFM_25G_AUI*/,  IF_Gbps(25) },
+   [MCX_ETHER_EXT_CAP_50G_AUI2]= { 0 /*IFM_50G_AUI*/,  IF_Gbps(50) },
+   [MCX_ETHER_EXT_CAP_50G_AUI1]= { 0 /*IFM_50G_AUI*/,  IF_Gbps(50) },
+   [MCX_ETHER_EXT_CAP_CAUI4]   = { 0 /*IFM_100G_AUI*/, IF_Gbps(100) },
+   [MCX_ETHER_EXT_CAP_100G_AUI2]   = { 0 /*IFM_100G_AUI*/, IF_Gbps(100) },
+   [MCX_ETHER_EXT_CAP_200G_AUI4]   = { 0 /*IFM_200G_AUI*/, IF_Gbps(200) },
+   [MCX_ETHER_EXT_CAP_400G_AUI8]   = { 0 /*IFM_400G_AUI*/, IF_Gbps(400) },
+};
+
 static int
 mcx_get_id(uint32_t val)
 {
@@ -7956,6 +7987,19 @@ mcx_media_add_types(struct mcx_softc *sc)
 
ifmedia

Dell Poweedge 750 Mellanox ConnectX-6 LX with 1G SFP SX

2024-03-14 Thread Joerg Streckfuss


Hi misc,

I am trying to connect a 1GE SFP of type 1G SFP SX (Flexoptix S.8512.02.D) to a 
Power Edge R750 with a Connect Mellanox ConnectX-6 Lx.


The mellanox driver supports the corresponding mode. I think it should be 
"1000base-SGMII":



mcx5: flags=8843 mtu 1500
lladdr a0:88:c2:33:d1:b7
index 8 priority 0 llprio 3
media: Ethernet autoselect
status: no carrier
supported media:
media 1000base-SGMII
media 1000base-KX
media 10GbaseKR
media 10GSFP+Cu
media 10GbaseSR
media 10GbaseLR
media 25GbaseCR
media 25GbaseKR
media 25GbaseSR
media autoselect


The SFP is recognized correctly:


fw# ifconfig mcx5 transceiver
mcx5: flags=8843 mtu 1500
lladdr a0:88:c2:33:d1:b7
index 8 priority 0 llprio 3
media: Ethernet autoselect
status: no carrier
transceiver: SFP LC, 850 nm, 270m OM1, 550m OM2
model: FLEXOPTIX S.8512.02.D rev A
serial: F7AM3CB, date: 2023-06-13
voltage: 3.30 V, bias current: 9.70 mA
temp: 40.60 C (low -10.00 C, high 85.00 C)
tx: -6.30 dBm (low -12.00 dBm, high -1.00 dBm)
rx: -8.40 dBm (low -20.00 dBm, high 1.00 dBm)


However, the status remains in state "no carrier". It is interesting to note 
that the interface can obviously receive network packets:



tcpdump: listening on mcx5, link-type EN10MB
Mar 14 09:04:16.675476 802.1Q vid 1004 pri 6 CARPv2-advertise 36: vhid=1 
advbase=1 advskew=0 demote=0 (DF) [tos 0x10]
Mar 14 09:04:16.675476 802.1Q vid 1003 pri 6 CARPv2-advertise 36: vhid=1 
advbase=1 advskew=0 demote=0 (DF) [tos 0x10]

...


These network packets are sent from another system. This means that things seem 
to work in the receiving direction.


In the same system there are mellanox connectX-5 nics. The 1GE-SFP works 
flawlessly with it.



Regards,

Joerg


--
Dipl.-Ing. (FH) Joerg Streckfuss M.Sc. (Senior IT-Specialist)

DFN-CERT Services GmbH, https://www.dfn-cert.de/, Phone  +49 40 808077-555
Sitz / Register: Hamburg, AG Hamburg, HRB 88805,  Ust-IdNr.:  DE 232129737
Nagelsweg 41, 20097 Hamburg, Germany. CEO: Dr. Klaus-Peter Kossakowski


smime.p7s
Description: S/MIME Cryptographic Signature