Re: Dell Poweedge 750 Mellanox ConnectX-6 LX with 1G SFP SX
On Thu, Mar 14, 2024 at 10:13:52AM +0100, Joerg Streckfuss wrote:
>
> Hi misc,
>
> I am trying to connect a 1GE SFP of type 1G SFP SX (Flexoptix S.8512.02.D)
> to a Power Edge R750 with a Connect Mellanox ConnectX-6 Lx.
>
> The mellanox driver supports the corresponding mode. I think it should be
> "1000base-SGMII":
>
>
> mcx5: flags=8843 mtu 1500
> lladdr a0:88:c2:33:d1:b7
> index 8 priority 0 llprio 3
> media: Ethernet autoselect
> status: no carrier
> supported media:
> media 1000base-SGMII
> media 1000base-KX
> media 10GbaseKR
> media 10GSFP+Cu
> media 10GbaseSR
> media 10GbaseLR
> media 25GbaseCR
> media 25GbaseKR
> media 25GbaseSR
> media autoselect
>
>
> The SFP is recognized correctly:
>
>
> fw# ifconfig mcx5 transceiver
> mcx5: flags=8843 mtu 1500
> lladdr a0:88:c2:33:d1:b7
> index 8 priority 0 llprio 3
> media: Ethernet autoselect
> status: no carrier
> transceiver: SFP LC, 850 nm, 270m OM1, 550m OM2
> model: FLEXOPTIX S.8512.02.D rev A
> serial: F7AM3CB, date: 2023-06-13
> voltage: 3.30 V, bias current: 9.70 mA
> temp: 40.60 C (low -10.00 C, high 85.00 C)
> tx: -6.30 dBm (low -12.00 dBm, high -1.00 dBm)
> rx: -8.40 dBm (low -20.00 dBm, high 1.00 dBm)
>
>
> However, the status remains in state "no carrier". It is interesting to note
> that the interface can obviously receive network packets:
There are some extra media status bits that the driver wasn't checking,
so it didn't see that the link was up, which in turn stops it from sending
packets.
This diff adds the missing media status bits, which should get it working
for you.
diff --git sys/dev/pci/if_mcx.c sys/dev/pci/if_mcx.c
index 9108423827d..6b156ed3210 100644
--- sys/dev/pci/if_mcx.c
+++ sys/dev/pci/if_mcx.c
@@ -199,6 +199,19 @@ CTASSERT(MCX_MAX_QUEUES * MCX_WQ_DOORBELL_STRIDE <
#define MCX_ETHER_CAP_50G_CR2 30
#define MCX_ETHER_CAP_50G_KR2 31
+#define MCX_ETHER_EXT_CAP_SGMII_1000
+#define MCX_ETHER_EXT_CAP_1000_X 1
+#define MCX_ETHER_EXT_CAP_5G_R 3
+#define MCX_ETHER_EXT_CAP_XAUI 4
+#define MCX_ETHER_EXT_CAP_XLAUI5
+#define MCX_ETHER_EXT_CAP_25G_AUI1 6
+#define MCX_ETHER_EXT_CAP_50G_AUI2 7
+#define MCX_ETHER_EXT_CAP_50G_AUI1 8
+#define MCX_ETHER_EXT_CAP_CAUI49
+#define MCX_ETHER_EXT_CAP_100G_AUI210
+#define MCX_ETHER_EXT_CAP_200G_AUI412
+#define MCX_ETHER_EXT_CAP_400G_AUI815
+
#define MCX_MAX_CQE32
#define MCX_CMD_QUERY_HCA_CAP 0x100
@@ -406,11 +419,14 @@ struct mcx_reg_ptys {
uint8_t rp_reserved2;
uint8_t rp_proto_mask;
#define MCX_REG_PTYS_PROTO_MASK_ETH(1 << 2)
- uint8_t rp_reserved3[8];
+ uint8_t rp_reserved3[4];
+ uint32_trp_ext_eth_proto_cap;
uint32_trp_eth_proto_cap;
- uint8_t rp_reserved4[8];
+ uint8_t rp_reserved4[4];
+ uint32_trp_ext_eth_proto_admin;
uint32_trp_eth_proto_admin;
- uint8_t rp_reserved5[8];
+ uint8_t rp_reserved5[4];
+ uint32_trp_ext_eth_proto_oper;
uint32_trp_eth_proto_oper;
uint8_t rp_reserved6[24];
} __packed __aligned(4);
@@ -2691,6 +2707,21 @@ static const struct mcx_eth_proto_capability
mcx_eth_cap_map[] = {
[MCX_ETHER_CAP_50G_KR2] = { IFM_50G_KR2,IF_Gbps(50) },
};
+static const struct mcx_eth_proto_capability mcx_ext_eth_cap_map[] = {
+ [MCX_ETHER_EXT_CAP_SGMII_100] = { IFM_100_FX, IF_Mbps(100) },
+ [MCX_ETHER_EXT_CAP_1000_X] = { IFM_1000_SX,IF_Gbps(1) },
+ [MCX_ETHER_EXT_CAP_5G_R]= { IFM_5000_T, IF_Gbps(5) },
+ [MCX_ETHER_EXT_CAP_XAUI]= { IFM_10G_SFI,IF_Gbps(10) },
+ [MCX_ETHER_EXT_CAP_XLAUI] = { IFM_40G_XLPPI, IF_Gbps(40) },
+ [MCX_ETHER_EXT_CAP_25G_AUI1]= { 0 /*IFM_25G_AUI*/, IF_Gbps(25) },
+ [MCX_ETHER_EXT_CAP_50G_AUI2]= { 0 /*IFM_50G_AUI*/, IF_Gbps(50) },
+ [MCX_ETHER_EXT_CAP_50G_AUI1]= { 0 /*IFM_50G_AUI*/, IF_Gbps(50) },
+ [MCX_ETHER_EXT_CAP_CAUI4] = { 0 /*IFM_100G_AUI*/, IF_Gbps(100) },
+ [MCX_ETHER_EXT_CAP_100G_AUI2] = { 0 /*IFM_100G_AUI*/, IF_Gbps(100) },
+ [MCX_ETHER_EXT_CAP_200G_AUI4] = { 0 /*IFM_200G_AUI*/, IF_Gbps(200) },
+ [MCX_ETHER_EXT_CAP_400G_AUI8] = { 0 /*IFM_400G_AUI*/, IF_Gbps(400) },
+};
+
static int
mcx_get_id(uint32_t val)
{
@@ -7956,6 +7987,19 @@ mcx_media_add_types(struct mcx_softc *sc)
Re: Confusion about hw.cpuspeed
On Thu, Mar 14, 2024 at 03:07:22PM +0100, Christer Solskogen via misc wrote: > I've got a hold of two iKOOLCORE R2 today and installed OpenBSD(latest > amd64 snapshot) on them, but I can't seem to wrap my head around if > it's running at full speed or not. > > hugs# sysctl hw > hw.machine=amd64 > hw.model=Intel(R) N95 > hw.ncpu=4 > hw.byteorder=1234 > hw.pagesize=4096 > hw.disknames=sd0:d93f521d3c55b907,sd1: > hw.diskcount=2 > hw.sensors.cpu0.temp0=44.00 degC > hw.sensors.cpu0.frequency0=27.00 Hz > hw.sensors.cpu1.frequency0=27.00 Hz > hw.sensors.cpu2.frequency0=27.00 Hz > hw.sensors.cpu3.frequency0=27.00 Hz > hw.sensors.acpitz0.temp0=27.80 degC (zone temperature) > hw.cpuspeed=1701 > hw.setperf=100 > hw.vendor=iKOOLCORE TECHNOLOGY > hw.product=iKOOLCORE R2 > hw.version=Default string > hw.serialno=VME50IKLGT8 > hw.uuid=20231018-4682-2636-0882-000392308688 > hw.physmem=8272797696 > hw.usermem=8272670720 > hw.ncpufound=4 > hw.allowpowerdown=1 > hw.perfpolicy=auto > hw.smt=0 > hw.ncpuonline=4 > hw.power=1 > hw.ucomnames= > > hw.cpuspeed seems to suggest that it's running on 1701MH, even if > apmd(-A) is running and/or if I have full load on the machine. While > hw.sensors.cpu*.frequency0 seems to suggest otherwise. That seems to The 1MHz higher is the turbo setting. When speedstep speeds are shown in dmesg it is the highest. The sensors use cpu_hz_update_sensor().
pf nat64 rule not matching
Hello,
I'm trying to get a basic OpenBSD NAT64 router setup. I'm following
along with these instructions:
- https://blog.obtusenet.com/dns64-nat64-on-openbsd/
My unbound instance looks like it's correctly configured and returning
correct IPv6 addresses, so that's good.
# dig ipv4.google.com +short
ipv4.l.google.com.
64:ff9b::8efa:bc0e
However, the pf rule using af-to does not appear to do anything and
I haven't been able to figure out why. When I try to ping6, I get 100%
packet loss.
I inspected packets through tcpdump (after adding "log" to everything
in pf.conf) and nothing seems to be getting blocked, though it also
appears the 64:ff9b::/96 address are not being translated either; I
think the packets are passing through pf unchanged (the rule doesn't
apply, but I don't know why).
Here is my entire pf.conf:
wan = "igc0"
trusted = "igc1"
untrusted = "igc2"
iot = "igc3"
cerberus_ssh = "36285"
table persist file "/etc/martians"
set block-policy drop
set loginterface egress
set skip on lo0
block in log quick from urpf-failed
block in log quick on egress from to any
block return out log quick on egress from any to
block return log all
pass
# allow IPv6 PD from ISP
pass in inet6 proto udp from fe80::/10 port dhcpv6-server to fe80::/10 port
dhcpv6-client no state
# allow ICMPv6 traffic (necessary for IPv6 to work)
pass inet6 proto icmp6 all
# perform nat64 (NOT WORKING)
pass in to 64:ff9b::/96 af-to inet from ($wan:0)
# allow outbound queries from local unbound and NTP
pass out inet6 proto { tcp, udp } from ::1 to port { domain, ntp }
# allow DNS & NTP queries from the iot network
pass in on $iot proto { tcp, udp } from $iot:network to port { domain, ntp }
# allow ssh, http, & https
pass inet6 proto tcp to port { ssh, http, https, $cerberus_ssh }
I have IP forwarding turned on:
# sysctl | grep forwarding
net.inet.ip.forwarding=1
net.inet.ip.mforwarding=0
net.inet6.ip6.forwarding=1
net.inet6.ip6.mforwarding=1
I have an IPv4 and IPv6 address for igc0 via autoconf.
Here's a rough sketch of my network topology:
+---+
| ISP modem |
+---+
|
|
igc0
+---+
| cerberus (OpenBSD router) |
+---+
igc1 igc2 igc3
| ||
| ||
... ... +-+
| vulpes (OpenBSD client) |
+-+
>From both vulpes and cerberus, ping6 ipv4.google.com hangs and never
returns.
I tried substituting ($wan:0) for my actual IPv4 address assigned to
igc0, but I got no change in behavior. I read in the man page that
:0 does not include aliases when used on an interface. When I print
the rules out using pfctl -vvsr, it gets expanded to (igc0:0:1),
which looks weird and I don't understand why. My understanding is
that it should be "... af-to inet from IPV4_ADDRESS_OF_WAN_IF", but
I don't know if (igc0:0:1) is the IPv4 address of igc0, and I can't
figure out how to verify if that's right... or even if that's
the problem in the first place and I'm chasing a red herring.
I feel like I'm missing something, but I can't see it. The Book of PF
doesn't have any information on NAT64 that I could see, and the man page
for pf.conf shows an example of what I'm already doing with no
additional instructions. I've found maybe 3 articles about NAT64 on
OpenBSD through searching, but none give me any more context or clues
beyond the one I mentioned earlier.
I'd appreciate any help I could get!
Evan
Here's my dmesg:
OpenBSD 7.4 (GENERIC.MP) #1397: Tue Oct 10 09:02:37 MDT 2023
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8332189696 (7946MB)
avail mem = 8059916288 (7686MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.5 @ 0x75d9f000 (122 entries)
bios0: vendor American Megatrends International, LLC. version "ALN4L102" date
11/08/2023
bios0: Default string Default string
efi0 at bios0: UEFI 2.8
efi0: American Megatrends rev 0x5001a
acpi0 at bios0: ACPI 6.4
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP FIDT SSDT SSDT SSDT SSDT SSDT HPET APIC MCFG SSDT UEFI
RTCT PSDS NHLT LPIT SSDT SSDT DBGP DBG2 SSDT DMAR FPDT SSDT SSDT SSDT SSDT TPM2
PHAT WSMT
acpi0: wakeup devices PEGP(S4) PEGP(S4) PEGP(S4) SIO1(S3) RP09(S4) PXSX(S4)
RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) PXSX(S4) RP13(S4) PXSX(S4)
RP14(S4) PXSX(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 1920 Hz
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) N100, 3392.18 MHz, 06-be-00, patch 0012
cpu0:
Re: Looking for a well supported wireless card
On Thu, 14 Mar 2024 11:41:22 +0100 Stefan Sperling wrote: > On Thu, Mar 14, 2024 at 12:01:40AM -0400, Stefan Moran via misc wrote: > > On Wed, 13 Mar 2024 20:58:12 +0100 > > Stefan Sperling wrote: > > > > > ... > > > > > > iwm should work just fine. > > > > I don't doubt it. I did some more research on my device (Intel AC > > 7260, should have put it in my original message but I forgot to), > > and it doesn't support MU-MIMO, which the router in the residence > > I'm staying was set to use. > > Interesting. This could be an interop issue, though in general the > AP should be backwards compatible with 11ac MIMO. > > There is a flag you can use to disable the use of MIMO in the driver: > > ifconfig iwm0 nwflag nomimo > > This flag is generally only required if the second antenna is absent > or damaged. Neither the driver nor firmware attempt to detect "dead" > antennas. Apparently, this is too difficult to do reliably. I notice the black antenna wire is very loose no matter how well I connect it, so it could very well be an antenna issue, I could try disconnecting it intentionally and see if my issues are the same to verify. > > > I got the admin to disable this on the router, and it > > seems to be working better. > > And the router is still using regular MIMO now? Yes. > > Plesae run this while associated: > > tcpdump -n -i iwm0 -y IEEE802_11_RADIO -v -s 4096 type mgt subtype > beacon > > And show the following fields: > > htcaps=<...> htop=<...> vhtcaps=<...> vhtop=<...> htcaps=<20/40MHz,LDPC,SGI@20MHz,SGI@40MHz,TXSTBC,RXSTBC 1 stream,A-MSDU 7935,A-MPDU max 65535,A-MPDU spacing 4.00us,RxMCS 0x> htop=<40MHz chan 149:153,htprot 20MHz,non-greenfield STA,non-HT STA,basic MCS set 0x> vhtcaps= vhtop=<80MHz chan,center chan 155,basic MCS set 0-7@1SS 0-7@2SS 0-7@3SS 0-7@4SS 0-7@5SS 0-7@6SS 0-7@7SS> > ...
Confusion about hw.cpuspeed
I've got a hold of two iKOOLCORE R2 today and installed OpenBSD(latest amd64 snapshot) on them, but I can't seem to wrap my head around if it's running at full speed or not. hugs# sysctl hw hw.machine=amd64 hw.model=Intel(R) N95 hw.ncpu=4 hw.byteorder=1234 hw.pagesize=4096 hw.disknames=sd0:d93f521d3c55b907,sd1: hw.diskcount=2 hw.sensors.cpu0.temp0=44.00 degC hw.sensors.cpu0.frequency0=27.00 Hz hw.sensors.cpu1.frequency0=27.00 Hz hw.sensors.cpu2.frequency0=27.00 Hz hw.sensors.cpu3.frequency0=27.00 Hz hw.sensors.acpitz0.temp0=27.80 degC (zone temperature) hw.cpuspeed=1701 hw.setperf=100 hw.vendor=iKOOLCORE TECHNOLOGY hw.product=iKOOLCORE R2 hw.version=Default string hw.serialno=VME50IKLGT8 hw.uuid=20231018-4682-2636-0882-000392308688 hw.physmem=8272797696 hw.usermem=8272670720 hw.ncpufound=4 hw.allowpowerdown=1 hw.perfpolicy=auto hw.smt=0 hw.ncpuonline=4 hw.power=1 hw.ucomnames= hw.cpuspeed seems to suggest that it's running on 1701MH, even if apmd(-A) is running and/or if I have full load on the machine. While hw.sensors.cpu*.frequency0 seems to suggest otherwise. That seems to be running at full speed if there's load on the machine or not. -- chs
Dell Poweedge 750 Mellanox ConnectX-6 LX with 1G SFP SX
Hi misc, I am trying to connect a 1GE SFP of type 1G SFP SX (Flexoptix S.8512.02.D) to a Power Edge R750 with a Connect Mellanox ConnectX-6 Lx. The mellanox driver supports the corresponding mode. I think it should be "1000base-SGMII": mcx5: flags=8843 mtu 1500 lladdr a0:88:c2:33:d1:b7 index 8 priority 0 llprio 3 media: Ethernet autoselect status: no carrier supported media: media 1000base-SGMII media 1000base-KX media 10GbaseKR media 10GSFP+Cu media 10GbaseSR media 10GbaseLR media 25GbaseCR media 25GbaseKR media 25GbaseSR media autoselect The SFP is recognized correctly: fw# ifconfig mcx5 transceiver mcx5: flags=8843 mtu 1500 lladdr a0:88:c2:33:d1:b7 index 8 priority 0 llprio 3 media: Ethernet autoselect status: no carrier transceiver: SFP LC, 850 nm, 270m OM1, 550m OM2 model: FLEXOPTIX S.8512.02.D rev A serial: F7AM3CB, date: 2023-06-13 voltage: 3.30 V, bias current: 9.70 mA temp: 40.60 C (low -10.00 C, high 85.00 C) tx: -6.30 dBm (low -12.00 dBm, high -1.00 dBm) rx: -8.40 dBm (low -20.00 dBm, high 1.00 dBm) However, the status remains in state "no carrier". It is interesting to note that the interface can obviously receive network packets: tcpdump: listening on mcx5, link-type EN10MB Mar 14 09:04:16.675476 802.1Q vid 1004 pri 6 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] Mar 14 09:04:16.675476 802.1Q vid 1003 pri 6 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] ... These network packets are sent from another system. This means that things seem to work in the receiving direction. In the same system there are mellanox connectX-5 nics. The 1GE-SFP works flawlessly with it. Regards, Joerg -- Dipl.-Ing. (FH) Joerg Streckfuss M.Sc. (Senior IT-Specialist) DFN-CERT Services GmbH, https://www.dfn-cert.de/, Phone +49 40 808077-555 Sitz / Register: Hamburg, AG Hamburg, HRB 88805, Ust-IdNr.: DE 232129737 Nagelsweg 41, 20097 Hamburg, Germany. CEO: Dr. Klaus-Peter Kossakowski smime.p7s Description: S/MIME Cryptographic Signature
