Hi List,
because i've been asked, yes, the ixl3.15 interface is also UP (even my
output in the last mail showed different, result of my desperate tries
to make it work.
One thing I didn't mention, this is an onboard ixl on a Intel S2600STB
Board, is has this wired config, were the first two interfaces are 10g
base-t and the other two are sfp+ ports. While looking around further I
noticed that only one of the sfp+ ports shows information on the sfp+
module correctly (both modules are genuine intel and have been cycled):
# ifconfig -v ixl2
ixl2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9184
options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOM
AP>
ether a4:bf:01:76:ef:9c
inet [...]
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
plugged: GBIC 100G AOC (Active Optical Cable (SC)
vendor: PN: SN: DATE: 20--
does not show the correct information (100G AOC, i'd wish ;) even thou
it has the same kind of sfp+ as ixl3 here:
# ifconfig -v ixl3
ixl3: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
ether a4:bf:01:76:ef:9d
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: Intel Corp PN: FTLX8571D3BCV-IT SN: MHT27ZB DATE:
2018-11-09
module temperature: 43.24 C voltage: 3.32 Volts
lane 1: RX power: 0.57 mW (-2.41 dBm) TX bias: 8.20 mA
this made me thinking; is this 'wired' base-t and sfp+ configuration
properly supported by the driver?
Any ideas from the list?
thanks & best
Daniel
On 1/23/24 16:03, Daniel wrote:
Hi List,
just recently I discovered a problem with the ixl(4) driver. Hopefully
someone here can help me. my setup is as follows:
Network ----- ixl3 interface ----- ixl3.15 vlan interface -----
bridge0 ----- vnet0.1 to jail
the problem now is that the jail can send data out (arp requests), i
do see the responses on the ixl3 interface of the host, but they never
make their way up to the ixl3.15 vlan interface (even though they are
tagged correctly). To rule out that my config or the network is the
cruel pit i did test the same setup with a cheap usb-ethernet adapter
and there everything works as expected. I'm on FreeBSD 13.2-RELEASE-p8
and I did test both, the in kernel driver and the driver from ports
intel-ixl-kmod-1.13.4_1.
Here is a bit of information on my environment:
# uname -a
FreeBSD mimir 13.2-RELEASE-p8 FreeBSD 13.2-RELEASE-p8 GENERIC amd64
# pciconf -lBbcevV pci0:25:0:3
ixl3@pci0:25:0:3: class=0x020000 rev=0x09 hdr=0x00 vendor=0x8086
device=0x37d3 subvendor=0x8086 subdevice=0x35d5
vendor = 'Intel Corporation'
device = 'Ethernet Connection X722 for 10GbE SFP+'
class = network
subclass = ethernet
bar [10] = type Prefetchable Memory, range 64, base 0xb0000000,
size 16777216, enabled
bar [1c] = type Prefetchable Memory, range 64, base 0xb5000000,
size 32768, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 129 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO
max read 512
link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
cap 03[e0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 9aef76ffff01bfa4
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI
disabled
0 VFs configured out of 32 supported
First VF RID Offset 0x006d, VF RID Stride 0x0001
VF Device ID 0x37cd
Page Sizes: 4096 (enabled), 8192, 65536, 262144,
1048576, 4194304
ecap 0017[1a0] = TPH Requester 1
ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation
Blocking unavailable
P2P Req Redirect unavailable, P2P Cmpl Redirect
unavailable
P2P Upstream Forwarding unavailable, P2P Egress
Control unavailable
P2P Direct Translated unavailable, Enhanced
Capability unavailable
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
VPD ident = 'Example VPD'
# ifconfig
[...]
ixl3: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500
options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
ether a4:bf:01:76:ef:9d
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ixl3.15: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500
options=4200001<RXCSUM,RXCSUM_IPV6,NOMAP>
ether a4:bf:01:76:ef:9d
groups: vlan
vlan: 15 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl3
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
mtu 1500
ether 58:9c:fc:10:dd:05
inet 192.168.55.20 netmask 0xffffff00 broadcast 192.168.55.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000
member: ixl3.15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 55
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
[...]
# cat /etc/rc.conf
[...]
ifconfig_ixl3="up"
vlans_ixl3="15"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm ixl3.15 up"
[...]
# dmesg | grep ixl
ixl0: <Intel(R) Ethernet Connection X722 for 10GBASE-T - 2.3.3-k> mem
0xb3000000-0xb3ffffff,0xb5018000-0xb501ffff at device 0.0 numa-domain
0 on pci6
ixl0: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0
ixl0: PF-ID[0]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, MDIO shared
ixl0: Using 1024 TX descriptors and 1024 RX descriptors
ixl0: Using 12 RX queues 12 TX queues
ixl0: Using MSI-X interrupts with 13 vectors
ixl0: Ethernet address: a4:bf:01:76:ef:9a
ixl0: Allocating 16 queues for PF LAN VSI; 12 queues active
ixl0: SR-IOV ready
ixl0: netmap queues/slots: TX 12/1024, RX 12/1024
ixl1: <Intel(R) Ethernet Connection X722 for 10GBASE-T - 2.3.3-k> mem
0xb2000000-0xb2ffffff,0xb5010000-0xb5017fff at device 0.1 numa-domain
0 on pci6
ixl1: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0
ixl1: PF-ID[1]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, MDIO shared
ixl1: Using 1024 TX descriptors and 1024 RX descriptors
ixl1: Using 12 RX queues 12 TX queues
ixl1: Using MSI-X interrupts with 13 vectors
ixl1: Ethernet address: a4:bf:01:76:ef:9b
ixl1: Allocating 16 queues for PF LAN VSI; 12 queues active
ixl1: SR-IOV ready
ixl1: netmap queues/slots: TX 12/1024, RX 12/1024
ixl2: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem
0xb1000000-0xb1ffffff,0xb5008000-0xb500ffff at device 0.2 numa-domain
0 on pci6
ixl2: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0
ixl2: PF-ID[2]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
ixl2: Using 1024 TX descriptors and 1024 RX descriptors
ixl2: Using 12 RX queues 12 TX queues
ixl2: Using MSI-X interrupts with 13 vectors
ixl2: Ethernet address: a4:bf:01:76:ef:9c
ixl2: Allocating 16 queues for PF LAN VSI; 12 queues active
ixl2: ixl_set_link: Error getting phy capabilities -7, aq error: 5
ixl2: SR-IOV ready
ixl2: netmap queues/slots: TX 12/1024, RX 12/1024
ixl3: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem
0xb0000000-0xb0ffffff,0xb5000000-0xb5007fff at device 0.3 numa-domain
0 on pci6
ixl3: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0
ixl3: PF-ID[3]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
ixl3: Using 1024 TX descriptors and 1024 RX descriptors
ixl3: Using 12 RX queues 12 TX queues
ixl3: Using MSI-X interrupts with 13 vectors
ixl3: Ethernet address: a4:bf:01:76:ef:9d
ixl3: Allocating 16 queues for PF LAN VSI; 12 queues active
ixl3: ixl_set_link: Error getting phy capabilities -7, aq error: 5
ixl3: SR-IOV ready
ixl3: netmap queues/slots: TX 12/1024, RX 12/1024
ixl2: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated
FEC: None, Autoneg: False, Flow Control: None
ixl2: link state changed to UP
ixl3: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated
FEC: None, Autoneg: False, Flow Control: None
ixl3: link state changed to UP
bridge0: can't disable some capabilities on ixl3.15: 0x400
ixl3: promiscuous mode enabled
ixl3.15: promiscuous mode enabled
from my packet traces:
# tcpdump -vvv -i ixl3
tcpdump: listening on ixl3, link-type EN10MB (Ethernet), capture size
262144 bytes
[...]
13:36:20.155843 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.55.1 tell 192.168.55.10, length 28
13:36:20.156285 ARP, Ethernet (len 6), IPv4 (len 4), Reply
192.168.55.1 is-at b8:27:eb:47:8f:43 (oui Unknown), length 46
13:36:21.169003 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.55.1 tell 192.168.55.10, length 28
13:36:21.169538 ARP, Ethernet (len 6), IPv4 (len 4), Reply
192.168.55.1 is-at b8:27:eb:47:8f:43 (oui Unknown), length 46
Here the answer can be see, its tagged with 802.1q tag 15
# tcpdump -vvv -i ixl3.15
tcpdump: listening on ixl3.15, link-type EN10MB (Ethernet), capture
size 262144 bytes
14:14:37.255429 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.55.1 tell 192.168.55.10, length 28
14:14:42.263475 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.55.1 tell 192.168.55.10, length 28
14:15:02.556311 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.55.1 tell 192.168.55.10, length 28
14:15:07.557644 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.55.1 tell 192.168.55.10, length 28
The answer cannot be seen on the VLAN interface ):
I hope the list can help me out here, as I am lost.
Thanks & best
Daniel
