Hi,
Just doing various testing and came across an issue (maybe checksum) when a
6.8/-CURRENT guest uses vio(4) to communicate on a netmap/VALE backed
switch. ICMP traffic, even with large payloads, works fine, however, TCP
and UDP traffic isn't acknowledged by the destination host. There are no
issues for FreeBSD 12.2-RC3 or Ubuntu 20.10 guests when configured exactly
the same way. This is a FreeBSD 13.0-HEAD bhyve hypervisor host.
Between the two guests, a VALE switch is created and then each guest is
added to it:
FreeBSD:
valectl -n gst1
valectl -n gst2
valectl -a valesw:gst1
valectl -a valesw:gst2
and then added to a bhyve guest with something like: -s
5:1,virtio-net,valesw:gst1,mac=58:9c:fc:08:aa:bb
Here is OpenBSD 6.8 guest (192.168.200.1) running iperf3, listening for
connections - with a connection trying to be established by a FreeBSD
12.2-RC3 (192.168.200.3) guest (same result when an OpenBSD guest is used):
host1# tcpdump -nli vio1
tcpdump: listening on vio1, link-type EN10MB
11:22:21.555822 192.168.200.3.41039 > 192.168.200.1.5201: S
2549988210:2549988210(0) win 65535 <mss 1460,nop,wscale 6,sackOK,timestamp
209483701 0> (DF)
11:22:22.562704 192.168.200.3.41039 > 192.168.200.1.5201: S
2549988210:2549988210(0) win 65535 <mss 1460,nop,wscale 6,sackOK,timestamp
209484704 0> (DF)
11:22:24.762789 192.168.200.3.41039 > 192.168.200.1.5201: S
2549988210:2549988210(0) win 65535 <mss 1460,nop,wscale 6,sackOK,timestamp
209486908 0> (DF)
11:22:29.034203 192.168.200.3.41039 > 192.168.200.1.5201: S
2549988210:2549988210(0) win 65535 <mss 1460,nop,wscale 6,sackOK,timestamp
209491180 0> (DF)
11:22:37.241967 192.168.200.3.41039 > 192.168.200.1.5201: S
2549988210:2549988210(0) win 65535 <mss 1460,nop,wscale 6,sackOK,timestamp
209499383 0> (DF)
11:22:53.472129 192.168.200.3.41039 > 192.168.200.1.5201: S
2549988210:2549988210(0) win 65535 <mss 1460,nop,wscale 6,sackOK,timestamp
209515618 0> (DF)
11:23:25.682630 192.168.200.3.41039 > 192.168.200.1.5201: S
2549988210:2549988210(0) win 65535 <mss 1460,nop,wscale 6,sackOK,timestamp
209547829 0> (DF)
Here is netstat -s -p tcp for the above traffic on host 192.168.200.1:
7 discarded for bad checksums
Now to try some ICMP traffic from 192.168.200.3:
$ ping 192.168.200.1
PING 192.168.200.1 (192.168.200.1): 56 data bytes
64 bytes from 192.168.200.1: icmp_seq=0 ttl=255 time=1.191 ms
64 bytes from 192.168.200.1: icmp_seq=1 ttl=255 time=0.623 ms
64 bytes from 192.168.200.1: icmp_seq=2 ttl=255 time=0.644 ms
64 bytes from 192.168.200.1: icmp_seq=3 ttl=255 time=0.702 ms
^C
--- 192.168.200.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.623/0.790/1.191/0.233 ms
(This is what guest 192.168.200.1 sees and responds to)
11:39:13.165697 arp who-has 192.168.200.1 tell 192.168.200.3
11:39:13.165785 arp reply 192.168.200.1 is-at 58:9c:fc:0b:5a:e4
11:39:13.166173 192.168.200.3 > 192.168.200.1: icmp: echo request
11:39:13.166264 192.168.200.1 > 192.168.200.3: icmp: echo reply
11:39:14.213710 192.168.200.3 > 192.168.200.1: icmp: echo request
11:39:14.213759 192.168.200.1 > 192.168.200.3: icmp: echo reply
11:39:15.288066 192.168.200.3 > 192.168.200.1: icmp: echo request
11:39:15.288118 192.168.200.1 > 192.168.200.3: icmp: echo reply
11:39:16.313659 192.168.200.3 > 192.168.200.1: icmp: echo request
11:39:16.313782 192.168.200.1 > 192.168.200.3: icmp: echo reply
dmesg:
OpenBSD 6.8-current (GENERIC.MP) #137: Sun Oct 25 11:53:50 MDT 2020
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8572321792 (8175MB)
avail mem = 8297414656 (7913MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0xbf957000 (13 entries)
bios0: vendor BHYVE version "1.00" date 03/14/2014
bios0: bhyve BHYVE
acpi0 at bios0: ACPI 4.0
acpi0: sleep states S5
acpi0: tables DSDT FACP HPET APIC MCFG SPCR
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpihpet0 at acpi0: 16777216 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, 2095.63 MHz, 06-55-04
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,DCA,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,AVX512F,RDSEED,AVX512CD,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: CPU supports MTRRs but not enabled by BIOS
cpu0: apic clock running at 134MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, 2095.10 MHz, 06-55-04
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,DCA,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,AVX512F,RDSEED,AVX512CD,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 0, package 1
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, 2095.11 MHz, 06-55-04
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,DCA,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,AVX512F,RDSEED,AVX512CD,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 0, package 2
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, 2095.10 MHz, 06-55-04
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,DCA,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,AVX512F,RDSEED,AVX512CD,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 0, package 3
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 11, 32 pins, remapped
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PC00)
acpipci0 at acpi0 PC00
acpicmos0 at acpi0
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: bhyve
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 vendor "AMD", unknown product 0x7432 rev 0x00
ahci0 at pci0 dev 3 function 0 "Intel 82801H AHCI" rev 0x00: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
cd0 at scsibus1 targ 0 lun 0: <BHYVE, BHYVE DVD-ROM, 001> removable
nvme0 at pci0 dev 4 function 0 unknown vendor 0xfb5d product 0x0a0a rev
0x00: msix, NVMe 1.3
nvme0: bhyve-NVMe, firmware 1.0, serial NVME-4-0
scsibus2 at nvme0: 2 targets, initiator 0
sd0 at scsibus2 targ 1 lun 0: <NVMe, bhyve-NVMe, 1.0>
sd0: 32768MB, 512 bytes/sector, 67108864 sectors
virtio0 at pci0 dev 5 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 58:9c:fc:02:f7:e7
virtio0: msix shared
virtio1 at pci0 dev 5 function 1 "Qumranet Virtio Network" rev 0x00
vio1 at virtio1: address 58:9c:fc:0b:5a:e4
virtio1: msix shared
virtio2 at pci0 dev 6 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio2
virtio2: msix shared
unknown vendor 0xfb5d product 0x40fb (class display subclass VGA, rev 0x00)
at pci0 dev 7 function 0 not configured
xhci0 at pci0 dev 8 function 0 "Intel 7 Series xHCI" rev 0x00: msi, xHCI 0.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev
3.00/1.00 addr 1
pcib0 at pci0 dev 31 function 0 "Intel 82371SB ISA" rev 0x00
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
efifb0 at mainbus0: 800x600, 32bpp
wsdisplay0 at efifb0 mux 1: console (std, vt100 emulation), using wskbd0
wsdisplay0: screen 1-5 added (std, vt100 emulation)
uhidev0 at uhub0 port 1 configuration 1 interface 0 "BHYVE HID Tablet" rev
3.00/0.00 addr 2
uhidev0: iclass 3/1
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (15d8cfc7e534575a.a) swap on sd0b dump on sd0b
