On Tue Sep 16, 2025 at 03:18:28PM +0200, Jan Klemkow wrote:
> Hi Rafael,
>
> On Tue, Sep 16, 2025 at 03:01:33PM +0200, Rafael Sadowski wrote:
> > WireGuard shows severe performance degradation (95% bandwidth
> > loss) on Intel 10Gb interfaces compared to direct connections,
> > with significant packet loss patterns.
> >
> > Performance Comparison:
> >
> > ServerA (Chicago) - Intel 10Gb interface (ix0)
> > ServerB (Atlanta) - Intel 10Gb interface (ix3)
> >
> > - Direct connection (iperf): 66.8 Mbps
> > - WireGuard tunnel (iperf): 3.3 Mbps
> > - Performance loss: 95%
> >
> > The physical Intel interface (ix3) shows 149426 output failures:
> >
> > ix3 1500 <Link> f8:f2:1e:3c:9c:09 195418012 0 144748154
> > 149426 0
> >
> > suggesting hardware/driver level problems that worsen with
> > WireGuard traffic processing?
> >
> > Are there known compatibility issues between ix(4) driver and
> > WireGuard packet processing?
> >
> > Could the bridge configuration (veb0 + vport0) be contributing to
> > the packet loss patterns?
>
> Yes. You will lose that kind of performance over bridge(4) and veb(4)
> because, they don't use segmentation offloading nor parallel processing
> of packets.
>
> The wg(4) device may have a similar missing performance features.
>
> > Any guidance on debugging approaches or known workarounds would be
> > greatly appreciated. I'm happy to provide additional data.
>
> Could you provide a netstat -s stats, before and after you've done you
> measurement? So, we can see if there are also any error or drop counter
> involved.
>
Before iperf:
ip:
261476447 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (duplicates or out of space)
0 malformed fragments dropped
0 fragments dropped after timeout
0 packets reassembled ok
257441439 packets for this host
2298 packets for unknown/unsupported protocol
789407 packets forwarded
0 packets not forwardable
0 redirects sent
186666566 packets sent from this host
192 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 fragment floods
0 packets with ip length > max ip packet size
0 tunneling packets that can't find gif
0 datagrams with bad address in header
108844283 input datagrams software-checksummed
176706792 output datagrams software-checksummed
0 multicast packets which we don't join
270340630 route cache hit
130030816 route cache miss
0 packets received on wrong interface
0 input packets dropped due to no bufs, etc.
icmp:
8420 calls to icmp_error
0 errors not generated because old message was icmp
0 errors not generated because of rate limitation
Output packet histogram:
echo reply: 72
destination unreachable: 8420
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
0 echo requests to broadcast/multicast rejected
Input packet histogram:
echo reply: 6
destination unreachable: 1652
routing redirect: 56
echo: 72
time exceeded: 673
72 message responses generated
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
ipencap:
0 total input packets
0 total output packets
0 packets shorter than header shows
0 packets dropped due to policy
0 packets with possibly spoofed local addresses
0 packets were dropped due to full output queue
0 input bytes
0 output bytes
0 protocol family mismatches
0 attempts to use tunnel with unspecified endpoint(s)
tcp:
94948757 packets sent
28710544 data packets (16624602340 bytes)
102472 data packets (42596644 bytes) retransmitted
83 fast retransmitted packets
50664046 ack-only packets (65368737 delayed)
0 URG only packets
0 window probe packets
13931200 window update packets
1781795 control packets
99184392 packets software-checksummed
740063 output TSO packets software chopped
230 output TSO packets hardware processed
3606746 output TSO packets generated
0 output TSO packets dropped
135967341 packets received
30001001 acks (for 16562209986 bytes)
3055906 duplicate acks
0 acks for unsent data
0 acks for old data
108194698 packets (102313362560 bytes) received in-sequence
864795 completely duplicate packets (1115421580 bytes)
3374 old duplicate packets
7558 packets with some duplicate data (4908373 bytes duplicated)
3209405 out-of-order packets (1208918804 bytes)
7 packets (0 bytes) of data after window
0 window probes
98237 window update packets
19064 packets received after close
27194 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded for missing IPsec protection
0 discarded due to memory shortage
97670476 packets software-checksummed
0 bad/missing md5 checksums
0 good md5 checksums
0 input LRO packets passed through pseudo device
0 input LRO generated packets from hardware
0 input LRO coalesced packets by network device
0 input bad LRO packets dropped
508590 connection requests
2822613 connection accepts
3330550 connections established (including accepts)
3337566 connections closed (including 117159 drops)
0 connections drained
9 embryonic connections dropped
30493863 segments updated rtt (of 21788757 attempts)
336765 retransmit timeouts
25453 connections dropped by rexmit timeout
0 persist timeouts
9352 keepalive timeouts
2969 keepalive probes sent
2 connections dropped by keepalive
595239 correct ACK header predictions
96201454 correct data packet header predictions
6274586 PCB cache misses
15555 dropped due to no socket
0 ECN connections accepted
0 ECE packets received
0 CWR packets received
0 CE packets received
0 ECT packets sent
0 ECE packets sent
0 CWR packets sent
cwr by fastrecovery: 8493
cwr by timeout: 336765
cwr by ecn: 0
23571 bad connection attempts
0 SYN packets dropped due to queue or memory full
3191317 SYN cache entries added
0 hash collisions
2822613 completed
0 aborted (no space to build PCB)
329303 timed out
0 dropped due to overflow
0 dropped due to bucket overflow
39399 dropped due to RST
0 dropped due to ICMP unreachable
1333994 SYN,ACKs retransmitted
46484 duplicate SYNs received for entries already in the cache
1864 SYNs dropped (no route or no space)
32 SYN cache seeds with new random
293 hash bucket array size in current SYN cache
0 entries in current SYN cache, limit is 10255
0 longest bucket length in current SYN cache, limit is 105
8683 uses of current SYN cache left
8410 SACK recovery episodes
11383 segment rexmits in SACK recovery episodes
15770037 byte rexmits in SACK recovery episodes
113995 SACK options received
834022 SACK options sent
83 SACK options dropped
udp:
121473937 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
0 with no checksum
9468707 input packets software-checksummed
76724393 output packets software-checksummed
8420 dropped due to no socket
0 broadcast/multicast datagrams dropped due to no socket
0 dropped due to missing IPsec protection
0 dropped due to full socket buffers
121465517 delivered
90072470 datagrams output
114216491 missed PCB cache
ipsec:
0 input IPsec packets
0 output IPsec packets
0 input bytes
0 output bytes
0 input bytes, decompressed
0 output bytes, uncompressed
0 packets dropped on input
0 packets dropped on output
0 packets that failed crypto processing
0 packets for which no XFORM was set in TDB received
0 packets for which no TDB was found
0 TDBs with hardlimit excess
esp:
0 input ESP packets
0 output ESP packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets with bad encryption received
0 packets that failed verification received
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 possibly replayed packets received
0 packets with bad payload size or padding received
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed crypto processing
0 output packets could not be sent
0 input UDP encapsulated ESP packets
0 output UDP encapsulated ESP packets
0 UDP packets for non-encapsulating TDB received
0 raw ESP packets for encapsulating TDB received
0 input bytes
0 output bytes
ah:
0 input AH packets
0 output AH packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets that failed verification received
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 possibly replayed packets received
0 packets with bad authenticator length received
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed crypto processing
0 output packets could not be sent
0 input bytes
0 output bytes
etherip:
0 packets shorter than header shows
0 packets were dropped due to full output queue
0 packets were dropped because of no interface/bridge information
0 packets dropped due to policy
0 packets dropped for other reasons
0 input ethernet-in-IP packets
0 output ethernet-in-IP packets
0 input bytes
0 output bytes
ipcomp:
0 input IPCOMP packets
0 output IPCOMP packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed (de)compression processing
0 output packets could not be sent
0 packets less than minimum compression length
0 input bytes
0 output bytes
carp:
0 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter than header
0 discarded for bad checksums
0 discarded packets with a bad version
0 discarded because packet too short
0 discarded for bad authentication
0 discarded for unknown vhid
0 discarded because of a bad address list
0 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 transitions to master
pfsync:
0 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for bad ttl
0 packets shorter than header
0 packets discarded for bad version
0 packets discarded for bad HMAC
0 packets discarded for bad action
0 packets discarded for short packet
0 states discarded for bad values
0 stale states
0 failed state lookup/inserts
0 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 send error
divert:
0 total packets received
0 dropped due to no socket
0 dropped due to full socket buffers
0 packets output
0 errors
pflow:
0 flows sent
0 packets sent
0 send failed due to mbuf memory error
0 send error
ip6:
1794 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (duplicates or out of space)
0 fragments dropped after timeout
0 fragments that exceeded limit
0 packets reassembled ok
0 packets for this host
0 packets forwarded
0 packets not forwardable
0 redirects sent
2 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 packets that violated scope rules
0 multicast packets which we don't join
Input packet histogram:
hop by hop: 108
ICMP6: 1686
Mbuf statistics:
0 one mbufs
0 one ext mbufs
0 two or more ext mbufs
0 tunneling packets that can't find gif
0 packets discarded due to too many headers
0 failures of source address selection
0 route cache hit
6356 route cache miss
0 packets received on wrong interface
0 input packets dropped due to no bufs, etc.
divert6:
0 total packets received
0 dropped due to no socket
0 dropped due to full socket buffers
0 packets output
0 errors
icmp6:
0 calls to icmp6_error
0 errors not generated because old message was icmp6 or so
0 errors not generated because of rate limitation
Output packet histogram:
multicast listener report: 2
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Histogram of error messages to be generated:
0 no route
0 administratively prohibited
0 beyond scope
0 address unreachable
0 port unreachable
0 packet too big
0 time exceed transit
0 time exceed reassembly
0 erroneous header field
0 unrecognized next header
0 unrecognized option
0 redirect
0 unknown
0 message responses generated
0 messages with too many ND options
0 messages with bad ND options
0 bad neighbor solicitation messages
0 bad neighbor advertisement messages
0 bad router solicitation messages
0 bad router advertisement messages
0 bad redirect messages
0 path MTU changes
rip6:
0 messages received
0 checksum calculations on inbound
0 messages with bad checksum
0 messages dropped due to no socket
0 multicast messages dropped due to no socket
0 messages dropped due to full socket buffers
0 delivered
0 datagrams output
After iperf via wg0
ip:
261480916 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (duplicates or out of space)
0 malformed fragments dropped
0 fragments dropped after timeout
0 packets reassembled ok
257445892 packets for this host
2298 packets for unknown/unsupported protocol
789407 packets forwarded
0 packets not forwardable
0 redirects sent
186670884 packets sent from this host
192 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 fragment floods
0 packets with ip length > max ip packet size
0 tunneling packets that can't find gif
0 datagrams with bad address in header
108845951 input datagrams software-checksummed
176712264 output datagrams software-checksummed
0 multicast packets which we don't join
270347510 route cache hit
130032853 route cache miss
0 packets received on wrong interface
0 input packets dropped due to no bufs, etc.
icmp:
8425 calls to icmp_error
0 errors not generated because old message was icmp
0 errors not generated because of rate limitation
Output packet histogram:
echo reply: 72
destination unreachable: 8425
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
0 echo requests to broadcast/multicast rejected
Input packet histogram:
echo reply: 6
destination unreachable: 1652
routing redirect: 56
echo: 72
time exceeded: 673
72 message responses generated
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
ipencap:
0 total input packets
0 total output packets
0 packets shorter than header shows
0 packets dropped due to policy
0 packets with possibly spoofed local addresses
0 packets were dropped due to full output queue
0 input bytes
0 output bytes
0 protocol family mismatches
0 attempts to use tunnel with unspecified endpoint(s)
tcp:
94950448 packets sent
28711966 data packets (16627717749 bytes)
102473 data packets (42596712 bytes) retransmitted
83 fast retransmitted packets
50664244 ack-only packets (65369286 delayed)
0 URG only packets
0 window probe packets
13931256 window update packets
1781810 control packets
99187484 packets software-checksummed
740745 output TSO packets software chopped
230 output TSO packets hardware processed
3608829 output TSO packets generated
0 output TSO packets dropped
135969868 packets received
30003080 acks (for 16565325460 bytes)
3055950 duplicate acks
0 acks for unsent data
0 acks for old data
108195340 packets (102313602925 bytes) received in-sequence
864795 completely duplicate packets (1115421580 bytes)
3374 old duplicate packets
7558 packets with some duplicate data (4908373 bytes duplicated)
3209452 out-of-order packets (1208918804 bytes)
7 packets (0 bytes) of data after window
0 window probes
98254 window update packets
19064 packets received after close
27194 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded for missing IPsec protection
0 discarded due to memory shortage
97672032 packets software-checksummed
0 bad/missing md5 checksums
0 good md5 checksums
0 input LRO packets passed through pseudo device
0 input LRO generated packets from hardware
0 input LRO coalesced packets by network device
0 input bad LRO packets dropped
508596 connection requests
2822664 connection accepts
3330607 connections established (including accepts)
3337621 connections closed (including 117160 drops)
0 connections drained
9 embryonic connections dropped
30495948 segments updated rtt (of 21789989 attempts)
336766 retransmit timeouts
25453 connections dropped by rexmit timeout
0 persist timeouts
9352 keepalive timeouts
2969 keepalive probes sent
2 connections dropped by keepalive
595277 correct ACK header predictions
96201746 correct data packet header predictions
6274689 PCB cache misses
15555 dropped due to no socket
0 ECN connections accepted
0 ECE packets received
0 CWR packets received
0 CE packets received
0 ECT packets sent
0 ECE packets sent
0 CWR packets sent
cwr by fastrecovery: 8493
cwr by timeout: 336766
cwr by ecn: 0
23571 bad connection attempts
0 SYN packets dropped due to queue or memory full
3191368 SYN cache entries added
0 hash collisions
2822664 completed
0 aborted (no space to build PCB)
329303 timed out
0 dropped due to overflow
0 dropped due to bucket overflow
39399 dropped due to RST
0 dropped due to ICMP unreachable
1333994 SYN,ACKs retransmitted
46484 duplicate SYNs received for entries already in the cache
1864 SYNs dropped (no route or no space)
32 SYN cache seeds with new random
293 hash bucket array size in current SYN cache
0 entries in current SYN cache, limit is 10255
0 longest bucket length in current SYN cache, limit is 105
8632 uses of current SYN cache left
8410 SACK recovery episodes
11383 segment rexmits in SACK recovery episodes
15770037 byte rexmits in SACK recovery episodes
113999 SACK options received
834022 SACK options sent
83 SACK options dropped
udp:
121475863 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
0 with no checksum
9468811 input packets software-checksummed
76726768 output packets software-checksummed
8425 dropped due to no socket
0 broadcast/multicast datagrams dropped due to no socket
0 dropped due to missing IPsec protection
0 dropped due to full socket buffers
121467438 delivered
90075091 datagrams output
114218296 missed PCB cache
ipsec:
0 input IPsec packets
0 output IPsec packets
0 input bytes
0 output bytes
0 input bytes, decompressed
0 output bytes, uncompressed
0 packets dropped on input
0 packets dropped on output
0 packets that failed crypto processing
0 packets for which no XFORM was set in TDB received
0 packets for which no TDB was found
0 TDBs with hardlimit excess
esp:
0 input ESP packets
0 output ESP packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets with bad encryption received
0 packets that failed verification received
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 possibly replayed packets received
0 packets with bad payload size or padding received
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed crypto processing
0 output packets could not be sent
0 input UDP encapsulated ESP packets
0 output UDP encapsulated ESP packets
0 UDP packets for non-encapsulating TDB received
0 raw ESP packets for encapsulating TDB received
0 input bytes
0 output bytes
ah:
0 input AH packets
0 output AH packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets that failed verification received
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 possibly replayed packets received
0 packets with bad authenticator length received
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed crypto processing
0 output packets could not be sent
0 input bytes
0 output bytes
etherip:
0 packets shorter than header shows
0 packets were dropped due to full output queue
0 packets were dropped because of no interface/bridge information
0 packets dropped due to policy
0 packets dropped for other reasons
0 input ethernet-in-IP packets
0 output ethernet-in-IP packets
0 input bytes
0 output bytes
ipcomp:
0 input IPCOMP packets
0 output IPCOMP packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed (de)compression processing
0 output packets could not be sent
0 packets less than minimum compression length
0 input bytes
0 output bytes
carp:
0 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter than header
0 discarded for bad checksums
0 discarded packets with a bad version
0 discarded because packet too short
0 discarded for bad authentication
0 discarded for unknown vhid
0 discarded because of a bad address list
0 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 transitions to master
pfsync:
0 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for bad ttl
0 packets shorter than header
0 packets discarded for bad version
0 packets discarded for bad HMAC
0 packets discarded for bad action
0 packets discarded for short packet
0 states discarded for bad values
0 stale states
0 failed state lookup/inserts
0 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 send error
divert:
0 total packets received
0 dropped due to no socket
0 dropped due to full socket buffers
0 packets output
0 errors
pflow:
0 flows sent
0 packets sent
0 send failed due to mbuf memory error
0 send error
ip6:
1794 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (duplicates or out of space)
0 fragments dropped after timeout
0 fragments that exceeded limit
0 packets reassembled ok
0 packets for this host
0 packets forwarded
0 packets not forwardable
0 redirects sent
2 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 packets that violated scope rules
0 multicast packets which we don't join
Input packet histogram:
hop by hop: 108
ICMP6: 1686
Mbuf statistics:
0 one mbufs
0 one ext mbufs
0 two or more ext mbufs
0 tunneling packets that can't find gif
0 packets discarded due to too many headers
0 failures of source address selection
0 route cache hit
6356 route cache miss
0 packets received on wrong interface
0 input packets dropped due to no bufs, etc.
divert6:
0 total packets received
0 dropped due to no socket
0 dropped due to full socket buffers
0 packets output
0 errors
icmp6:
0 calls to icmp6_error
0 errors not generated because old message was icmp6 or so
0 errors not generated because of rate limitation
Output packet histogram:
multicast listener report: 2
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Histogram of error messages to be generated:
0 no route
0 administratively prohibited
0 beyond scope
0 address unreachable
0 port unreachable
0 packet too big
0 time exceed transit
0 time exceed reassembly
0 erroneous header field
0 unrecognized next header
0 unrecognized option
0 redirect
0 unknown
0 message responses generated
0 messages with too many ND options
0 messages with bad ND options
0 bad neighbor solicitation messages
0 bad neighbor advertisement messages
0 bad router solicitation messages
0 bad router advertisement messages
0 bad redirect messages
0 path MTU changes
rip6:
0 messages received
0 checksum calculations on inbound
0 messages with bad checksum
0 messages dropped due to no socket
0 multicast messages dropped due to no socket
0 messages dropped due to full socket buffers
0 delivered
0 datagrams output
