On Jan 30, 2023, at 9:13 PM, Eugene Grosbein <[email protected]> wrote:
> 31.01.2023 4:17, Paul Mather wrote: > >> TL;DR: When working from home, I can max out my residential 200 Mbit network >> connection when downloading from remote Linux hosts at $JOB but only manage >> about 20% of my max residential connection speed when downloading from >> remote FreeBSD hosts at $JOB. When at $JOB, both FreeBSD and Linux hosts >> have no problem saturating their GbE connections transferring between each >> other. Why is this and how can I debug and fix it? >> >> I have a 200 Mbit residential cable connection (Xfinity, 200 Mbit down/~10 >> Mbit up). I've noticed recently that I can easily get 10--20 MB/s download >> speeds when transferring data from Linux hosts at work but when I try to >> download that same data from the FreeBSD hosts I use the speed usually tops >> out at 3--4 MB/s. These are Linux and FreeBSD hosts that are on the same >> subnet at work. Transfers from the FreeBSD hosts at work (within-subnet and >> within-site) are fine and match those of the Linux hosts---often 112 MB/s. >> So, it just appears to be the traffic over the WAN to my home that is >> affected. The WAN path from home to this subnet is typically 15 hops with a >> typical average ping latency of about 23 ms. >> >> The FreeBSD hosts are a mixture of -CURRENT, 13-STABLE, and 13.1-RELEASE. I >> had done some TCP tuning based upon the calomel.org <http://calomel.org/> >> tuning document (https://calomel.org/freebsd_network_tuning.html), but >> removed those tuning settings when I noticed the problem but the problem >> still persists. The only remaining customisation is that the 13-STABLE has >> "net.inet.tcp.cc.algorithm=cubic". (I notice that -CURRENT now has this as >> default so wanted to try that on 13-STABLE, too.) The FreeBSD systems are >> using either igb or em NICs. The Linux systems are using similar hardware. >> None has a problem maintaining local GbE transfer speeds---it's only the >> slower/longer WAN connections that have problems for the FreeBSD hosts. >> >> It seems that Linux hosts cope with the WAN path to my home better than the >> FreeBSD systems. Has anyone else noticed this? Does anyone have any idea >> as to what is obviously going wrong here and how I might debug/fix the >> FreeBSD hosts to yield faster speeds? My workaround at the moment is to >> favour using the remote Linux hosts for bulk data transfers. (I don't like >> this workaround.) >> >> Any help/insight is gratefully appreciated. > > I bet speedy traffic does not cross any NAT boxes but perhaps you employ NAT > at your own place. > Both pfnat and ipfw nat are not compatible with TSO, also sometimes RX/TX > checksum offload for NIC produce broken checksums, > and all that creates excessive retransmissions and timeouts greatly reducing > traffic speed. My IPv4 clients at home are behind a NAT router (OPNsense 23.1 with HTCP CC enabled and FQ_CoDel traffic shaper configured). The remote systems at $JOB are not behind NAT. On my OPNsense router I have disabled hardware offloading as (IIRC) these are generally not recommended for routers. As such, I have "Disable hardware checksum offload"; "Disable hardware TCP segmentation offload"; and "Disable hardware large receive offload" all checked in Interfaces -> Settings in OPNsense. The home router running OPNsense uses NICs that identify as "82583V Gigabit Network Connection" (emX@pci0:X:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x150c subvendor=0x8086 subdevice=0x0000). The FreeBSD systems at $JOB default to having TSO and RXCSUM/TXCSUM enabled. I disabled these but it didn't make any apparent difference in improving speeds. I looked at a Linux system on similar hardware and "ethtool -k" indicates it also has TSO and RX/TX checksum offloading enabled. > You may want to inspect traffic with Wireshark, as it shows retransmissions > and generally anomalies with colors, > or just go ahead and disable TSO and rxcsum/txcsum for external interface. I'm going to pursue this next. I made Wireshark captures at the client end but need to collect packet traces at the server side. Also, with the help of your suggestion, I managed to find a colour configuration to highlight retransmissions. Hopefully that might help differentiate the FreeBSD vs. Linux situation. I'm not great at grokking Wireshark traces but I guess I'm about to get better in the future. :-) Thanks for the suggestions. Cheers, Paul.
