Hi, After my other thread about performance issues on OpenBSD we decided to switch OSes on our HAProxy boxes to FreeBSD 13.1. In the test environment everything worked perfectly with transparent proxying but when cutting production over to FreeBSD I ran into an issue and had to revert for now.
With "source 0.0.0.0 usesrc client" everything seemed OK for about a minute but then many client connections across all backends started getting 503 errors (or connection failures for TCP backends). The haproxy logs have a lot of: - Cannot bind to tproxy source address before connect() for backend my_backend - Connect() failed for backend my_backend: local address already in use. This was during a late night maintenance with less than 100 connections, netstat had less than 500 total connections, and I didn't see any limits that were close to being hit (these are all auto-detected except nbthread=36 to leave some breathing room on this 40 core HW): pid = 76927 (process #1, nbproc = 1, nbthread = 36) uptime = 0d 0h14m15s system limits: memmax = unlimited; ulimit-n = 940059 maxsock = 940059; maxconn = 469926; maxpipes = 0 current conns = 40; current pipes = 0/0; conn rate = 8/sec; bit rate = 1.086 Mbps Running tasks: 0/248; idle = 100 % Health checks were always OK and the 503s/failures were intermittent with a majority but not all requests failing. With "clientip" instead of "client" it seemed to be worse with almost all requests failing. Looking back at the test env there were some occurrences of the "Connect() failed ... address already in use" in logs but there were no noticeable request failures. There were zero "Cannot bind to tproxy source address before connect()" in test even with much higher stress testing load. Configs between test and production are nearly identical except there are more backends and VLANs in production and prod traffic comes from more IPs than the 2-3 IPs that hit test. On OpenBSD transparent works out of the box with no PF divert rules needed (only standard pass rules). Unfortunately with FreeBSD's PF (which is behind OpenBSD on features but ahead on performance) I have not found any combination of rules that allow transparent to work at all (and divert-reply isn't implemented), so here ipfw is diverting and passing all traffic stateless like so: $cmd 0001 fwd localhost tcp from 10.10.15.0/24 80,443 to any in recv vlan123 $cmd 65534 allow all from any to any Then PF handles the main stateful firewall rules and outbound NAT for internal servers. All PF rules have logging and there was nothing being blocked by PF during the issue. It isn't ideal to run both ipfw and PF like this but it would be a major undertaking to rework the tooling we have built around PF to a different firewall, and since transparent is required it seems ipfw is needed to handle the divert part on FreeBSD. Am I overlooking something in my setup or might this be an OS or haproxy issue to debug further? Thanks in advance for any thoughts! $ haproxy -vv HAProxy version 2.6.9-3a3700a 2023/02/14 - https://haproxy.org/ Status: long-term supported branch - will stop receiving fixes around Q2 2027. Known bugs: http://www.haproxy.org/bugs/bugs-2.6.9.html Running on: FreeBSD 13.1-RELEASE-p6 FreeBSD 13.1-RELEASE-p6 GENERIC amd64 Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -O2 -pipe -fstack-protector-strong -fno-strict-aliasing -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wnull-dereference -fwrapv -Wno-unknown-warning-option -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment -DFREEBSD_PORTS OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ACCEPT4=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_PROMEX=1 DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS Feature list : -51DEGREES +ACCEPT4 -BACKTRACE +CLOSEFROM +CPU_AFFINITY -CRYPT_H -DEVICEATLAS -DL -ENGINE -EPOLL -EVPORTS +GETADDRINFO +KQUEUE +LIBCRYPT -LINUX_SPLICE -LINUX_TPROXY +LUA -MEMORY_PROFILING -NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL -PRCTL +PROCCTL +PROMEX -QUIC -RT -SLZ -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD -TFO +THREAD -THREAD_DUMP +TPROXY -WURFL +ZLIB Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_THREADS=64, default=40). Built with OpenSSL version : OpenSSL 1.1.1o-freebsd 3 May 2022 Running on OpenSSL version : OpenSSL 1.1.1o-freebsd 3 May 2022 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 Built with Lua version : Lua 5.3.6 Built with the Prometheus exporter as a service Support for malloc_trim() is enabled. Built with zlib version : 1.2.12 Running on zlib version : 1.2.12 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY Built with PCRE2 version : 10.40 2022-04-14 PCRE2 library supports JIT : yes Encrypted password support via crypt(3): yes Built with clang compiler version 13.0.0 (g...@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303) Available polling systems : kqueue : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use kqueue. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG <default> : mode=HTTP side=FE|BE mux=H1 flags=HTX none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG <default> : mode=TCP side=FE|BE mux=PASS flags= Available services : prometheus-exporter Available filters : [CACHE] cache [COMP] compression [FCGI] fcgi-app [SPOE] spoe [TRACE] trace $
