After my other thread about performance issues on OpenBSD we decided to
switch OSes on our HAProxy boxes to FreeBSD 13.1. In the test
environment everything worked perfectly with transparent proxying but
when cutting production over to FreeBSD I ran into an issue and had to 
revert for now.

With "source usesrc client" everything seemed OK for about a
minute but then many client connections across all backends started
getting 503 errors (or connection failures for TCP backends). The
haproxy logs have a lot of:

- Cannot bind to tproxy source address before connect() for backend my_backend
- Connect() failed for backend my_backend: local address already in use.

This was during a late night maintenance with less than 100 connections,
netstat had less than 500 total connections, and I didn't see any limits 
that were close to being hit (these are all auto-detected except 
nbthread=36 to leave some breathing room on this 40 core HW):

pid = 76927 (process #1, nbproc = 1, nbthread = 36)
uptime = 0d 0h14m15s
system limits: memmax = unlimited; ulimit-n = 940059
maxsock = 940059; maxconn = 469926; maxpipes = 0
current conns = 40; current pipes = 0/0; conn rate = 8/sec; bit rate = 1.086 
Running tasks: 0/248; idle = 100 %

Health checks were always OK and the 503s/failures were intermittent
with a majority but not all requests failing. With "clientip" instead of
"client" it seemed to be worse with almost all requests failing.

Looking back at the test env there were some occurrences of the
"Connect() failed ... address already in use" in logs but there were no
noticeable request failures. There were zero "Cannot bind to tproxy
source address before connect()" in test even with much higher stress 
testing load. Configs between test and production are nearly identical
except there are more backends and VLANs in production and prod traffic 
comes from more IPs than the 2-3 IPs that hit test.

On OpenBSD transparent works out of the box with no PF divert rules
needed (only standard pass rules). Unfortunately with FreeBSD's PF
(which is behind OpenBSD on features but ahead on performance) I have
not found any combination of rules that allow transparent to work at
all (and divert-reply isn't implemented), so here ipfw is diverting and 
passing all traffic stateless like so:

$cmd 0001 fwd localhost tcp from 80,443 to any in recv vlan123
$cmd 65534 allow all from any to any

Then PF handles the main stateful firewall rules and outbound NAT for
internal servers. All PF rules have logging and there was nothing being
blocked by PF during the issue. 

It isn't ideal to run both ipfw and PF like this but it would be a
major undertaking to rework the tooling we have built around PF to a
different firewall, and since transparent is required it seems ipfw is
needed to handle the divert part on FreeBSD.

Am I overlooking something in my setup or might this be an OS or haproxy
issue to debug further? Thanks in advance for any thoughts!

$ haproxy -vv
HAProxy version 2.6.9-3a3700a 2023/02/14 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.9.html
Running on: FreeBSD 13.1-RELEASE-p6 FreeBSD 13.1-RELEASE-p6 GENERIC amd64
Build options :
  TARGET  = freebsd
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -pipe -fstack-protector-strong -fno-strict-aliasing -Wall 
-Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits 
-Wshift-negative-value -Wnull-dereference -fwrapv -Wno-unknown-warning-option 
-Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare 
-Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers 
-Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment 


Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=40).
Built with OpenSSL version : OpenSSL 1.1.1o-freebsd  3 May 2022
Running on OpenSSL version : OpenSSL 1.1.1o-freebsd  3 May 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.6
Built with the Prometheus exporter as a service
Support for malloc_trim() is enabled.
Built with zlib version : 1.2.12
Running on zlib version : 1.2.12
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with PCRE2 version : 10.40 2022-04-14
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 13.0.0 (g...@github.com:llvm/llvm-project.git 

Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=

Available services : prometheus-exporter
Available filters :
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Reply via email to