Here is the full haproxy -vv:
HAProxy version 2.7-dev4-16972e-5 2022/08/22 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.15.0-1017-aws #21~20.04.1-Ubuntu SMP Fri Aug 5
11:44:14 UTC 2022 x86_64
Build options :
TARGET = linux-glibc
CPU = native
CC = cc
CFLAGS = -O2 -march=native -g -Wall -Wextra -Wundef
-Wdeclaration-after-statement -Wfatal-errors -Wtype-limits
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond
-Wnull-dereference -fwrapv -Wno-address-of-packed-member
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
-Wno-missing-field-initializers -Wno-cast-function-type
-Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1
USE_QUIC=1
DEBUG =
Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT -PCRE2
+PCRE2_JIT +POLL +THREAD -PTHREAD_EMULATION +BACKTRACE -STATIC_PCRE
-STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H
-ENGINE +GETADDRINFO +OPENSSL -LUA +ACCEPT4 -CLOSEFROM +ZLIB -SLZ
+CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD
-OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT +QUIC -PROMEX
-MEMORY_PROFILING
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256,
default=2).
Built with OpenSSL version : OpenSSL 3.0.5+quic 5 Jul 2022
Running on OpenSSL version : OpenSSL 3.0.5+quic 5 Jul 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.34 2019-11-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.4.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
quic : mode=HTTP side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : none
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
The same problem also happens with 2.6.4, built with the same options as
the dev version.
HAProxy version 2.6.4 2022/08/22 - https://haproxy.org/
I have documentation for the problem details in another project's bug
tracker:
https://issues.apache.org/jira/browse/SOLR-16327?focusedCommentId=17582990&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17582990
It appears so far as if haproxy is getting a 503 from the backend,
logging a 503, but actually sending a 403. Here is the config snippet
when it works correctly:
A top-level config section:
http-errors myerrors
errorfile 404 /etc/haproxy/errors/404.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/50x.http
errorfile 503 /etc/haproxy/errors/50x.http
errorfile 504 /etc/haproxy/errors/50x.http
In the frontend:
errorfiles myerrors
http-response return status 404 default-errorfiles if
!real_errors { status 404 }
http-response return status 403 default-errorfiles if
!real_errors { status 403 }
http-response return status 500 default-errorfiles if
!real_errors { status 500 }
http-response return status 502 default-errorfiles if
!real_errors { status 502 }
http-response return status 503 default-errorfiles if
!real_errors { status 503 }
http-response return status 504 default-errorfiles if
!real_errors { status 504 }
Removing the "!real_errors" part and restarting haproxy is when the
problem occurs. I created and used the real_errors acl as a working
bandaid for the issue -- turn off the custom error pages for the solr
hostname.
