Re: Debugging PH state
On 2023-03-13 15:50, Christopher Faulet wrote: Le 3/6/23 à 08:50, Christian Ruppert a écrit : On 2023-03-03 16:37, Christopher Faulet wrote: Le 3/3/23 à 15:40, Christian Ruppert a écrit : So I can reproduce it. I captured the response, extracted the data which includes to entire header + payload. I put that into a file like foo.bin and I start a "nc" that just serves that one response on every request. It works fine, no 500, no PH. Even with the same request so I really don't get it. I really wonder if that's a HAProxy bug. Hi Christian, It is indeed possible. Could you share your configuration ? Is there any special in the response ? Hi Christopher, I can share the config off-list. I don't see anything special, neither in the request nor the response. Do you know where the 500/PH is being set in the source by chance? Perhaps I can add some additional debugging there. If it's a invalid response, I'd guess that Apache would already block that from the actual backend (Apache is in front of it). The Response has a small header, small body. XML. Too bad I cannot get that 500 on every request/response :( Hi Christian, Sorry for the delay, just back from vacation. So sure, you can send me your configuration off-list. If you have a simple reproducer, share it too. It will be easier to understand what happens. In the code, you can find rewrite errors by grepping on "failed_rewrites". It a frontend/backend/listener/server counter. A rewrite error may happen every time haproxy tries to alter the request or the response. Hi Christopher, no problem! I've been busy with other stuff as well. The customer decided to not dig into it any further, since it happens sometimes only and only in some specific cases. I can still try to get a copy of the relevant parts of the config anyway, perhaps also including one of those responses. But as I said, I simulated it via "nc" and replied with the same response that failed but anything was fine. So I don't have anything to trigger that one for sure :/ -- Regards, Christian Ruppert
Re: Debugging PH state
Le 3/6/23 à 08:50, Christian Ruppert a écrit : On 2023-03-03 16:37, Christopher Faulet wrote: Le 3/3/23 à 15:40, Christian Ruppert a écrit : So I can reproduce it. I captured the response, extracted the data which includes to entire header + payload. I put that into a file like foo.bin and I start a "nc" that just serves that one response on every request. It works fine, no 500, no PH. Even with the same request so I really don't get it. I really wonder if that's a HAProxy bug. Hi Christian, It is indeed possible. Could you share your configuration ? Is there any special in the response ? Hi Christopher, I can share the config off-list. I don't see anything special, neither in the request nor the response. Do you know where the 500/PH is being set in the source by chance? Perhaps I can add some additional debugging there. If it's a invalid response, I'd guess that Apache would already block that from the actual backend (Apache is in front of it). The Response has a small header, small body. XML. Too bad I cannot get that 500 on every request/response :( Hi Christian, Sorry for the delay, just back from vacation. So sure, you can send me your configuration off-list. If you have a simple reproducer, share it too. It will be easier to understand what happens. In the code, you can find rewrite errors by grepping on "failed_rewrites". It a frontend/backend/listener/server counter. A rewrite error may happen every time haproxy tries to alter the request or the response. -- Christopher Faulet
Re: Debugging PH state
On 2023-03-03 16:37, Christopher Faulet wrote: Le 3/3/23 à 15:40, Christian Ruppert a écrit : So I can reproduce it. I captured the response, extracted the data which includes to entire header + payload. I put that into a file like foo.bin and I start a "nc" that just serves that one response on every request. It works fine, no 500, no PH. Even with the same request so I really don't get it. I really wonder if that's a HAProxy bug. Hi Christian, It is indeed possible. Could you share your configuration ? Is there any special in the response ? Hi Christopher, I can share the config off-list. I don't see anything special, neither in the request nor the response. Do you know where the 500/PH is being set in the source by chance? Perhaps I can add some additional debugging there. If it's a invalid response, I'd guess that Apache would already block that from the actual backend (Apache is in front of it). The Response has a small header, small body. XML. Too bad I cannot get that 500 on every request/response :( -- Regards, Christian Ruppert
Re: Debugging PH state
Le 3/3/23 à 15:40, Christian Ruppert a écrit : So I can reproduce it. I captured the response, extracted the data which includes to entire header + payload. I put that into a file like foo.bin and I start a "nc" that just serves that one response on every request. It works fine, no 500, no PH. Even with the same request so I really don't get it. I really wonder if that's a HAProxy bug. Hi Christian, It is indeed possible. Could you share your configuration ? Is there any special in the response ? -- Christopher Faulet
Re: Debugging PH state
So I can reproduce it. I captured the response, extracted the data which includes to entire header + payload. I put that into a file like foo.bin and I start a "nc" that just serves that one response on every request. It works fine, no 500, no PH. Even with the same request so I really don't get it. I really wonder if that's a HAProxy bug. -- Regards, Christian Ruppert
Debugging PH state
Hi list,
we have a case where a response results into a 500 (PH):
Mar 1 14:44:05.000 n095211 haproxy[2427573]: 1.2.3.4:22917
[01/Mar/2023:14:44:05.108] genfrontend_somefoo_stage~
genbackend_49140-somefoo_stage/localhost 1/0/0/-1/533 500 20378 - - PH--
24/1/0/0/0 0/0
{stage.somedom|someUA|https://stage.somedom/customizing/index.xhtml?cid=4||https://stage.somedom||somcapture||id-ecPublicKey}
"POST /customizing/index.xhtml?cid=4 HTTP/1.1"
On the backend:
192.168.95.211 - - [01/Mar/2023:14:44:05 +0100] "POST
/customizing/index.xhtml?cid=4 HTTP/1.1" 200 27643
"https://stage.somedom/customizing/index.xhtml?cid=4; "someUA"
So PH is stated as following:
PH The proxy blocked the server's response, because it was
invalid,
incomplete, dangerous (cache control), or matched a security
filter.
In any case, an HTTP 502 error is sent to the client. One
possible
cause for this error is an invalid syntax in an HTTP header
name
containing unauthorized characters. It is also possible but
quite
rare, that the proxy blocked a chunked-encoding request from
the
client due to an invalid syntax, before the server responded.
In this
case, an HTTP 400 error is sent to the client and reported in
the
logs. Finally, it may be due to an HTTP header rewrite failure
on the
response. In this case, an HTTP 500 error is sent (see
"tune.maxrewrite" and "http-response strict-mode" for more
inforomation).
In this case, the only relevant part seems to be:
Finally, it may be due to an HTTP header rewrite failure on the
response. In this case, an HTTP 500 error is sent (see
"tune.maxrewrite" and "http-response strict-mode" for more
inforomation).
tune.maxrewrite, hm, ok, it's set to 4096 in this case already. Also the
response header is really small and even below 1000 bytes so that
shouldn't be the problem, shouldn't it?
The response body is also pretty small compared to others but also the
body shouldn't matter in this case all all, shouldn't it? It's gzipped,
coming from an Apache, compressed it's between ~17-30kb, uncompressed
it's around 70-80kb.
Disabling strict mode is no option because we don't want to ignore
actual errors when processing / rewriting stuff internally in HAProxy.
I captured some of those responses but I don't see anything wrong or
weird at all to be honest. Another problem is, we cannot reproduce it
specifically. It only occurs sometimes, in some cases.
Any ideas?
HAProxy version 2.7.2-7e295dd 2023/01/20 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2024.
Known bugs: http://www.haproxy.org/bugs/bugs-2.7.2.html
Running on: Linux 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21)
x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = cc
CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement
-Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2
-Wduplicated-cond -Wnull-dereference -fwrapv
-Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare
-Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers
-Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
-DMAX_SESS_STKCTR=12
OPTIONS = USE_PCRE= USE_PCRE_JIT= USE_PCRE2=1 USE_PCRE2_JIT=
USE_LIBCRYPT=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB= USE_SLZ=1 USE_NS=
USE_SYSTEMD=1 USE_PROMEX=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY
+CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE
+LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER
-NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2
-PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION
-QUIC +RT +SHM_OPEN +SLZ -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO
+THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256,
default=4).
Built with OpenSSL version : OpenSSL 1.1.1n 15 Mar 2022
Running on OpenSSL version : OpenSSL 1.1.1n 15 Mar 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Support for malloc_trim() is enabled.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.36 2020-12-04
PCRE2 library supports JIT : no (USE_PCRE2_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.2.1 20210110
Available
