Hi Short: sometimes Haproxy ignores "http-request"-rules when NBTHREAD is in use.
Conditions: 1. Haproxy 1.9.8 from source + Debian with 4.19 kernel 2. Large config with thousands backends + 15MB Lua 3. Significant (far from critical) load and extremely often reloads 4. Everything works OK with NBPROC When I change NBPROC to NBTHREAD: 1. Everything still works OK according to logs. I couldn't find any failed real request. 2. My local monitoring (on the same server) screams about problems up to few times each hour. This monitoring is simplest ever but a bit uncommon: - PHP-script with just empty CURL-request to http://127.0.0.1/ (each 1-2 seconds) - No domain or any other headers 3. For such monitoring requests (without domain), my Haproxy answers 403 as expected. I've tested different ways to provide 403: - Directly in frontend: http-request deny if !{ req.fhdr(host) -m found } - And similar with "LUA -> backend -> deny" Both ways are 100% OK with NBPROC and 99.999% with NBTHREAD ) 4. The problem. 1 of ~500 checks doesn't return 403. At first, seeing mostly 503, - I thought that buggy answers are generated by Haproxy, but they are not. Request logs for http-frontend show: - Correct 403-answers: with "PR" termination flags and "<NOSRV>" server - Buggy answers: without any termination flags and always with the same backend (id=1, probably it is default since I don't use "default_backend" option) * Backend contains multiple servers and the chosen server is always 127.0.0.1, but I think it is coincidence As far as I understand: - In some very exotic conditions at least "http-request"-rules in frontend are completely ignored -> request just goes to default backend without processing - Looks like the conditions might be related to localhost-requests or to lack of headers I've tried to disable TFO, activate debug-logs, match reload\exit timings with problem - no luck for more info at the moment. -- Wert
