Re: Possible problem with custom error pages -- backend server returns 503, haproxy logs 503, but the browser gets 403

[Jarno Huuskonen]

Hello,

On 8/22/22 17:37, Shawn Heisey wrote:
The same problem also happens with 2.6.4, built with the same options as 
the dev version.


HAProxy version 2.6.4 2022/08/22 - https://haproxy.org/

I have documentation for the problem details in another project's bug 
tracker:


https://issues.apache.org/jira/browse/SOLR-16327?focusedCommentId=17582990=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17582990 





Does this happen with only HTTP/3.0(quic) or also with http/1.1 and 
http/2.0 ?


Are you able to capture the response coming from solr where haproxy 
sends wrong error ?


Testing with (2.6.4)+curl and this config (http/2 / http/1.1 only):
...
frontend test
bind ipv4@127.0.0.1:8001 alpn h2,http/1.1 ssl crt somecrt.pem

errorfiles myerrors
http-response return status 404 default-errorfiles if { status 404 }
http-response return status 403 default-errorfiles if { status 403 }
http-response return status 500 default-errorfiles if { status 500 }
http-response return status 502 default-errorfiles if { status 502 }
http-response return status 503 default-errorfiles if { status 503 }
http-response return status 504 default-errorfiles if { status 504 }
default_backend test_be

backend test_be
server srv1 127.0.0.1:9000 id 1

listen responder
bind ipv4@127.0.0.1:9000
http-request deny deny_status 503

And I receive the correct error file.

-Jarno

--
Jarno Huuskonen

Boost Your Outreach

[Nicole Davis]

Hi,

This is a quick check if you are looking for an Email Campaign Tool?

 

I would like to introduce you to one of the most affordable and
comprehensive Email Campaign Tool (ReachEngine.io), which allow you to send
unlimited cold emails to all of your ideal prospects within just few clicks.

 

If you are interested to learn more about ReachEngine.io, Kindly let me know
your convenient time and date so that we can schedule the call accordingly
for a quick Demo.

 

Please let me know if there are any specific queries, I can assist you with.

Regards,

Nicole Davis

Marketing Manager|  ReachEngine.io

3080 Olcott St D205, Santa Clara, CA 95054, United States

 

To Opt-Out, please reply unsubscribe in the subject line.

Server state file: port doesn't change after config update

[Bren]

Hello,

We've been seeing another minor issue I've been meaning to ask about. We're 
using a server state file:

server-state-file /var/lib/haproxy/server_state

In my systemd config for haproxy I've added a couple lines to save the server 
state on reload/stop:

ExecReload=/usr/local/sbin/haproxy -Ws -f $CONFIG -c -q $EXTRAOPTS
ExecReload=/opt/bin/save_server_state.sh
ExecReload=/bin/kill -USR2 $MAINPID
ExecStop=/opt/bin/save_server_state.sh

The script simply runs:

echo "show servers state" | socat /var/run/haproxy/admin.sock - > \
  /var/lib/haproxy/server_state

I've noticed that when I change the port on a backend server and reload, 
haproxy does not update the port for that server. I have to shut down haproxy, 
delete the state file, then start it back up for it to update the port 
(changing the port and renaming the server, reloading, then renaming it back 
and reloading again works too).

When I change other parts of the config, for example, if I add "disabled" to a 
server line and reload, haproxy updates the status of the server. It seems like 
haproxy isn't looking at the port when deciding if it should update the config 
for a server when using a state file.

Is this expected behavior?

Bren

Lua: processing expired on timeout when using core.msleep

[Bren]

Greetings,

This is a minor issue I've been meaning to ask about. I have a fairly simple 
Lua script that simply runs core.msleep on certain requests for a random number 
of ms to slow them down. I've noticed this in the logs:

[ALERT]    (3650884) : Lua function 'delay_request': aborting Lua processing on 
expired timeout.

I've always been under the impression that a sleep shouldn't cause any 
timeouts. Both tune.lua.session-timeout and tune.lua.service-timeout says:

If the Lua does a sleep, the sleep is not taken in account.

Am I missing something?

Bren

Re: [*EXT*] [ANNOUNCE] haproxy-2.6.3

[Bren]

--- Original Message ---
On Monday, August 22nd, 2022 at 7:03 AM, William Lallemand 
 wrote:

> I'm going to issue a 2.6.4 today with the patch.

Just rolled out 2.6.4 and everything appears to be working as expected now. 
Thanks!

Possible problem with custom error pages -- backend server returns 503, haproxy logs 503, but the browser gets 403

[Shawn Heisey]

Here is the full haproxy -vv:

HAProxy version 2.7-dev4-16972e-5 2022/08/22 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.15.0-1017-aws #21~20.04.1-Ubuntu SMP Fri Aug 5 
11:44:14 UTC 2022 x86_64

Build options :
  TARGET  = linux-glibc
  CPU = native
  CC  = cc
  CFLAGS  = -O2 -march=native -g -Wall -Wextra -Wundef 
-Wdeclaration-after-statement -Wfatal-errors -Wtype-limits 
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond 
-Wnull-dereference -fwrapv -Wno-address-of-packed-member 
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered 
-Wno-missing-field-initializers -Wno-cast-function-type 
-Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 
USE_QUIC=1

  DEBUG   =

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT -PCRE2 
+PCRE2_JIT +POLL +THREAD -PTHREAD_EMULATION +BACKTRACE -STATIC_PCRE 
-STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H 
-ENGINE +GETADDRINFO +OPENSSL -LUA +ACCEPT4 -CLOSEFROM +ZLIB -SLZ 
+CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD 
-OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT +QUIC -PROMEX 
-MEMORY_PROFILING


Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, 
default=2).

Built with OpenSSL version : OpenSSL 3.0.5+quic 5 Jul 2022
Running on OpenSSL version : OpenSSL 3.0.5+quic 5 Jul 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND

Built with PCRE2 version : 10.34 2019-11-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.4.0

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' keyword)
   quic : mode=HTTP  side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
 h2 : mode=HTTP  side=FE|BE  mux=H2 flags=HTX|HOL_RISK|NO_UPG
   fcgi : mode=HTTP  side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
   : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
 h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
   : mode=TCP   side=FE|BE  mux=PASS  flags=
   none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
    [BWLIM] bwlim-in
    [BWLIM] bwlim-out
    [CACHE] cache
    [COMP] compression
    [FCGI] fcgi-app
    [SPOE] spoe
    [TRACE] trace


The same problem also happens with 2.6.4, built with the same options as 
the dev version.


HAProxy version 2.6.4 2022/08/22 - https://haproxy.org/

I have documentation for the problem details in another project's bug 
tracker:


https://issues.apache.org/jira/browse/SOLR-16327?focusedCommentId=17582990=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17582990

It appears so far as if haproxy is getting a 503 from the backend, 
logging a 503, but actually sending a 403.  Here is the config snippet 
when it works correctly:


A top-level config section:
http-errors myerrors
    errorfile 404 /etc/haproxy/errors/404.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/50x.http
    errorfile 503 /etc/haproxy/errors/50x.http
    errorfile 504 /etc/haproxy/errors/50x.http


In the frontend:
    errorfiles myerrors
    http-response return status 404 default-errorfiles if 
!real_errors { status 404 }
    http-response return status 403 default-errorfiles if 
!real_errors { status 403 }
    http-response return status 500 default-errorfiles if 
!real_errors { status 500 }
    http-response return status 502 default-errorfiles if 
!real_errors { status 502 }
    http-response return status 503 default-errorfiles if 
!real_errors { status 503 }
    http-response return status 504 default-errorfiles if 
!real_errors { status 504 }


Removing the "!real_errors" part and restarting haproxy is when the 
problem occurs.  I created and used the real_errors acl as a working 
bandaid for the issue -- turn off the custom error pages for the solr 
hostname.

[ANNOUNCE] haproxy-2.6.4

[William Lallemand]

Hi,

HAProxy 2.6.4 was released on 2022/08/22. It added 2 new commits
after version 2.6.3.

This version was quickly released after the 2.6.3 because it introduced a
regression which could cause an infinite loop.

When in mworker mode, and without any master CLI configured (-S), after forking
the workers, and re-executing itself, the master will try to iterate the
proxies list but this one is empty, causing a infinite loop. The master is left
in a unusable state.

With systemd, the process is killed since the READY=1 signal is never sent.

This problem does not occur with the default systemd service which provides the
-S argument.


Please find the usual URLs below :
   Site index   : http://www.haproxy.org/
   Documentation: http://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse: http://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : http://www.haproxy.org/download/2.6/src/
   Git repository   : http://git.haproxy.org/git/haproxy-2.6.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy-2.6.git
   Changelog: http://www.haproxy.org/download/2.6/src/CHANGELOG
   Pending bugs : http://www.haproxy.org/l/pending-bugs
   Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs
   Code reports : http://www.haproxy.org/l/code-reports
   Latest builds: http://www.haproxy.org/l/dev-packages


---
Complete changelog :
Emeric Brun (1):
  BUG/MAJOR: mworker: fix infinite loop on master with no proxies.

William Lallemand (1):
  BUG/MINOR: ssl/cli: error when the ca-file is empty

---

-- 
William Lallemand

Re: [*EXT*] [ANNOUNCE] haproxy-2.6.3

[William Lallemand]

On Sat, Aug 20, 2022 at 09:56:40PM +0200, Willy Tarreau wrote:
> On Sat, Aug 20, 2022 at 09:36:21PM +0200, Ionel GARDAIS wrote:
> > That was it :
> > - remove the EXTRAOPTS from /etc/default/haproxy
> > - stop the running process referencing -x /run/haproxy/admin.sock on the CLI
> > - upgrade
> > 
> > All is OK.
> > First processes do not list -x on the CLI and a reload spawn a process with 
> > -x sockpair@
> 
> Very interesting, thank you Ionel.
> 
> There is this fix in 2.6.3 which seems in part related to this:
> 
>   commit ae1e5a8c72ced4c9527f05bd5b14916d1b9d92c8
>   Author: William Lallemand 
>   Date:   Mon Jul 25 15:51:30 2022 +0200
> 
> BUG/MINOR: sockpair: wrong return value for fd_send_uxst()
> 
> The fd_send_uxst() function which is used to send a socket over the
> socketpair returns 1 upon error instead of -1, which means the error
> case of the sendmsg() is never catched correctly.
> 
> Must be backported as far as 1.9.
> 
> (cherry picked from commit f67e8fb92c795808f60b2406ae395ebc0ca180c5)
> Signed-off-by: Christopher Faulet 
> 
> However I'm confused by that part in the code because it seems to try
> to send FDs in the function that's supposed to retrieve them, so I'm
> missing something related to the context where this is used. But if
> the caller was not able to properly handle an error, it could possibly
> explain (at least in part) why it failed for you after the upgrade when
> trying to connect if there's an issue with this socket. I'll have to
> check with William on Monday to clarify all this and try to reproduce. 
> 
> Thanks,
> Willy

It looked like we found the culprit, the patch 3b68b60 ("BUG/MAJOR: log-forward:
Fix log-forward proxies not fully initialized") was not handling the
case where no proxies are in the list.

In master-worker mode, the process is exec() again without any
configuration in order to free the memory, when -S is used to create a
master CLI the list is not empty and everything is good... however
without the -S argument, the list is empty and the master will
indefinitely loop.

So the workaround for this bug is to keep the -S in EXTRAOPTS... which
is the default in haproxy.service. If you rewrote EXTRAOPTS in
/etc/default/haproxy, don't forgot to put it there.

Emeric provided a new patch to fix the problem,
https://github.com/haproxy/haproxy/commit/8032a276ce8007020366d18ebd7400ad5dedc4f4

I'm going to issue a 2.6.4 today with the patch.

-- 
William Lallemand

Re: [*EXT*] [ANNOUNCE] haproxy-2.6.3

[William Lallemand]

On Sat, Aug 20, 2022 at 08:50:25PM +, Bren wrote:
> --- Original Message ---
> On Saturday, August 20th, 2022 at 3:43 PM, Vincent Bernat  
> wrote:
> 
> > Do you have something here too?
> 
> Nope. In fact I just removed that from the build.
> 
> > This does not match the file shipped by HAProxy, but this may explain
> > why you also run into this bug.
> 
> What ships with the source is:
> 
> Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid" 
> "EXTRAOPTS=-S /run/haproxy-master.sock"
> 
> I'm using the config for this:
> 
> stats socket /run/haproxy/admin.sock user haproxy group haproxy mode 660 
> level admin
> 
> So I probably removed that last part.

Hello Bren,

Do you have any error into your logs at startup? Could you share them?

Thanks.

-- 
William Lallemand

Re: haproxy 2.6.2 warnings while installation

[Willy Tarreau]

Hi Amol,

On Mon, Aug 22, 2022 at 11:42:01AM +0530, Amol Arote wrote:
> We are using haproxy 2.6.2 on Centos 7.9 ,
> 
> Following are warning seen during
> 
> *make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1*
> 
> [image: image.png]
> 
> src/http_fetch.c:356:6: warning: âhtxâ may be used uninitialized in this
> function [-Wmaybe-uninitialized]
>sl = http_get_stline(htx);
>   ^
> src/http_fetch.c: At top level:
> cc1: warning: unrecognized command line option "-Wno-atomic-alignment"
> [enabled by default]
> cc1: warning: unrecognized command line option "-Wno-string-plus-int"
> [enabled by default]
> cc1: warning: unrecognized command line option "-Wno-cast-function-type"
> [enabled by default]
> cc1: warning: unrecognized command line option
> "-Wno-address-of-packed-member" [enabled by default]
> 
> 
> 
> Please let us know if any concern

These are harmless and already fixed in 2.6.3.

Regards,
Willy

haproxy 2.6.2 warnings while installation

[Amol Arote]

We are using haproxy 2.6.2 on Centos 7.9 ,

Following are warning seen during

*make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1*

[image: image.png]

src/http_fetch.c:356:6: warning: âhtxâ may be used uninitialized in this
function [-Wmaybe-uninitialized]
   sl = http_get_stline(htx);
  ^
src/http_fetch.c: At top level:
cc1: warning: unrecognized command line option "-Wno-atomic-alignment"
[enabled by default]
cc1: warning: unrecognized command line option "-Wno-string-plus-int"
[enabled by default]
cc1: warning: unrecognized command line option "-Wno-cast-function-type"
[enabled by default]
cc1: warning: unrecognized command line option
"-Wno-address-of-packed-member" [enabled by default]



Please let us know if any concern

Regards,



Amol Arote

Senior IT Manager



*Mobile*: 9773868585 | 8097988585

*Phone:*  (022) 61934700 Ext 444

*Email:* amol.ar...@naaptol.com

*Web:* *https://www.naaptol.com *

--