Re: Issue with checks after 2.0.6
Hello, I had the same problem after upgrading from 2.0.5 to 2.0.6. I ignored the mistake and rolled back. I thought the mistake was mine. I use the self compiled versions only privately. The logs, config and build-script are in the attachment. HAProxy runs on a debian 9 VM cheers Michael Am 14.09.19 um 13:08 schrieb GARDAIS Ionel: > Hi, > > I've just upgraded to 2.0.6 and all server checks went erratic. > I had to disable checks for the servers to be reachable. > > The observed behavior was a flip-flap (but mostly down) of server > availability with L4TOUT when the server was considered unresponsive. > > Ionel > > > build-haproxy.sh Description: application/shellscript Sep 16 21:06:13 mail haproxy[21253]: Proxy http started. Sep 16 21:06:13 mail haproxy[21253]: Proxy bk_apache started. Sep 16 21:06:13 mail haproxy[21253]: [NOTICE] 258/210613 (21253) : New worker #1 (21255) forked Sep 16 21:06:13 mail haproxy[21253]: Proxy bk_gogs started. Sep 16 21:06:13 mail haproxy[21253]: Proxy bk_prosody started. Sep 16 21:06:13 mail haproxy[21253]: Proxy bk_smokeping started. Sep 16 21:06:13 mail haproxy[21253]: Proxy bk_odroid started. Sep 16 21:06:13 mail haproxy[21253]: Proxy bk_stats started. Sep 16 21:00:33 mail haproxy[19453]: [WARNING] 258/210033 (19453) : Exiting Master process... Sep 16 21:00:33 mail haproxy[19453]: [ALERT] 258/210033 (19453) : Current worker #1 (19454) exited with code 143 (Terminated) Sep 16 21:00:33 mail haproxy[19453]: [WARNING] 258/210033 (19453) : All workers exited. Exiting... (0) Sep 16 21:00:33 mail haproxy[20273]: Proxy http started. Sep 16 21:00:33 mail haproxy[20273]: Proxy bk_apache started. Sep 16 21:00:33 mail haproxy[20273]: [NOTICE] 258/210033 (20273) : New worker #1 (20274) forked Sep 16 21:00:33 mail haproxy[20273]: Proxy bk_gogs started. Sep 16 21:00:33 mail haproxy[20273]: Proxy bk_prosody started. Sep 16 21:00:33 mail haproxy[20273]: Proxy bk_smokeping started. Sep 16 21:00:33 mail haproxy[20273]: Proxy bk_odroid started. Sep 16 21:00:33 mail haproxy[20273]: Proxy bk_stats started. Sep 16 21:00:34 mail ansible-systemd: Invoked with no_block=False force=None name=haproxy daemon_reexec=False enabled=None daemon_reload=False state=reloaded masked=None scope=None user=None Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20273) : Reexecuting Master process Sep 16 21:00:34 mail haproxy[20273]: Proxy http started. Sep 16 21:00:34 mail haproxy[20273]: Proxy bk_apache started. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping frontend GLOBAL in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping frontend http in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: Proxy bk_gogs started. Sep 16 21:00:34 mail haproxy[20273]: [NOTICE] 258/210034 (20273) : New worker #1 (20303) forked Sep 16 21:00:34 mail haproxy[20273]: Proxy bk_prosody started. Sep 16 21:00:34 mail haproxy[20273]: [ALERT] 258/210034 (20274) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2) Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping backend bk_apache in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping backend bk_gogs in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping backend bk_prosody in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping backend bk_smokeping in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping backend bk_odroid in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping backend bk_stats in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy GLOBAL stopped (FE: 1 conns, BE: 1 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy http stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy bk_apache stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy bk_gogs stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy bk_prosody stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy bk_smokeping stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy bk_odroid stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy bk_stats stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: Proxy bk_smokeping started. Sep 16 21:00:34 mail haproxy[20273]: Proxy bk_odroid started. Sep 16 21:00:34 mail haproxy[20273]: Proxy bk_stats started. Sep 16 21:00:34 mail haproxy[20273]: libgcc_s.so.1 must be installed for pthread_cancel to work Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20273) : Former worker #1 (20274) exited
Re: Issue with checks after 2.0.6
Done : https://github.com/haproxy/haproxy/issues/278 -- Ionel GARDAIS Tech'Advantage CIO - IT Team manager - Mail original - De: "Lukas Tribus" À: "Ionel GARDAIS" , "Willy Tarreau" Cc: "haproxy" Envoyé: Lundi 16 Septembre 2019 11:20:00 Objet: Re: Issue with checks after 2.0.6 Hello! On Mon, Sep 16, 2019 at 8:50 AM GARDAIS Ionel wrote: > > Hi Lukas, > > Same with nbthread 1. > > I gave my first try to git bisect and it looks like the offending commit is : > > ab160a47acde9dc9c341b328c8716a721a389ab4 is the first bad commit > commit ab160a47acde9dc9c341b328c8716a721a389ab4 > Author: Willy Tarreau > Date: Thu Sep 5 17:38:40 2019 +0200 > > BUG/MINOR: checks: do not uselessly poll for reads before the connection > is up Thanks for this, could you file a github issue with those informations: https://github.com/haproxy/haproxy/issues/new/choose Lukas -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
Re: Issue with checks after 2.0.6
Am 16.09.2019 um 12:21 schrieb Willy Tarreau: > Hi guys, > > On Mon, Sep 16, 2019 at 11:20:00AM +0200, Lukas Tribus wrote: >> Hello! >> >> On Mon, Sep 16, 2019 at 8:50 AM GARDAIS Ionel >> wrote: >>> >>> Hi Lukas, >>> >>> Same with nbthread 1. >>> >>> I gave my first try to git bisect and it looks like the offending commit is >>> : >>> >>> ab160a47acde9dc9c341b328c8716a721a389ab4 is the first bad commit >>> commit ab160a47acde9dc9c341b328c8716a721a389ab4 >>> Author: Willy Tarreau >>> Date: Thu Sep 5 17:38:40 2019 +0200 >>> >>> BUG/MINOR: checks: do not uselessly poll for reads before the >>> connection is up >> >> Thanks for this, could you file a github issue with those informations: > > Yes, please add it, I got the same report yesterday. It looks like it's > becoming urgent that we delete all the checks code and rewrite them from > scratch. We've reached a point where it seems impossible to make all of > them work at the same time, even with dirty hacks spread all over the > stack and causing trouble in other areas :-( In short, either we piss > off postfix users with aborted connections or we break other pure TCP > checks. And to be honest I don't even feel brave enough to try tcp-checks... Wow for me sounds like a huge task as the checks are one of the best features of haproxy, I fully understand your motivation behind that change. > Willy >
Re: Issue with checks after 2.0.6
Hi guys, On Mon, Sep 16, 2019 at 11:20:00AM +0200, Lukas Tribus wrote: > Hello! > > On Mon, Sep 16, 2019 at 8:50 AM GARDAIS Ionel > wrote: > > > > Hi Lukas, > > > > Same with nbthread 1. > > > > I gave my first try to git bisect and it looks like the offending commit is > > : > > > > ab160a47acde9dc9c341b328c8716a721a389ab4 is the first bad commit > > commit ab160a47acde9dc9c341b328c8716a721a389ab4 > > Author: Willy Tarreau > > Date: Thu Sep 5 17:38:40 2019 +0200 > > > > BUG/MINOR: checks: do not uselessly poll for reads before the > > connection is up > > Thanks for this, could you file a github issue with those informations: Yes, please add it, I got the same report yesterday. It looks like it's becoming urgent that we delete all the checks code and rewrite them from scratch. We've reached a point where it seems impossible to make all of them work at the same time, even with dirty hacks spread all over the stack and causing trouble in other areas :-( In short, either we piss off postfix users with aborted connections or we break other pure TCP checks. And to be honest I don't even feel brave enough to try tcp-checks... Willy
Re: Issue with checks after 2.0.6
Hello! On Mon, Sep 16, 2019 at 8:50 AM GARDAIS Ionel wrote: > > Hi Lukas, > > Same with nbthread 1. > > I gave my first try to git bisect and it looks like the offending commit is : > > ab160a47acde9dc9c341b328c8716a721a389ab4 is the first bad commit > commit ab160a47acde9dc9c341b328c8716a721a389ab4 > Author: Willy Tarreau > Date: Thu Sep 5 17:38:40 2019 +0200 > > BUG/MINOR: checks: do not uselessly poll for reads before the connection > is up Thanks for this, could you file a github issue with those informations: https://github.com/haproxy/haproxy/issues/new/choose Lukas
Re: Issue with checks after 2.0.6
Hi Lukas, Same with nbthread 1. I gave my first try to git bisect and it looks like the offending commit is : ab160a47acde9dc9c341b328c8716a721a389ab4 is the first bad commit commit ab160a47acde9dc9c341b328c8716a721a389ab4 Author: Willy Tarreau Date: Thu Sep 5 17:38:40 2019 +0200 BUG/MINOR: checks: do not uselessly poll for reads before the connection is up It's pointless to start to perform a recv() call on a connection that is not yet established. The only purpose used to be to subscribe but that causes many extra syscalls when we know we can do it later. This patch only attempts a read if the connection is established or if there is no write planed, since we want to be certain to be called. And in wake_srv_chk() we continue to attempt to read if the reader was not subscribed, so as to perform the first read attempt. In case a first result is provided, __event_srv_chk_r() will not do anything anyway so this is totally harmless in this case. This fix requires that commit "BUG/MINOR: checks: make __event_chk_srv_r() report success before closing" is applied before, otherwise it will break some checks (notably SSL) by doing them again after the connection is shut down. This completes the fixes on the checks described in issue #253 by roughly cutting the number of syscalls in half. It must be backported to 2.0. (cherry picked from commit c5940392255e5a5a7eb0d27be62e155f1aec26c6) Signed-off-by: Christopher Faulet :04 04 4cd93f8ab452b7092e56620c4a9f7672a3f9cd85 cc618d82eea0b8e421274410c61dc579a68cf7ce M src -- Ionel GARDAIS Tech'Advantage CIO - IT Team manager - Mail original - De: "Lukas Tribus" À: "Ionel GARDAIS" , "haproxy" Envoyé: Dimanche 15 Septembre 2019 20:37:09 Objet: Re: Issue with checks after 2.0.6 Hello, On Sat, Sep 14, 2019 at 4:58 PM GARDAIS Ionel wrote: > > What was the previous release that worked for you? 2.0.5 or something older? > > 2.0.5 worked well from the checks point of vue. Ok, so this is a regression in 2.0.6. Please try whether limiting the threads to 1 (global section: nbthread 1) changes something for you. Also I suggest you file a bug on github: https://github.com/haproxy/haproxy/issues/new/choose Lukas -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
Re: Issue with checks after 2.0.6
Hello, On Sat, Sep 14, 2019 at 4:58 PM GARDAIS Ionel wrote: > > What was the previous release that worked for you? 2.0.5 or something older? > > 2.0.5 worked well from the checks point of vue. Ok, so this is a regression in 2.0.6. Please try whether limiting the threads to 1 (global section: nbthread 1) changes something for you. Also I suggest you file a bug on github: https://github.com/haproxy/haproxy/issues/new/choose Lukas
Re: Issue with checks after 2.0.6
Same.
I had to disable HTX because I had issues with some corrupted payloads.
I'll give a new try to HTX as 2.0.6 corrects issues with TLS.
--
Ionel GARDAIS
Tech'Advantage CIO - IT Team manager
- Mail original -
De: "Aleksandar Lazic"
À: "Ionel GARDAIS"
Cc: "haproxy"
Envoyé: Samedi 14 Septembre 2019 14:16:30
Objet: Re: Issue with checks after 2.0.6
When you enable htx do you have the same problems?
Comment in `no option http-use-htx`
Regards Aleks
Sat Sep 14 14:12:30 GMT+02:00 2019 GARDAIS Ionel
:
> Also, haproxy and servers are on the same subnet : no filtering nor routing
> between them.
> Ping as no troubles, servers are not overloaded by other connections.
>
> --
> Ionel GARDAIS
> Tech'Advantage CIO - IT Team manager
>
> - Mail original -
> De: "Ionel GARDAIS"
> À: "Aleksandar Lazic"
> Cc: "haproxy"
> Envoyé: Samedi 14 Septembre 2019 14:07:42
> Objet: Re: Issue with checks after 2.0.6
>
> Sure.
> Note : as soon as I remove the check from the server line then 'systemctl
> reload haproxy', access is OK.
>
> # haproxy -vv
> HA-Proxy version 2.0.6-1~bpo9+1 2019/09/14 - https://haproxy.org/
> Build options :
> TARGET = linux-glibc
> CPU = generic
> CC = gcc
> CFLAGS = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-2.0.6=.
> -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
> -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement
> -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
> -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
> -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value
> -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
> OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1
> USE_ZLIB=1 USE_SYSTEMD=1
>
> Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT
> +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM
> -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT
> +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4
> +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL
> +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
>
> Default settings :
> bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
>
> Built with multi-threading support (MAX_THREADS=64, default=2).
> Built with OpenSSL version : OpenSSL 1.1.0k 28 May 2019
> Running on OpenSSL version : OpenSSL 1.1.0k 28 May 2019
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
> Built with Lua version : Lua 5.3.3
> Built with network namespace support.
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
> IP_FREEBIND
> Built with zlib version : 1.2.8
> Running on zlib version : 1.2.8
> Compression algorithms supported : identity("identity"), deflate("deflate"),
> raw-deflate("deflate"), gzip("gzip")
> Built with PCRE2 version : 10.22 2016-07-29
> PCRE2 library supports JIT : yes
> Encrypted password support via crypt(3): yes
> Built with the Prometheus exporter as a service
>
> Available polling systems :
> epoll : pref=300, test result OK
>poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 3 (3 usable), will use epoll.
>
> Available multiplexer protocols :
> (protocols marked as cannot be specified using 'proto' keyword)
> h2 : mode=HTXside=FE|BE mux=H2
> h2 : mode=HTTP side=FEmux=H2
> : mode=HTXside=FE|BE mux=H1
> : mode=TCP|HTTP side=FE|BE mux=PASS
>
> Available services :
> prometheus-exporter
>
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [CACHE] cache
> [TRACE] trace
>
>
>
>
>
>
> # cat /etc/haproxy/haproxy.cfg
> global
> log /dev/loglocal0 info
> log /dev/loglocal1 notice
> chroot /var/lib/haproxy
> stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd
> listeners
> stats timeout 30s
> user haproxy
> group haproxy
> daemon
>
> # Default SSL material locations
> ca-base /etc/ssl/certs
> crt-base /etc/ssl/private
>
> # Default ciphers to use on SSL-enabled listening sockets.
> # For more information, see ciphers(1SSL). This list is from:
> # https://hynek.me/articles/hardening-your-web-servers-
Re: Issue with checks after 2.0.6
Hello, On Sat, Sep 14, 2019 at 1:08 PM GARDAIS Ionel wrote: > > Hi, > > I've just upgraded to 2.0.6 and all server checks went erratic. > I had to disable checks for the servers to be reachable. What was the previous release that worked for you? 2.0.5 or something older? Thanks, Lukas
Re: Issue with checks after 2.0.6
When you enable htx do you have the same problems?
Comment in `no option http-use-htx`
Regards Aleks
Sat Sep 14 14:12:30 GMT+02:00 2019 GARDAIS Ionel
:
> Also, haproxy and servers are on the same subnet : no filtering nor routing
> between them.
> Ping as no troubles, servers are not overloaded by other connections.
>
> --
> Ionel GARDAIS
> Tech'Advantage CIO - IT Team manager
>
> - Mail original -
> De: "Ionel GARDAIS"
> À: "Aleksandar Lazic"
> Cc: "haproxy"
> Envoyé: Samedi 14 Septembre 2019 14:07:42
> Objet: Re: Issue with checks after 2.0.6
>
> Sure.
> Note : as soon as I remove the check from the server line then 'systemctl
> reload haproxy', access is OK.
>
> # haproxy -vv
> HA-Proxy version 2.0.6-1~bpo9+1 2019/09/14 - https://haproxy.org/
> Build options :
> TARGET = linux-glibc
> CPU = generic
> CC = gcc
> CFLAGS = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-2.0.6=.
> -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
> -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement
> -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
> -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
> -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value
> -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
> OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1
> USE_ZLIB=1 USE_SYSTEMD=1
>
> Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT
> +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM
> -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT
> +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4
> +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL
> +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
>
> Default settings :
> bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
>
> Built with multi-threading support (MAX_THREADS=64, default=2).
> Built with OpenSSL version : OpenSSL 1.1.0k 28 May 2019
> Running on OpenSSL version : OpenSSL 1.1.0k 28 May 2019
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
> Built with Lua version : Lua 5.3.3
> Built with network namespace support.
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
> IP_FREEBIND
> Built with zlib version : 1.2.8
> Running on zlib version : 1.2.8
> Compression algorithms supported : identity("identity"), deflate("deflate"),
> raw-deflate("deflate"), gzip("gzip")
> Built with PCRE2 version : 10.22 2016-07-29
> PCRE2 library supports JIT : yes
> Encrypted password support via crypt(3): yes
> Built with the Prometheus exporter as a service
>
> Available polling systems :
> epoll : pref=300, test result OK
>poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 3 (3 usable), will use epoll.
>
> Available multiplexer protocols :
> (protocols marked as cannot be specified using 'proto' keyword)
> h2 : mode=HTXside=FE|BE mux=H2
> h2 : mode=HTTP side=FEmux=H2
> : mode=HTXside=FE|BE mux=H1
> : mode=TCP|HTTP side=FE|BE mux=PASS
>
> Available services :
> prometheus-exporter
>
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [CACHE] cache
> [TRACE] trace
>
>
>
>
>
>
> # cat /etc/haproxy/haproxy.cfg
> global
> log /dev/loglocal0 info
> log /dev/loglocal1 notice
> chroot /var/lib/haproxy
> stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd
> listeners
> stats timeout 30s
> user haproxy
> group haproxy
> daemon
>
> # Default SSL material locations
> ca-base /etc/ssl/certs
> crt-base /etc/ssl/private
>
> # Default ciphers to use on SSL-enabled listening sockets.
> # For more information, see ciphers(1SSL). This list is from:
> # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
> ssl-default-bind-ciphers
> EECDH+AES:+AES128:+AES256:+SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA
> ssl-default-bind-options no-sslv3
> tune.ssl.default-dh-param 2048
>
> defaults
> log global
> modehttp
> option httplog
> option dontlognull
> timeout connect 5000
> timeout client
Re: Issue with checks after 2.0.6
Also, haproxy and servers are on the same subnet : no filtering nor routing
between them.
Ping as no troubles, servers are not overloaded by other connections.
--
Ionel GARDAIS
Tech'Advantage CIO - IT Team manager
- Mail original -
De: "Ionel GARDAIS"
À: "Aleksandar Lazic"
Cc: "haproxy"
Envoyé: Samedi 14 Septembre 2019 14:07:42
Objet: Re: Issue with checks after 2.0.6
Sure.
Note : as soon as I remove the check from the server line then 'systemctl
reload haproxy', access is OK.
# haproxy -vv
HA-Proxy version 2.0.6-1~bpo9+1 2019/09/14 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-2.0.6=.
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1
USE_ZLIB=1 USE_SYSTEMD=1
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT
+PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM
-STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT
+CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB
-SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD
-OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.1.0k 28 May 2019
Running on OpenSSL version : OpenSSL 1.1.0k 28 May 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.22 2016-07-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
h2 : mode=HTXside=FE|BE mux=H2
h2 : mode=HTTP side=FEmux=H2
: mode=HTXside=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS
Available services :
prometheus-exporter
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
# cat /etc/haproxy/haproxy.cfg
global
log /dev/loglocal0 info
log /dev/loglocal1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd
listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers
EECDH+AES:+AES128:+AES256:+SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA
ssl-default-bind-options no-sslv3
tune.ssl.default-dh-param 2048
defaults
log global
modehttp
option httplog
option dontlognull
timeout connect 5000
timeout client 5
timeout server 5
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
option forwardfor except 127.0.0.1/8
option redispatch
option http-keep-alive
no option http-use-htx
frontend ssl
bind ${HAPROXY_VRRP}:443 ssl crt tad-2019-chain.crt
bind ${HAPROXY_IPV4}:443 ssl crt tad-2019-chain.crt
bind ${HAPROXY_IPV6}:443 ssl crt tad
Re: Issue with checks after 2.0.6
_cookie res.hdr(Set-Cookie),lower -m sub secure rspirep ^(Set-Cookie:.*) \1;\ Secure unless secured_cookie acl host-tools hdr(host) tools.example.com acl to-etap path_beg /etap use_backend bck-etap if host-tools to-etap backend bck-etap server etap 192.168.1.69:8080 check >From haproxy.log : Sep 14 13:57:35 haproxy-1 haproxy[9978]: Server bck-etap/etap is DOWN, reason: Layer4 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 0 sessions active, 0 dequeued, 0 remaining in queue. Sep 14 13:57:35 haproxy-1 haproxy[9976]: [WARNING] 256/135735 (9978) : Server bck-etap/etap is DOWN, reason: Layer4 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 0 sessions active, 0 dequeued, 0 remaining in queue. Sep 14 13:57:35 haproxy-1 haproxy[9978]: Server bck-etap/etap is DOWN, reason: Layer4 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 0 sessions active, 0 dequeued, 0 remaining in queue. Sep 14 13:57:35 haproxy-1 haproxy[9978]: backend bck-etap has no server available! Sep 14 13:57:35 haproxy-1 haproxy[9978]: backend bck-etap has no server available! Sep 14 13:57:35 haproxy-1 haproxy[9976]: [ALERT] 256/135735 (9978) : backend 'bck-etap' has no server available! Sep 14 13:58:16 haproxy-1 haproxy[9978]: 172.17.10.1:51523 [14/Sep/2019:13:58:16.024] ssl~ bck-etap/ 0/-1/-1/-1/0 503 213 - - SC-- 16/15/0/0/0 0/0 "GET /etap/ HTTP/1.1" ^C -- Ionel GARDAIS Tech'Advantage CIO - IT Team manager - Mail original - De: "Aleksandar Lazic" À: "Ionel GARDAIS" , "haproxy" Envoyé: Samedi 14 Septembre 2019 13:12:49 Objet: Re: Issue with checks after 2.0.6 Hi. Am 14.09.2019 um 13:08 schrieb GARDAIS Ionel: > Hi, > > I've just upgraded to 2.0.6 and all server checks went erratic. > I had to disable checks for the servers to be reachable. > > The observed behavior was a flip-flap (but mostly down) of server availability > with L4TOUT when the server was considered unresponsive. Please can you share some more informations like some configs and log lines. > Ionel Best regards Aleks -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
Re: Issue with checks after 2.0.6
Hi. Am 14.09.2019 um 13:08 schrieb GARDAIS Ionel: > Hi, > > I've just upgraded to 2.0.6 and all server checks went erratic. > I had to disable checks for the servers to be reachable. > > The observed behavior was a flip-flap (but mostly down) of server availability > with L4TOUT when the server was considered unresponsive. Please can you share some more informations like some configs and log lines. > Ionel Best regards Aleks
