Re: Issue with checks after 2.0.6
Hello, I had the same problem after upgrading from 2.0.5 to 2.0.6. I ignored the mistake and rolled back. I thought the mistake was mine. I use the self compiled versions only privately. The logs, config and build-script are in the attachment. HAProxy runs on a debian 9 VM cheers Michael Am 14.09.19 um 13:08 schrieb GARDAIS Ionel: > Hi, > > I've just upgraded to 2.0.6 and all server checks went erratic. > I had to disable checks for the servers to be reachable. > > The observed behavior was a flip-flap (but mostly down) of server > availability with L4TOUT when the server was considered unresponsive. > > Ionel > > > build-haproxy.sh Description: application/shellscript Sep 16 21:06:13 mail haproxy[21253]: Proxy http started. Sep 16 21:06:13 mail haproxy[21253]: Proxy bk_apache started. Sep 16 21:06:13 mail haproxy[21253]: [NOTICE] 258/210613 (21253) : New worker #1 (21255) forked Sep 16 21:06:13 mail haproxy[21253]: Proxy bk_gogs started. Sep 16 21:06:13 mail haproxy[21253]: Proxy bk_prosody started. Sep 16 21:06:13 mail haproxy[21253]: Proxy bk_smokeping started. Sep 16 21:06:13 mail haproxy[21253]: Proxy bk_odroid started. Sep 16 21:06:13 mail haproxy[21253]: Proxy bk_stats started. Sep 16 21:00:33 mail haproxy[19453]: [WARNING] 258/210033 (19453) : Exiting Master process... Sep 16 21:00:33 mail haproxy[19453]: [ALERT] 258/210033 (19453) : Current worker #1 (19454) exited with code 143 (Terminated) Sep 16 21:00:33 mail haproxy[19453]: [WARNING] 258/210033 (19453) : All workers exited. Exiting... (0) Sep 16 21:00:33 mail haproxy[20273]: Proxy http started. Sep 16 21:00:33 mail haproxy[20273]: Proxy bk_apache started. Sep 16 21:00:33 mail haproxy[20273]: [NOTICE] 258/210033 (20273) : New worker #1 (20274) forked Sep 16 21:00:33 mail haproxy[20273]: Proxy bk_gogs started. Sep 16 21:00:33 mail haproxy[20273]: Proxy bk_prosody started. Sep 16 21:00:33 mail haproxy[20273]: Proxy bk_smokeping started. Sep 16 21:00:33 mail haproxy[20273]: Proxy bk_odroid started. Sep 16 21:00:33 mail haproxy[20273]: Proxy bk_stats started. Sep 16 21:00:34 mail ansible-systemd: Invoked with no_block=False force=None name=haproxy daemon_reexec=False enabled=None daemon_reload=False state=reloaded masked=None scope=None user=None Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20273) : Reexecuting Master process Sep 16 21:00:34 mail haproxy[20273]: Proxy http started. Sep 16 21:00:34 mail haproxy[20273]: Proxy bk_apache started. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping frontend GLOBAL in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping frontend http in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: Proxy bk_gogs started. Sep 16 21:00:34 mail haproxy[20273]: [NOTICE] 258/210034 (20273) : New worker #1 (20303) forked Sep 16 21:00:34 mail haproxy[20273]: Proxy bk_prosody started. Sep 16 21:00:34 mail haproxy[20273]: [ALERT] 258/210034 (20274) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2) Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping backend bk_apache in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping backend bk_gogs in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping backend bk_prosody in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping backend bk_smokeping in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping backend bk_odroid in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Stopping backend bk_stats in 0 ms. Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy GLOBAL stopped (FE: 1 conns, BE: 1 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy http stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy bk_apache stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy bk_gogs stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy bk_prosody stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy bk_smokeping stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy bk_odroid stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20274) : Proxy bk_stats stopped (FE: 0 conns, BE: 0 conns). Sep 16 21:00:34 mail haproxy[20273]: Proxy bk_smokeping started. Sep 16 21:00:34 mail haproxy[20273]: Proxy bk_odroid started. Sep 16 21:00:34 mail haproxy[20273]: Proxy bk_stats started. Sep 16 21:00:34 mail haproxy[20273]: libgcc_s.so.1 must be installed for pthread_cancel to work Sep 16 21:00:34 mail haproxy[20273]: [WARNING] 258/210034 (20273) : Former worker #1 (20274) exited
How to wait some time before retry?
Hello!
I have a question about HAProxy configuration. Maybe someone has a solution
;)
I have a HAProxy (v2.0) load balancer in front of many web servers.
When I restart the web servers the TCP socket remains closed for a few
seconds (~10s). For this reason I would like to retry failed attempts to
connect after some seconds.
I already use option redispatch, however it seems that does not solve my
issue. The problem is that the request is retried immediately (after 1s),
thus causing all the retries to fail. From the HAProxy docs:
In order to avoid immediate reconnections to a server which is restarting,
a turn-around timer of min("timeout connect", one second) is applied before
a retry occurs.
Is there any option to wait some more time (e.g. 10s) before retrying? Or
do you have any other solution?
Re: Issue with checks after 2.0.6
Done : https://github.com/haproxy/haproxy/issues/278 -- Ionel GARDAIS Tech'Advantage CIO - IT Team manager - Mail original - De: "Lukas Tribus" À: "Ionel GARDAIS" , "Willy Tarreau" Cc: "haproxy" Envoyé: Lundi 16 Septembre 2019 11:20:00 Objet: Re: Issue with checks after 2.0.6 Hello! On Mon, Sep 16, 2019 at 8:50 AM GARDAIS Ionel wrote: > > Hi Lukas, > > Same with nbthread 1. > > I gave my first try to git bisect and it looks like the offending commit is : > > ab160a47acde9dc9c341b328c8716a721a389ab4 is the first bad commit > commit ab160a47acde9dc9c341b328c8716a721a389ab4 > Author: Willy Tarreau > Date: Thu Sep 5 17:38:40 2019 +0200 > > BUG/MINOR: checks: do not uselessly poll for reads before the connection > is up Thanks for this, could you file a github issue with those informations: https://github.com/haproxy/haproxy/issues/new/choose Lukas -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
PATCH: install golang-1.13 during travis-ci build as it is required for BoringSSL
please see attached patch
From 5fa12aec93f1e8989e06628fc5e41fc2556f532b Mon Sep 17 00:00:00 2001
From: Ilya Shipitsin
Date: Mon, 16 Sep 2019 16:13:10 +0500
Subject: [PATCH] BUILD: CI: install golang-1.13 when building BoringSSL
---
scripts/build-ssl.sh | 4
1 file changed, 4 insertions(+)
diff --git a/scripts/build-ssl.sh b/scripts/build-ssl.sh
index cec81e04a..384438a22 100755
--- a/scripts/build-ssl.sh
+++ b/scripts/build-ssl.sh
@@ -79,6 +79,10 @@ fi
if [ ! -z ${BORINGSSL+x} ]; then
(
+
+ # travis-ci comes with go-1.11, while boringssl requires go-1.13
+ eval "$(curl -sL https://raw.githubusercontent.com/travis-ci/gimme/master/gimme | GIMME_GO_VERSION=1.13 bash)"
+
download_boringssl
cd download-cache/boringssl
if [ -d build ]; then rm -rf build; fi
--
2.20.1
Re: Issue with checks after 2.0.6
Am 16.09.2019 um 12:21 schrieb Willy Tarreau: > Hi guys, > > On Mon, Sep 16, 2019 at 11:20:00AM +0200, Lukas Tribus wrote: >> Hello! >> >> On Mon, Sep 16, 2019 at 8:50 AM GARDAIS Ionel >> wrote: >>> >>> Hi Lukas, >>> >>> Same with nbthread 1. >>> >>> I gave my first try to git bisect and it looks like the offending commit is >>> : >>> >>> ab160a47acde9dc9c341b328c8716a721a389ab4 is the first bad commit >>> commit ab160a47acde9dc9c341b328c8716a721a389ab4 >>> Author: Willy Tarreau >>> Date: Thu Sep 5 17:38:40 2019 +0200 >>> >>> BUG/MINOR: checks: do not uselessly poll for reads before the >>> connection is up >> >> Thanks for this, could you file a github issue with those informations: > > Yes, please add it, I got the same report yesterday. It looks like it's > becoming urgent that we delete all the checks code and rewrite them from > scratch. We've reached a point where it seems impossible to make all of > them work at the same time, even with dirty hacks spread all over the > stack and causing trouble in other areas :-( In short, either we piss > off postfix users with aborted connections or we break other pure TCP > checks. And to be honest I don't even feel brave enough to try tcp-checks... Wow for me sounds like a huge task as the checks are one of the best features of haproxy, I fully understand your motivation behind that change. > Willy >
Re: Issue with checks after 2.0.6
Hi guys, On Mon, Sep 16, 2019 at 11:20:00AM +0200, Lukas Tribus wrote: > Hello! > > On Mon, Sep 16, 2019 at 8:50 AM GARDAIS Ionel > wrote: > > > > Hi Lukas, > > > > Same with nbthread 1. > > > > I gave my first try to git bisect and it looks like the offending commit is > > : > > > > ab160a47acde9dc9c341b328c8716a721a389ab4 is the first bad commit > > commit ab160a47acde9dc9c341b328c8716a721a389ab4 > > Author: Willy Tarreau > > Date: Thu Sep 5 17:38:40 2019 +0200 > > > > BUG/MINOR: checks: do not uselessly poll for reads before the > > connection is up > > Thanks for this, could you file a github issue with those informations: Yes, please add it, I got the same report yesterday. It looks like it's becoming urgent that we delete all the checks code and rewrite them from scratch. We've reached a point where it seems impossible to make all of them work at the same time, even with dirty hacks spread all over the stack and causing trouble in other areas :-( In short, either we piss off postfix users with aborted connections or we break other pure TCP checks. And to be honest I don't even feel brave enough to try tcp-checks... Willy
Re: haproxy=2.0.5: A bogus APPCTX is spinning and refuses to die
Created an issue: https://github.com/haproxy/haproxy/issues/277
пт, 6 сент. 2019 г. в 12:36, Максим Куприянов :
> Hi everybody!
>
> Any news on this issue? Maybe you need some more detailed info? I still
> getting these errors on instances with high request rates.
>
> чт, 29 авг. 2019 г. в 14:21, Максим Куприянов >:
>
>> Hi!
>>
>> Sometimes on reload of 2.0.5 I got this in logs:
>> A bogus APPCTX [0x7fc1a06ff0e0] is spinning at 122591 calls per second
>> and refuses to die, aborting now! Please report this error to developers
>> [strm=0x557eb7f4e630 src=xxx fe=yyy be=yyy dst= rqf=c48202 rqa=0
>> rpf=80048202 rpa=0 sif=EST,200040 sib=EST,244058 af=(nil),0
>> csf=0x7fc1ac085680,200 ab=0x7fc1a06ff0e0,7 csb=(nil),0
>> cof=0x557eb7fe9f50,201366:PASS(0x7fc16c0ad080)/RAW((nil))/tcpv6(1275)
>> cob=(nil),0:NONE((nil))/NONE((nil))/NONE(0) ]
>>
>> ...and this in coredump:
>> Program terminated with signal SIGABRT, Aborted.
>> #0 0x7fc1e7c0b428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> [Current thread is 1 (Thread 0x7fc1e95021c0 (LWP 3539))]
>> (gdb) bt
>> #0 0x7fc1e7c0b428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #1 0x7fc1e7c0d02a in abort () from /lib/x86_64-linux-gnu/libc.so.6
>> #2 0x557ea75b8b5e in stream_dump_and_crash
>> (obj=obj@entry=0x7fc1a06ff0e0,
>> rate=122591) at src/stream.c:2983
>> #3 0x557ea7696d2d in task_run_applet (t=0x7fc1746a5790,
>> context=0x7fc1a06ff0e0, state=) at src/applet.c:80
>> #4 0x557ea7692fd5 in process_runnable_tasks () at src/task.c:414
>> #5 0x557ea75fbc18 in run_poll_loop () at src/haproxy.c:2517
>> #6 run_thread_poll_loop (data=) at src/haproxy.c:2638
>> #7 0x557ea7558cb7 in main (argc=,
>> argv=0x7ffd7db57428) at src/haproxy.c:3315
>>
>> Version: haproxy -vv
>> HA-Proxy version 2.0.5-1 2019/08/27 - https://haproxy.org/
>> Build options :
>> TARGET = linux-glibc
>> CPU = generic
>> CC = gcc
>> CFLAGS = -O2 -g -O2 -fPIE -fstack-protector-strong -Wformat
>> -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
>> -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
>> -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
>> -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
>> -Wno-missing-field-initializers -Wtype-limits
>> OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_GETADDRINFO=1
>> USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_TFO=1 USE_SYSTEMD=1
>>
>> Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE
>> -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED
>> +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE
>> +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4
>> -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS
>> -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
>>
>> Default settings :
>> bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
>>
>> Built with multi-threading support (MAX_THREADS=64, default=56).
>> Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
>> Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
>> OpenSSL library supports TLS extensions : yes
>> OpenSSL library supports SNI : yes
>> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
>> Built with Lua version : Lua 5.3.1
>> Built with network namespace support.
>> Built with transparent proxy support using: IP_TRANSPARENT
>> IPV6_TRANSPARENT IP_FREEBIND
>> Built with zlib version : 1.2.8
>> Running on zlib version : 1.2.8
>> Compression algorithms supported : identity("identity"),
>> deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
>> Built with PCRE2 version : 10.21 2016-01-12
>> PCRE2 library supports JIT : yes
>> Encrypted password support via crypt(3): yes
>> Built with the Prometheus exporter as a service
>>
>> Available polling systems :
>> epoll : pref=300, test result OK
>>poll : pref=200, test result OK
>> select : pref=150, test result OK
>> Total: 3 (3 usable), will use epoll.
>>
>> Available multiplexer protocols :
>> (protocols marked as cannot be specified using 'proto' keyword)
>> h2 : mode=HTXside=FE|BE mux=H2
>> h2 : mode=HTTP side=FEmux=H2
>> : mode=HTXside=FE|BE mux=H1
>> : mode=TCP|HTTP side=FE|BE mux=PASS
>>
>> Available services :
>> prometheus-exporter
>>
>> Available filters :
>> [SPOE] spoe
>> [COMP] compression
>> [CACHE] cache
>> [TRACE] trace
>>
>> --
>> Best regards,
>> Maksim Kupriianov
>>
>
Re: Issue with checks after 2.0.6
Hello! On Mon, Sep 16, 2019 at 8:50 AM GARDAIS Ionel wrote: > > Hi Lukas, > > Same with nbthread 1. > > I gave my first try to git bisect and it looks like the offending commit is : > > ab160a47acde9dc9c341b328c8716a721a389ab4 is the first bad commit > commit ab160a47acde9dc9c341b328c8716a721a389ab4 > Author: Willy Tarreau > Date: Thu Sep 5 17:38:40 2019 +0200 > > BUG/MINOR: checks: do not uselessly poll for reads before the connection > is up Thanks for this, could you file a github issue with those informations: https://github.com/haproxy/haproxy/issues/new/choose Lukas
Re: Issue with checks after 2.0.6
Hi Lukas, Same with nbthread 1. I gave my first try to git bisect and it looks like the offending commit is : ab160a47acde9dc9c341b328c8716a721a389ab4 is the first bad commit commit ab160a47acde9dc9c341b328c8716a721a389ab4 Author: Willy Tarreau Date: Thu Sep 5 17:38:40 2019 +0200 BUG/MINOR: checks: do not uselessly poll for reads before the connection is up It's pointless to start to perform a recv() call on a connection that is not yet established. The only purpose used to be to subscribe but that causes many extra syscalls when we know we can do it later. This patch only attempts a read if the connection is established or if there is no write planed, since we want to be certain to be called. And in wake_srv_chk() we continue to attempt to read if the reader was not subscribed, so as to perform the first read attempt. In case a first result is provided, __event_srv_chk_r() will not do anything anyway so this is totally harmless in this case. This fix requires that commit "BUG/MINOR: checks: make __event_chk_srv_r() report success before closing" is applied before, otherwise it will break some checks (notably SSL) by doing them again after the connection is shut down. This completes the fixes on the checks described in issue #253 by roughly cutting the number of syscalls in half. It must be backported to 2.0. (cherry picked from commit c5940392255e5a5a7eb0d27be62e155f1aec26c6) Signed-off-by: Christopher Faulet :04 04 4cd93f8ab452b7092e56620c4a9f7672a3f9cd85 cc618d82eea0b8e421274410c61dc579a68cf7ce M src -- Ionel GARDAIS Tech'Advantage CIO - IT Team manager - Mail original - De: "Lukas Tribus" À: "Ionel GARDAIS" , "haproxy" Envoyé: Dimanche 15 Septembre 2019 20:37:09 Objet: Re: Issue with checks after 2.0.6 Hello, On Sat, Sep 14, 2019 at 4:58 PM GARDAIS Ionel wrote: > > What was the previous release that worked for you? 2.0.5 or something older? > > 2.0.5 worked well from the checks point of vue. Ok, so this is a regression in 2.0.6. Please try whether limiting the threads to 1 (global section: nbthread 1) changes something for you. Also I suggest you file a bug on github: https://github.com/haproxy/haproxy/issues/new/choose Lukas -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
