April 30, 2019 - The scion of one of India’s biggest conglomerates was arrested for possessing drugs
PATCH: s6 readiness notification support
I have added working (for me at least) s6 notification support to
haproxy so it can signal that it's ready when running under the s6
supervision suite.
The process used under s6 is that the daemon signals readiness by
writing a line to an arbitrary FD (that the supervision suite has been
told about), then closing the FD as documented here:
https://skarnet.org/software/s6/notifywhenup.html
This signals the transition from started to ready.
At this stage the FD written to and closed is hard coded as fd@3 as I
have reached the end of my C expertise getting this far...
Although it seems like it's not something that many people are hanging
out for based on the responses to a previous email to the list, I
thought I'd send the patch through as-is anyway.
Regards,
Andrew Heberle
>From 9668e699b85bc5494d1a27e550e987ce43cb Mon Sep 17 00:00:00 2001
From: Andrew Heberle
Date: Wed, 1 May 2019 09:40:41 +0800
Subject: [PATCH] Add s6 notification support
---
include/types/global.h | 1 +
src/haproxy.c | 12
2 files changed, 13 insertions(+)
diff --git a/include/types/global.h b/include/types/global.h
index ba3738b..9481d64 100644
--- a/include/types/global.h
+++ b/include/types/global.h
@@ -72,6 +72,7 @@
#define GTUNE_BUSY_POLLING (1<<11)
#define GTUNE_LISTENER_MQ (1<<12)
#define GTUNE_SET_DUMPABLE (1<<13)
+#define GTUNE_USE_S6_NOTIFY (1<<14)
/* Access level for a stats socket */
#define ACCESS_LVL_NONE 0
diff --git a/src/haproxy.c b/src/haproxy.c
index 4c5c839..99a6f67 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -446,6 +446,7 @@ static void usage(char *name)
#if defined(USE_SYSTEMD)
" -Ws master-worker mode with systemd notify support.\n"
#endif
+ " -Wn master-worker mode with s6 notify support.\n"
" -q quiet mode : don't display messages\n"
" -c check mode : only check config files and exit\n"
" -n sets the maximum total # of connections (uses ulimit -n)\n"
@@ -652,6 +653,13 @@ static void mworker_loop()
if (global.tune.options & GTUNE_USE_SYSTEMD)
sd_notifyf(0, "READY=1\nMAINPID=%lu", (unsigned long)getpid());
#endif
+ /* if -Wn mode was requested then open fd 3, write an arbritrary
line and close the fd */
+ if (global.tune.options & GTUNE_USE_S6_NOTIFY) {
+ FILE *notifyfd;
+ notifyfd = fdopen(3, "w");
+ fprintf(notifyfd, "UP\n");
+ fclose(notifyfd);
+ }
/* Busy polling makes no sense in the master :-) */
global.tune.options &= ~GTUNE_BUSY_POLLING;
@@ -1427,6 +1435,10 @@ static void init(int argc, char **argv)
usage(progname);
#endif
}
+ else if (*flag == 'W' && flag[1] == 'n') {
+ arg_mode |= MODE_MWORKER | MODE_FOREGROUND;
+ global.tune.options |= GTUNE_USE_S6_NOTIFY;
+ }
else if (*flag == 'W')
arg_mode |= MODE_MWORKER;
else if (*flag == 'q')
--
2.9.0.windows.1
Re: [PATCH] MINOR: systemd: Make use of master socket in systemd unit
*From:* Willy Tarreau [mailto:w...@1wt.eu] *Sent:* Monday, April 29, 2019, 23:55 EDT *To:* William Lallemand *Cc:* Tim Düsterhus , Patrick Hemmer , haproxy@formilux.org *Subject:* [PATCH] MINOR: systemd: Make use of master socket in systemd unit On Tue, Apr 30, 2019 at 01:47:13AM +0200, William Lallemand wrote: the systemd API is limited and it's not possible to report a fail during a reload with sd_notify. Anyway a service manager which requires daemons to stay in foreground, does not allow processes to be replaced, and relies on asynchronous and unidirectional signal delivery to perform critical actions such as config updates is obviously misdesigned and can only result from the lack of experience of production servers and people making them work every day. I tend to think that the only longterm solution to work around all the systemd imposed mess is to have haproxy become its own service manager. It is a shame as it significantly complicates haproxy management on Linux compared to sane operating systems, but I'm having a hard time accepting that haproxy has become less reliable on Linux due to systemd. We've already made progress in this direction with the master-worker model, but we need to go further and make haproxy be able to serve as a client to talk to an existing master and return relevant status codes. This way we'll be able to emulate the start/reload/restart operations that are performed on sane systems and gain reliability again, at the expense of using a total of 3 processes talking to each other during a reload instead of just one. The advantage is that it should work on other operating systems too if needed. In my opinion this is something we seriously need to work on for 2.1, and address remaining limitations (doubled memory usage and CLI being closed on reload). By the way, does systemd support passing actions on stdin and getting the result on stdout ? Given that the master process is forced to stay in foreground, at least we could have the equivalent of the CLI on stdio used as a "console" and not require a third process to talk to it. Willy Not to start a systemd flame war, but you can always tell systemd to stop tracking the process, and you can then do whatever you want. Of course you won't get automatic restarts if it dies, but that's how all the other "sane operating systems" behave. Just add to the service file: Type=oneshot RemainAfterExit=yes Assuming we don't do that, and going back to the original issue at hand of putting the master socket in the systemd unit: do we want to do this? From all the discussion, whether or not we add a `haproxy reload` command to the haproxy binary, it still seems like we will need master socket support to make it work. So should this patch be adopted? -Patrick
Re: PATCH: fix typo
Hi Ilya, On Tue, Apr 30, 2019 at 09:23:21PM +0500, ?? wrote: > Hello, > > typo was found by cppcheck > > thank! > Ilya Shipitsin > From 62c6b78843e9c1b6f829872a54175f68332a2859 Mon Sep 17 00:00:00 2001 > From: Ilya Shipitsin > Date: Tue, 30 Apr 2019 21:21:28 +0500 > Subject: [PATCH] MINOR: fix typo "src" instead of "srv" > > --- > src/server.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/src/server.c b/src/server.c > index 12a14adc..9916361d 100644 > --- a/src/server.c > +++ b/src/server.c > @@ -1639,7 +1639,7 @@ static void srv_settings_cpy(struct server *srv, struct > server *src, int srv_tmp > srv->check.port = src->check.port; > srv->check.sni= src->check.sni; > srv->check.alpn_str = src->check.alpn_str; > - srv->check.alpn_len = srv->check.alpn_len; > + srv->check.alpn_len = src->check.alpn_len; > /* Note: 'flags' field has potentially been already initialized. */ > srv->flags |= src->flags; > srv->do_check = src->do_check; Oops, that seems right, it's been pushed, thanks a lot ! Olivier
PATCH: fix typo
Hello, typo was found by cppcheck thank! Ilya Shipitsin From 62c6b78843e9c1b6f829872a54175f68332a2859 Mon Sep 17 00:00:00 2001 From: Ilya Shipitsin Date: Tue, 30 Apr 2019 21:21:28 +0500 Subject: [PATCH] MINOR: fix typo "src" instead of "srv" --- src/server.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/server.c b/src/server.c index 12a14adc..9916361d 100644 --- a/src/server.c +++ b/src/server.c @@ -1639,7 +1639,7 @@ static void srv_settings_cpy(struct server *srv, struct server *src, int srv_tmp srv->check.port = src->check.port; srv->check.sni= src->check.sni; srv->check.alpn_str = src->check.alpn_str; - srv->check.alpn_len = srv->check.alpn_len; + srv->check.alpn_len = src->check.alpn_len; /* Note: 'flags' field has potentially been already initialized. */ srv->flags |= src->flags; srv->do_check = src->do_check; -- 2.21.0
Re: MEDIUM: Adding upstream socks4 proxy support
Alec,
Am 30.04.19 um 13:59 schrieb Alec Liu:
> What if we just have a reference here, and using malloc to alloc some memory
> for those connections which are going to use socks proxy only. It will be
> NULL for those connections which won't need it. When the socks proxy handshake
> finished, we can have this part of memory freed, before turn it over
> for data usage.
>
> I will try to look into it, if we can reuse the "proxy protocol
> offset", if we want to keep the ability to
> allow using PROXY Protocol and socks proxy at the same time.
>
I can't comment on whether using a pointer here makes sense or not, but
please note that HAProxy includes a memory pool implementation in
include/common/memory.h ("REGISTER_POOL"). I believe all your
allocations will have the same size which fits and you don't have to
annoy the system libc malloc all the time.
Best regards
Tim Düsterhus
Re: MEDIUM: Adding upstream socks4 proxy support
Hi Willy,
Thank for the look into it.
Here are some of my thoughts about your comments, pls check below.
> I do however have some comments, mainly about the way used to retrieve
> configuration elements as at the moment it doubles the memory consumption
> of each connection, but there are solutions to fix this that I describe
> below.
>
> > diff --git a/doc/SOCKS4.protocol.txt b/doc/SOCKS4.protocol.txt
> > new file mode 100644
> > index ..16483dee
> > --- /dev/null
> > +++ b/doc/SOCKS4.protocol.txt
> > @@ -0,0 +1,150 @@
> > + SOCKS: A protocol for TCP proxy across firewalls
> > +
> > + Ying-Da Lee
> > + yin...@best.com or yin...@esd.sgi.com
>
> (...)
>
> I'm seeing that you copied a doc retrieved from somewhere else that is
> found at various places on the net. Have you checked the license for
> this doc to be sure we can copy and distribute it like this ? It might
> be perfectly fine but we need to be sure. In any case we'll have to
> mention the license if it differs from the other ones.
I have the "SOCKS: A protocol for TCP proxy across firewalls" doc
found at openssh's website
under OpenSSH Sprcifications along with those RFCs.
"https://www.openssh.com/specs.html;,
"https://www.openssh.com/txt/socks4.protocol;
And OpenSSH is using GPL, so I believe it should be fine.
>
> > diff --git a/examples/socks4.cfg b/examples/socks4.cfg
> > new file mode 100644
> > index ..22cbad44
> > --- /dev/null
> > +++ b/examples/socks4.cfg
> > (...)
> > + server SMTPS1_Via_SocksProxy1 169.38.103.42:25 check inter 3
> > fastinter 1000
> > + server SMTPS2_Via_SocksProxy2 161.202.148.179:25 socks4
> > 127.0.0.1:1080 check-via-socks check inter 3 fastinter 1000 backup
>
> These ones are real mail servers which do respond on the net, it's not
> really friendly nor recommended to put real server addresses there, you
> can use the 192.0.2.0/24 range instead, which is reserved for
> documentation.
Will have it changed.
>
> > diff --git a/include/types/checks.h b/include/types/checks.h
> > index f89abcba..b09b5f89 100644
> > --- a/include/types/checks.h
> > +++ b/include/types/checks.h
> > @@ -189,6 +189,8 @@ struct check {
> > char *sni; /* Server name */
> > char *alpn_str; /* ALPN to use for checks */
> > int alpn_len; /* ALPN string length */
> > +
> > + int via_socks; /* check the connection via
> > socks proxy */
> > };
>
> At some point we'll have to think about creating a bit field for all
> these booleans because we're progressively inflating the struct by
> 32 bits every time we need a new flag. But let's do this later and
> not annoy you with this for now.
Totally agree, thx.
>
> > struct check_status {
> > diff --git a/include/types/connection.h b/include/types/connection.h
> > index 308be7d7..2a1b23b4 100644
> > --- a/include/types/connection.h
> > +++ b/include/types/connection.h
> > @@ -47,6 +47,15 @@ struct server;
> > struct session;
> > struct pipe;
> >
> > +/* socks4 upstream proxy definitions */
> > +struct socks4_request {
> > + uint8_t version;/* SOCKS version number, 1 byte, must be 0x04
> > for this version */
> > + uint8_t command;/* 0x01 = establish a TCP/IP stream
> > connection */
> > + uint16_t port; /* port number, 2 bytes (in network byte
> > order) */
> > + uint32_t ip;/* IP address, 4 bytes (in network byte
> > order) */
> > + char user_id[8];/* the user ID string, variable length,
> > terminated with a null (0x00); Using "HAProxy\0" */
> > +};
> > +
> > /* Note: subscribing to these events is only valid after the caller has
> > really
> > * attempted to perform the operation, and failed to proceed or complete.
> > */
> > @@ -156,7 +165,7 @@ enum {
> >
> > CO_FL_EARLY_SSL_HS = 0x4000, /* We have early data pending,
> > don't start SSL handshake yet */
> > CO_FL_EARLY_DATA= 0x8000, /* At least some of the data are
> > early data */
> > - /* unused : 0x0001 */
> > + CO_FL_SOCKS_HS = 0x0001, /* handshaking with upstream SOCKS
> > proxy */
> > /* unused : 0x0002 */
>
> From what I'm seeing later in the code, it's not exactly a handshake,
> you have to send a header and to wait for the response header to arrive.
> I'd rather have one flag per direction which save from having to deal
> with state management.
Actually, I have the same idea as you, using two flags only. But ends up with
this design later on. The reason I am doing it in this way, that's because:
1) I might want to add the socks5 protocol support onto it later on, which
will become a little more complex, not just one request and response.
We will get something like authentication method handshake, login,
connection request, etc.
2) Looks like do not have much unused bits leftover within CO_FL_*.
>
Re: [1.9.7] One of haproxy processes using 100% CPU
Hi Olivier, Thank you very much. I'll test it and get back with feedback! Regards, wt., 30 kwi 2019 o 13:12 Olivier Houchard napisał(a): > Hi Maciej, > > On Tue, Apr 30, 2019 at 08:43:07AM +0200, Maciej Zdeb wrote: > > Filtered results from show fd for that particular virtual server: > > > > 10 : st=0x22(R:pRa W:pRa) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x1 > > umask=0x0 owner=0x53a5690 iocb=0x59d689(conn_fd_handler) back=0 > > cflg=0x80243300 fe=virtual-server_front mux=H2 ctx=0x6502860 h2c.st0=2 > > .err=0 .maxid=17 .lastid=-1 .flg=0x1 .nbst=0 .nbcs=1 .fctl_cnt=0 > > .send_cnt=0 .tree_cnt=1 .orph_cnt=0 .sub=0 .dsi=13 .dbuf=0@(nil)+0/0 > > .msi=-1 .mbuf=0@(nil)+0/0 last_h2s=0x5907040 .id=13 .flg=0x4005 > > .rxbuf=0@(nil)+0/0 > > .cs=0x905b1b0 .cs.flg=0x00106a00 .cs.data=0x5d1d228 > > 98 : st=0x22(R:pRa W:pRa) ev=0x01(heopI) [lc] cnext=-1809 cprev=-2 > > tmask=0x1 umask=0x0 owner=0xa3bb7f0 iocb=0x59d689(conn_fd_handler) back=0 > > cflg=0x80201300 fe=virtual-server_front mux=H2 ctx=0xa71f310 h2c.st0=3 > > .err=0 .maxid=0 .lastid=-1 .flg=0x0008 .nbst=0 .nbcs=0 .fctl_cnt=0 > > .send_cnt=0 .tree_cnt=0 .orph_cnt=0 .sub=0 .dsi=3 > > .dbuf=16384@0x5873f10+61/16384 > > .msi=-1 .mbuf=0@(nil)+0/0 > > I see that it seems to be HTTP/2. Not saying it's your problem, but a bug > that would cause haproxy to use 100% of the CPU has been fixed in the > HTTP/2 > code just after the 1.9.7 release was done. > Any chance you can see if it still happens with that commit : > commit c980b511bfef566e9890eb9a06d607c193d63828 > Author: Willy Tarreau > Date: Mon Apr 29 10:20:21 2019 +0200 > > BUG/MEDIUM: mux-h2: properly deal with too large headers frames > > Regards, > > Olivier >
Re: [1.9.7] One of haproxy processes using 100% CPU
Hi Maciej, On Tue, Apr 30, 2019 at 08:43:07AM +0200, Maciej Zdeb wrote: > Filtered results from show fd for that particular virtual server: > > 10 : st=0x22(R:pRa W:pRa) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x1 > umask=0x0 owner=0x53a5690 iocb=0x59d689(conn_fd_handler) back=0 > cflg=0x80243300 fe=virtual-server_front mux=H2 ctx=0x6502860 h2c.st0=2 > .err=0 .maxid=17 .lastid=-1 .flg=0x1 .nbst=0 .nbcs=1 .fctl_cnt=0 > .send_cnt=0 .tree_cnt=1 .orph_cnt=0 .sub=0 .dsi=13 .dbuf=0@(nil)+0/0 > .msi=-1 .mbuf=0@(nil)+0/0 last_h2s=0x5907040 .id=13 .flg=0x4005 > .rxbuf=0@(nil)+0/0 > .cs=0x905b1b0 .cs.flg=0x00106a00 .cs.data=0x5d1d228 > 98 : st=0x22(R:pRa W:pRa) ev=0x01(heopI) [lc] cnext=-1809 cprev=-2 > tmask=0x1 umask=0x0 owner=0xa3bb7f0 iocb=0x59d689(conn_fd_handler) back=0 > cflg=0x80201300 fe=virtual-server_front mux=H2 ctx=0xa71f310 h2c.st0=3 > .err=0 .maxid=0 .lastid=-1 .flg=0x0008 .nbst=0 .nbcs=0 .fctl_cnt=0 > .send_cnt=0 .tree_cnt=0 .orph_cnt=0 .sub=0 .dsi=3 > .dbuf=16384@0x5873f10+61/16384 > .msi=-1 .mbuf=0@(nil)+0/0 I see that it seems to be HTTP/2. Not saying it's your problem, but a bug that would cause haproxy to use 100% of the CPU has been fixed in the HTTP/2 code just after the 1.9.7 release was done. Any chance you can see if it still happens with that commit : commit c980b511bfef566e9890eb9a06d607c193d63828 Author: Willy Tarreau Date: Mon Apr 29 10:20:21 2019 +0200 BUG/MEDIUM: mux-h2: properly deal with too large headers frames Regards, Olivier
PATCH: enable cirrus-ci (freebsd builds)
Hello. I enabled basic freebsd CI (yet, it need to be enabled on github.com in few easy steps) thanks! Ilya Shipitcin From c0feb90929b1e8f1987dde27fc9f57c9b6d8fc36 Mon Sep 17 00:00:00 2001 From: Ilya Shipitsin Date: Tue, 30 Apr 2019 15:33:36 +0500 Subject: [PATCH] BUILD: enable freebsd builds on cirrus-ci --- .cirrus.yml | 14 ++ 1 file changed, 14 insertions(+) create mode 100644 .cirrus.yml diff --git a/.cirrus.yml b/.cirrus.yml new file mode 100644 index ..df5e354a --- /dev/null +++ b/.cirrus.yml @@ -0,0 +1,14 @@ +FreeBSD_task: + freebsd_instance: +image: freebsd-12-0-release-amd64 + env: +ASSUME_ALWAYS_YES: TRUE # required for unattended "pkg" invocation + script: +- pkg install openssl111 git gmake lua53 +- git clone https://github.com/VTest/VTest.git ../vtest +- make -C ../vtest +- gmake CC=clang V=1 TARGET=freebsd USE_ZLIB=1 USE_PCRE=1 USE_OPENSSL=1 USE_LUA=1 LUA_INC=/usr/local/include/lua53 LUA_LIB=/usr/local/lib LUA_LIB_NAME=lua-5.3 +- ./haproxy -vv +- ldd haproxy +- env VTEST_PROGRAM=../vtest/vtest gmake reg-tests || (for folder in /tmp/*regtest*/vtc.*; do cat $folder/INFO $folder/LOG; done && exit 1) + -- 2.20.1
Re: reg-tests fail on FreeBSD 12.0
On Tue, Apr 30, 2019 at 11:43:31AM +0200, Frederic Lecaille wrote: > On 4/30/19 11:24 AM, ??? wrote: > > you are right. > > > > let us exclude that particular test from freebsd (what your patch > > exactly does) > > Ok Thank you Ilya. > > Willy could you merge this patch if we decide to disable this reg test > on FreeBSD systems. It had already been fixed to make it succeed on a > FreeBSD version I do not remember (see commit 9ffb88d). This issue came back > with 12.0 version with a similar regex. So I think it is time to disabled > this reg test for FreeBSD. Furthermore, it was originally dedicated to > Linux. OK merged, thanks. Willy
Re: reg-tests fail on FreeBSD 12.0
It still fails on 12, I applied patch, run test. So, I suggest to disable it On Tue, Apr 30, 2019, 2:43 PM Frederic Lecaille wrote: > On 4/30/19 11:24 AM, Илья Шипицин wrote: > > you are right. > > > > let us exclude that particular test from freebsd (what your patch > > exactly does) > > Ok Thank you Ilya. > > Willy could you merge this patch if we decide to disable this reg test > on FreeBSD systems. It had already been fixed to make it succeed on a > FreeBSD version I do not remember (see commit 9ffb88d). This issue came > back with 12.0 version with a similar regex. So I think it is time to > disabled this reg test for FreeBSD. Furthermore, it was originally > dedicated to Linux. > > If we do not decide to disable it on FreeBSD, we must rewrite the syslog > message regexes to make them more permissive as with 9ffb88d commit. > > Fred. > > > > > >
Re: [PATCH] MINOR: systemd: Make use of master socket in systemd unit
Willy, Am 30.04.19 um 05:55 schrieb Willy Tarreau: > By the way, does systemd support passing actions on stdin and getting > the result on stdout ? Given that the master process is forced to > stay in foreground, at least we could have the equivalent of the CLI > on stdio used as a "console" and not require a third process to talk > to it. stdin is usually closed, but can be connected to a TTY, socket or file: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#StandardInput= stdout (right below stdin in the docs) goes to the journal by default and I would not touch it (because you need to log with the journal API instead of stdout then). Best regards Tim Düsterhus
Re: reg-tests fail on FreeBSD 12.0
On 4/30/19 11:24 AM, Илья Шипицин wrote:
you are right.
let us exclude that particular test from freebsd (what your patch
exactly does)
Ok Thank you Ilya.
Willy could you merge this patch if we decide to disable this reg test
on FreeBSD systems. It had already been fixed to make it succeed on a
FreeBSD version I do not remember (see commit 9ffb88d). This issue came
back with 12.0 version with a similar regex. So I think it is time to
disabled this reg test for FreeBSD. Furthermore, it was originally
dedicated to Linux.
If we do not decide to disable it on FreeBSD, we must rewrite the syslog
message regexes to make them more permissive as with 9ffb88d commit.
Fred.
>From 0989af55e5a3d2d87defd64a68a78ff1057ad5db Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fr=C3=A9d=C3=A9ric=20L=C3=A9caille?=
Date: Mon, 29 Apr 2019 16:10:12 +0200
Subject: [PATCH] REGTEST: Make this reg test be Linux specific.
This patch reverts 9ffb88 commit (REGTEST: Be less Linux specific with a syslog
regex.) and makes this script be Linux specific.
---
reg-tests/checks/4be_1srv_smtpchk_httpchk_layer47errors.vtc | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/reg-tests/checks/4be_1srv_smtpchk_httpchk_layer47errors.vtc b/reg-tests/checks/4be_1srv_smtpchk_httpchk_layer47errors.vtc
index ffd31235e..e3c959f16 100644
--- a/reg-tests/checks/4be_1srv_smtpchk_httpchk_layer47errors.vtc
+++ b/reg-tests/checks/4be_1srv_smtpchk_httpchk_layer47errors.vtc
@@ -1,6 +1,7 @@
varnishtest "Check: smptchk option"
feature ignore_unknown_macro
+#EXCLUDE_TARGETS=freebsd,osx,generic
#REGTEST_TYPE=slow
barrier b cond 3
@@ -29,7 +30,7 @@ syslog S3 -level notice {
recv
expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be3 started"
recv
-expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be3/srv3 failed, reason: Layer4 .* check duration: [[:digit:]]+ms, status: 0/1 DOWN."
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be3/srv3 failed, reason: Layer4 connection problem, info: \"General socket error \\(Network is unreachable\\)\", check duration: [[:digit:]]+ms, status: 0/1 DOWN."
} -start
syslog S4 -level notice {
--
2.11.0
Re: reg-tests fail on FreeBSD 12.0
you are right.
let us exclude that particular test from freebsd (what your patch exactly
does)
вт, 30 апр. 2019 г. в 13:57, Илья Шипицин :
>
>
> вт, 30 апр. 2019 г. в 13:32, Frederic Lecaille :
>
>> On 4/30/19 10:04 AM, Илья Шипицин wrote:
>> > Hello, Frederic
>> >
>> > I have applied only that part
>> >
>> > barrier b cond 3
>> > @@ -29,7 +30,7 @@ syslog S3 -level notice {
>> > recv
>> > expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be3 started"
>> > recv
>> > -expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server
>> be3/srv3
>> > failed, reason: Layer4 .* check duration: [[:digit:]]+ms, status: 0/1
>> DOWN."
>> > +expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server
>> be3/srv3
>> > failed, reason: Layer4 connection problem, info: \"General socket error
>> > \\(Network is unreachable\\)\", check duration: [[:digit:]]+ms, status:
>> > 0/1 DOWN."
>> > } -start
>> >
>> >
>> > it works, test is green now (on freebsd)
>> >
>> >
>> > https://cirrus-ci.com/task/6252507322908672
>> >
>> > пн, 29 апр. 2019 г. в 19:14, Frederic Lecaille > > >:
>> >
>> > On 4/29/19 3:56 PM, Илья Шипицин wrote:
>> > > well, in cirrus-ci we can choose several freebsd images. I just
>> > picked
>> > > up the most recent one.
>> > >
>> > > I'll try older images soon.
>> > >
>> > > actually, there are several options
>> > >
>> > > 1) add special "ignore" to those tests (can we ignore certain
>> > tests, for
>> >
>> > Well this reg test was originally Linux specific.
>> >
>> > With the patch attached we can make it be Linux specific again.
>> >
>> >
>> > Fred.
>> >
>>
>> Hello Ilya,
>>
>> I am not sure to understand everything.
>>
>
> I did play with output from various cirrus-ci steps and I forgot that
> script currently looks like
>
> script:
> - pkg install openssl111 git gmake lua53
> #- git clone https://github.com/VTest/VTest.git ../vtest
> #- make -C ../vtest
> #- gmake CC=clang V=1 TARGET=freebsd USE_ZLIB=1 USE_PCRE=1
> USE_OPENSSL=1 USE_LUA=1 LUA_INC=/usr/local/include/lua53
> LUA_LIB=/usr/local/lib LUA_LIB_NAME=lua-5.3
> #- ./haproxy -vv
> #- ldd haproxy
> #- env VTEST_PROGRAM=../vtest/vtest gmake reg-tests || for folder in
> /tmp/*regtest*/vtc.*; do cat $folder/INFO $folder/LOG; done
>
> I'll test later
>
>
>
>>
>> In fact the patch I sent is supposed to disable the reg test on freebsd
>> because it also restores a Linux specific regex on syslog messages. The
>> reg test should not be run on FreeBSD, thanks to this patch. The part of
>> the patch you applied is Linux specific. If you do not take the other
>> part of the patch, the reg test is not disabled on freebsd. So it should
>> fail contrary to what you reported.
>> https://cirrus-ci.com/build/6289311702974464 does not demonstrate the
>> reg test succeeds. Isn't this only a build report? If I use the task ID
>> 6252507322908672 and have a look at the log here:
>>
>> https://api.cirrus-ci.com/v1/task/6252507322908672/logs/main.log
>>
>> I do not see any reg test result.
>>
>>
>> Fred.
>>
>>
>>
>>
Re: reg-tests fail on FreeBSD 12.0
вт, 30 апр. 2019 г. в 13:32, Frederic Lecaille :
> On 4/30/19 10:04 AM, Илья Шипицин wrote:
> > Hello, Frederic
> >
> > I have applied only that part
> >
> > barrier b cond 3
> > @@ -29,7 +30,7 @@ syslog S3 -level notice {
> > recv
> > expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be3 started"
> > recv
> > -expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be3/srv3
> > failed, reason: Layer4 .* check duration: [[:digit:]]+ms, status: 0/1
> DOWN."
> > +expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be3/srv3
> > failed, reason: Layer4 connection problem, info: \"General socket error
> > \\(Network is unreachable\\)\", check duration: [[:digit:]]+ms, status:
> > 0/1 DOWN."
> > } -start
> >
> >
> > it works, test is green now (on freebsd)
> >
> >
> > https://cirrus-ci.com/task/6252507322908672
> >
> > пн, 29 апр. 2019 г. в 19:14, Frederic Lecaille > >:
> >
> > On 4/29/19 3:56 PM, Илья Шипицин wrote:
> > > well, in cirrus-ci we can choose several freebsd images. I just
> > picked
> > > up the most recent one.
> > >
> > > I'll try older images soon.
> > >
> > > actually, there are several options
> > >
> > > 1) add special "ignore" to those tests (can we ignore certain
> > tests, for
> >
> > Well this reg test was originally Linux specific.
> >
> > With the patch attached we can make it be Linux specific again.
> >
> >
> > Fred.
> >
>
> Hello Ilya,
>
> I am not sure to understand everything.
>
I did play with output from various cirrus-ci steps and I forgot that
script currently looks like
script:
- pkg install openssl111 git gmake lua53
#- git clone https://github.com/VTest/VTest.git ../vtest
#- make -C ../vtest
#- gmake CC=clang V=1 TARGET=freebsd USE_ZLIB=1 USE_PCRE=1
USE_OPENSSL=1 USE_LUA=1 LUA_INC=/usr/local/include/lua53
LUA_LIB=/usr/local/lib LUA_LIB_NAME=lua-5.3
#- ./haproxy -vv
#- ldd haproxy
#- env VTEST_PROGRAM=../vtest/vtest gmake reg-tests || for folder in
/tmp/*regtest*/vtc.*; do cat $folder/INFO $folder/LOG; done
I'll test later
>
> In fact the patch I sent is supposed to disable the reg test on freebsd
> because it also restores a Linux specific regex on syslog messages. The
> reg test should not be run on FreeBSD, thanks to this patch. The part of
> the patch you applied is Linux specific. If you do not take the other
> part of the patch, the reg test is not disabled on freebsd. So it should
> fail contrary to what you reported.
> https://cirrus-ci.com/build/6289311702974464 does not demonstrate the
> reg test succeeds. Isn't this only a build report? If I use the task ID
> 6252507322908672 and have a look at the log here:
>
> https://api.cirrus-ci.com/v1/task/6252507322908672/logs/main.log
>
> I do not see any reg test result.
>
>
> Fred.
>
>
>
>
Re: reg-tests fail on FreeBSD 12.0
On 4/30/19 10:04 AM, Илья Шипицин wrote:
Hello, Frederic
I have applied only that part
barrier b cond 3
@@ -29,7 +30,7 @@ syslog S3 -level notice {
recv
expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be3 started"
recv
- expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be3/srv3
failed, reason: Layer4 .* check duration: [[:digit:]]+ms, status: 0/1 DOWN."
+ expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be3/srv3
failed, reason: Layer4 connection problem, info: \"General socket error
\\(Network is unreachable\\)\", check duration: [[:digit:]]+ms, status:
0/1 DOWN."
} -start
it works, test is green now (on freebsd)
https://cirrus-ci.com/task/6252507322908672
пн, 29 апр. 2019 г. в 19:14, Frederic Lecaille >:
On 4/29/19 3:56 PM, Илья Шипицин wrote:
> well, in cirrus-ci we can choose several freebsd images. I just
picked
> up the most recent one.
>
> I'll try older images soon.
>
> actually, there are several options
>
> 1) add special "ignore" to those tests (can we ignore certain
tests, for
Well this reg test was originally Linux specific.
With the patch attached we can make it be Linux specific again.
Fred.
Hello Ilya,
I am not sure to understand everything.
In fact the patch I sent is supposed to disable the reg test on freebsd
because it also restores a Linux specific regex on syslog messages. The
reg test should not be run on FreeBSD, thanks to this patch. The part of
the patch you applied is Linux specific. If you do not take the other
part of the patch, the reg test is not disabled on freebsd. So it should
fail contrary to what you reported.
https://cirrus-ci.com/build/6289311702974464 does not demonstrate the
reg test succeeds. Isn't this only a build report? If I use the task ID
6252507322908672 and have a look at the log here:
https://api.cirrus-ci.com/v1/task/6252507322908672/logs/main.log
I do not see any reg test result.
Fred.
Re: reg-tests fail on FreeBSD 12.0
Hello, Frederic
I have applied only that part
barrier b cond 3
@@ -29,7 +30,7 @@ syslog S3 -level notice {
recv
expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be3 started"
recv
-expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be3/srv3
failed, reason: Layer4 .* check duration: [[:digit:]]+ms, status: 0/1 DOWN."
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be3/srv3
failed, reason: Layer4 connection problem, info: \"General socket error
\\(Network is unreachable\\)\", check duration: [[:digit:]]+ms, status: 0/1
DOWN."
} -start
it works, test is green now (on freebsd)
https://cirrus-ci.com/task/6252507322908672
пн, 29 апр. 2019 г. в 19:14, Frederic Lecaille :
> On 4/29/19 3:56 PM, Илья Шипицин wrote:
> > well, in cirrus-ci we can choose several freebsd images. I just picked
> > up the most recent one.
> >
> > I'll try older images soon.
> >
> > actually, there are several options
> >
> > 1) add special "ignore" to those tests (can we ignore certain tests, for
>
> Well this reg test was originally Linux specific.
>
> With the patch attached we can make it be Linux specific again.
>
>
> Fred.
>
Re: [1.9.7] One of haproxy processes using 100% CPU
Filtered results from show fd for that particular virtual server:
10 : st=0x22(R:pRa W:pRa) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x1
umask=0x0 owner=0x53a5690 iocb=0x59d689(conn_fd_handler) back=0
cflg=0x80243300 fe=virtual-server_front mux=H2 ctx=0x6502860 h2c.st0=2
.err=0 .maxid=17 .lastid=-1 .flg=0x1 .nbst=0 .nbcs=1 .fctl_cnt=0
.send_cnt=0 .tree_cnt=1 .orph_cnt=0 .sub=0 .dsi=13 .dbuf=0@(nil)+0/0
.msi=-1 .mbuf=0@(nil)+0/0 last_h2s=0x5907040 .id=13 .flg=0x4005
.rxbuf=0@(nil)+0/0
.cs=0x905b1b0 .cs.flg=0x00106a00 .cs.data=0x5d1d228
98 : st=0x22(R:pRa W:pRa) ev=0x01(heopI) [lc] cnext=-1809 cprev=-2
tmask=0x1 umask=0x0 owner=0xa3bb7f0 iocb=0x59d689(conn_fd_handler) back=0
cflg=0x80201300 fe=virtual-server_front mux=H2 ctx=0xa71f310 h2c.st0=3
.err=0 .maxid=0 .lastid=-1 .flg=0x0008 .nbst=0 .nbcs=0 .fctl_cnt=0
.send_cnt=0 .tree_cnt=0 .orph_cnt=0 .sub=0 .dsi=3
.dbuf=16384@0x5873f10+61/16384
.msi=-1 .mbuf=0@(nil)+0/0
184 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lC] cnext=-3 cprev=-2
tmask=0x umask=0x0 owner=0x23eb040
iocb=0x57e662(listener_accept) l.st=RDY fe=virtual-server_front
660 : st=0x22(R:pRa W:pRa) ev=0x11(HeopI) [lc] cnext=-3 cprev=-2 tmask=0x1
umask=0x0 owner=0x533d6e0 iocb=0x59d689(conn_fd_handler) back=0
cflg=0x80243300 fe=virtual-server_front mux=H2 ctx=0x8031b90 h2c.st0=2
.err=0 .maxid=49 .lastid=-1 .flg=0x1 .nbst=0 .nbcs=1 .fctl_cnt=0
.send_cnt=0 .tree_cnt=1 .orph_cnt=0 .sub=0 .dsi=49 .dbuf=0@(nil)+0/0
.msi=-1 .mbuf=0@(nil)+0/0 last_h2s=0x70f1e80 .id=31 .flg=0x4005
.rxbuf=0@(nil)+0/0
.cs=0x6f373d0 .cs.flg=0x00106a00 .cs.data=0x56bb788
699 : st=0x22(R:pRa W:pRa) ev=0x11(HeopI) [lc] cnext=-87 cprev=-2 tmask=0x1
umask=0x0 owner=0x6694b60 iocb=0x59d689(conn_fd_handler) back=0
cflg=0x80243300 fe=virtual-server_front mux=H2 ctx=0x56e7b00 h2c.st0=2
.err=0 .maxid=111 .lastid=-1 .flg=0x1 .nbst=0 .nbcs=1 .fctl_cnt=0
.send_cnt=0 .tree_cnt=1 .orph_cnt=0 .sub=0 .dsi=111 .dbuf=0@(nil)+0/0
.msi=-1 .mbuf=0@(nil)+0/0 last_h2s=0x5931bf0 .id=47 .flg=0x4105
.rxbuf=0@(nil)+0/0
.cs=0x5943120 .cs.flg=0x00106a00 .cs.data=0x77af4c8
970 : st=0x22(R:pRa W:pRa) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x1
umask=0x0 owner=0x67684b0 iocb=0x59d689(conn_fd_handler) back=0
cflg=0x80243300 fe=virtual-server_front mux=H2 ctx=0x5c90c30 h2c.st0=2
.err=0 .maxid=125 .lastid=-1 .flg=0x1 .nbst=0 .nbcs=1 .fctl_cnt=0
.send_cnt=0 .tree_cnt=1 .orph_cnt=0 .sub=0 .dsi=125 .dbuf=0@(nil)+0/0
.msi=-1 .mbuf=0@(nil)+0/0 last_h2s=0x7ac8650 .id=43 .flg=0x4005
.rxbuf=0@(nil)+0/0
.cs=0x7901a20 .cs.flg=0x00106a00 .cs.data=0x882c388
1282 : st=0x22(R:pRa W:pRa) ev=0x11(HeopI) [lc] cnext=-3 cprev=-2 tmask=0x1
umask=0x0 owner=0x6f23720 iocb=0x59d689(conn_fd_handler) back=0
cflg=0x80243300 fe=virtual-server_front mux=H2 ctx=0x6090cf0 h2c.st0=2
.err=0 .maxid=129 .lastid=-1 .flg=0x1 .nbst=0 .nbcs=1 .fctl_cnt=0
.send_cnt=0 .tree_cnt=1 .orph_cnt=0 .sub=0 .dsi=129 .dbuf=0@(nil)+0/0
.msi=-1 .mbuf=0@(nil)+0/0 last_h2s=0x5cc0890 .id=17 .flg=0x4005
.rxbuf=0@(nil)+0/0
.cs=0x64d33f0 .cs.flg=0x00106a00 .cs.data=0x639a3e8
3041 : st=0x22(R:pRa W:pRa) ev=0x11(HeopI) [lc] cnext=-955 cprev=-2
tmask=0x1 umask=0x0 owner=0x6de8980 iocb=0x59d689(conn_fd_handler) back=0
cflg=0x80243300 fe=virtual-server_front mux=H2 ctx=0x5beca10 h2c.st0=2
.err=0 .maxid=89 .lastid=-1 .flg=0x1 .nbst=0 .nbcs=1 .fctl_cnt=0
.send_cnt=0 .tree_cnt=1 .orph_cnt=0 .sub=0 .dsi=89 .dbuf=0@(nil)+0/0
.msi=-1 .mbuf=0@(nil)+0/0 last_h2s=0x82f5900 .id=15 .flg=0x4005
.rxbuf=0@(nil)+0/0
.cs=0x7e027a0 .cs.flg=0x00106a00 .cs.data=0x6e5d398
wt., 30 kwi 2019 o 08:31 Maciej Zdeb napisał(a):
> Forgot to attach information about HAProxy (for tracing the issue I
> compiled it with debug symbols and without optimizations):
>
> haproxy -vv
> HA-Proxy version 1.9.7 2019/04/25 - https://haproxy.org/
> Build options :
> TARGET = linux2628
> CPU = generic
> CC = gcc
> CFLAGS = -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement
> -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
> -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
> -Wno-missing-field-initializers -Wtype-limits -DIP_BIND_ADDRESS_NO_PORT=24
> OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_DL=1
> USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1
>
> Default settings :
> maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
>
> Built with OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019
> Running on OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
> Built with Lua version : Lua 5.3.5
> Built with transparent proxy support using: IP_TRANSPARENT
> IPV6_TRANSPARENT IP_FREEBIND
> Built with zlib version : 1.2.8
> Running on zlib version : 1.2.8
> Compression algorithms supported : identity("identity"),
> deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
> Built with PCRE version : 8.43
Re: [1.9.7] One of haproxy processes using 100% CPU
Forgot to attach information about HAProxy (for tracing the issue I
compiled it with debug symbols and without optimizations):
haproxy -vv
HA-Proxy version 1.9.7 2019/04/25 - https://haproxy.org/
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wtype-limits -DIP_BIND_ADDRESS_NO_PORT=24
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_DL=1
USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019
Running on OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.43 2019-02-23
Running on PCRE version : 8.43 2019-02-23
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE
h2 : mode=HTXside=FE|BE
: mode=HTXside=FE|BE
: mode=TCP|HTTP side=FE|BE
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
wt., 30 kwi 2019 o 08:16 Maciej Zdeb napisał(a):
> Hi,
>
> I'm returning with problem similar as in thread "[1.9.6] One of haproxy
> processes using 100% CPU".
>
> Thanks to Olivier hard work, some issues where fixed but still not all of
> them. :( Currently it is much harder to trigger and it occurs on https
> virtual server and not tcp one. I'm observing this problem for HAProxy
> 1.9.7, process starts using 100% cpu and admin socket is still responsive.
>
> Attached gdb session. Please let me know if you need some more info from
> gdb, unfortunately I'm not gdb expert and I'm not sure what to look for. It
> is production server, however I can keep it for a couple hours isolated
> from production traffic.
>
> Pasting some anonymized info from show sess, please note that it was
> dumped on 30/Apr/2019:06:53:54 (before dump, traffic was switched to
> another server) but some connections are from 29/Apr/2019 and persist in
> that strange state.
>
> CC: Oliver :)
>
> socat /var/run/haproxy/haproxy1.sock - <<< "show sess all"
> 0x56bb4f0: [29/Apr/2019:12:31:37.186578] id=14500574 proto=tcpv4
> source=C.C.C.C:56567
> flags=0x44e, conn_retries=2, srv_conn=0x24db060, pend_pos=(nil) waiting=0
> frontend=virtual-server_front (id=45 mode=http), listener=? (id=1)
> addr=V.V.V.V:443
> backend=V.V.V.V:443_back (id=46 mode=http) addr=S.S.S.S:57654
> server=slot_9_3 (id=78) addr=B.B.B.B:31160
> task=0x6ec4070 (state=0x00 nice=0 calls=2 exp= tmask=0x1
> age=18h22m)
> txn=0x57eadf0 flags=0x88003000 meth=1 status=-1 req.st=MSG_DONE rsp.st
> =MSG_RPBEFORE
> req.f=0x0c blen=0 chnk=0 next=0
> rsp.f=0x00 blen=0 chnk=0 next=0
> si[0]=0x56bb788 (state=EST flags=0x48008 endp0=CS:0x6f373d0 exp=
> et=0x000 sub=0)
> si[1]=0x56bb7c8 (state=CON flags=0x111 endp1=CS:0x53a9250 exp=
> et=0x008 sub=2)
> co0=0x533d6e0 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM
> target=LISTENER:0x23eb040
> flags=0x80243300 fd=660 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1
> cs=0x6f373d0 csf=0x00106a00 ctx=0x70f1e80
> co1=0x91263b0 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM
> target=SERVER:0x24db060
> flags=0x00403370 fd=51 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1
> cs=0x53a9250 csf=0x0200 ctx=(nil)
> req=0x56bb500 (f=0x4cc0 an=0x8000 pipe=0 tofwd=0 total=1610)
> an_exp= rex= wex=
> buf=0x56bb508 data=0x58ba660 o=1831 p=1831 req.next=0 i=0 size=16384
> res=0x56bb560 (f=0x8000 an=0x0 pipe=0 tofwd=0 total=0)
> an_exp= rex= wex=
> buf=0x56bb568 data=(nil) o=0 p=0 rsp.next=0 i=0 size=0
> 0x639a150: [29/Apr/2019:12:31:53.231220] id=14501038 proto=tcpv4
> source=C2.C2.C2.C2:30107
> flags=0x44e, conn_retries=2, srv_conn=0x24cdad0, pend_pos=(nil) waiting=0
> frontend=virtual-server_front (id=45 mode=http), listener=? (id=1)
> addr=V.V.V.V:443
> backend=V.V.V.V:443_back (id=46 mode=http) addr=S2.S2.S2.S2:8760
> server=slot_9_2 (id=61) addr=B.B.B.B:31160
>
[1.9.7] One of haproxy processes using 100% CPU
Hi, I'm returning with problem similar as in thread "[1.9.6] One of haproxy processes using 100% CPU". Thanks to Olivier hard work, some issues where fixed but still not all of them. :( Currently it is much harder to trigger and it occurs on https virtual server and not tcp one. I'm observing this problem for HAProxy 1.9.7, process starts using 100% cpu and admin socket is still responsive. Attached gdb session. Please let me know if you need some more info from gdb, unfortunately I'm not gdb expert and I'm not sure what to look for. It is production server, however I can keep it for a couple hours isolated from production traffic. Pasting some anonymized info from show sess, please note that it was dumped on 30/Apr/2019:06:53:54 (before dump, traffic was switched to another server) but some connections are from 29/Apr/2019 and persist in that strange state. CC: Oliver :) socat /var/run/haproxy/haproxy1.sock - <<< "show sess all" 0x56bb4f0: [29/Apr/2019:12:31:37.186578] id=14500574 proto=tcpv4 source=C.C.C.C:56567 flags=0x44e, conn_retries=2, srv_conn=0x24db060, pend_pos=(nil) waiting=0 frontend=virtual-server_front (id=45 mode=http), listener=? (id=1) addr=V.V.V.V:443 backend=V.V.V.V:443_back (id=46 mode=http) addr=S.S.S.S:57654 server=slot_9_3 (id=78) addr=B.B.B.B:31160 task=0x6ec4070 (state=0x00 nice=0 calls=2 exp= tmask=0x1 age=18h22m) txn=0x57eadf0 flags=0x88003000 meth=1 status=-1 req.st=MSG_DONE rsp.st =MSG_RPBEFORE req.f=0x0c blen=0 chnk=0 next=0 rsp.f=0x00 blen=0 chnk=0 next=0 si[0]=0x56bb788 (state=EST flags=0x48008 endp0=CS:0x6f373d0 exp= et=0x000 sub=0) si[1]=0x56bb7c8 (state=CON flags=0x111 endp1=CS:0x53a9250 exp= et=0x008 sub=2) co0=0x533d6e0 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM target=LISTENER:0x23eb040 flags=0x80243300 fd=660 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 cs=0x6f373d0 csf=0x00106a00 ctx=0x70f1e80 co1=0x91263b0 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM target=SERVER:0x24db060 flags=0x00403370 fd=51 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 cs=0x53a9250 csf=0x0200 ctx=(nil) req=0x56bb500 (f=0x4cc0 an=0x8000 pipe=0 tofwd=0 total=1610) an_exp= rex= wex= buf=0x56bb508 data=0x58ba660 o=1831 p=1831 req.next=0 i=0 size=16384 res=0x56bb560 (f=0x8000 an=0x0 pipe=0 tofwd=0 total=0) an_exp= rex= wex= buf=0x56bb568 data=(nil) o=0 p=0 rsp.next=0 i=0 size=0 0x639a150: [29/Apr/2019:12:31:53.231220] id=14501038 proto=tcpv4 source=C2.C2.C2.C2:30107 flags=0x44e, conn_retries=2, srv_conn=0x24cdad0, pend_pos=(nil) waiting=0 frontend=virtual-server_front (id=45 mode=http), listener=? (id=1) addr=V.V.V.V:443 backend=V.V.V.V:443_back (id=46 mode=http) addr=S2.S2.S2.S2:8760 server=slot_9_2 (id=61) addr=B.B.B.B:31160 task=0x6625ab0 (state=0x00 nice=0 calls=2 exp= tmask=0x1 age=18h22m) txn=0x5af9a10 flags=0x88003000 meth=1 status=-1 req.st=MSG_DONE rsp.st =MSG_RPBEFORE req.f=0x0c blen=0 chnk=0 next=0 rsp.f=0x00 blen=0 chnk=0 next=0 si[0]=0x639a3e8 (state=EST flags=0x48008 endp0=CS:0x64d33f0 exp= et=0x000 sub=0) si[1]=0x639a428 (state=CON flags=0x111 endp1=CS:0x5d681c0 exp= et=0x008 sub=2) co0=0x6f23720 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM target=LISTENER:0x23eb040 flags=0x80243300 fd=1282 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 cs=0x64d33f0 csf=0x00106a00 ctx=0x5cc0890 co1=0x6039c60 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM target=SERVER:0x24cdad0 flags=0x00403370 fd=577 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 cs=0x5d681c0 csf=0x0200 ctx=0x5d681e0 req=0x639a160 (f=0x4cc0 an=0x8000 pipe=0 tofwd=0 total=1137) an_exp= rex= wex= buf=0x639a168 data=0x5408ef0 o=1349 p=1349 req.next=0 i=0 size=16384 res=0x639a1c0 (f=0x8000 an=0x0 pipe=0 tofwd=0 total=0) an_exp= rex= wex= buf=0x639a1c8 data=(nil) o=0 p=0 rsp.next=0 i=0 size=0 0x882c0f0: [29/Apr/2019:12:31:46.503263] id=14503967 proto=tcpv4 source= 5.173.79.240:7434 flags=0x44e, conn_retries=2, srv_conn=0x24cdad0, pend_pos=(nil) waiting=0 frontend=virtual-server_front (id=45 mode=http), listener=? (id=1) addr=V.V.V.V:443 backend=V.V.V.V:443_back (id=46 mode=http) addr=S2.S2.S2.S2:24688 server=slot_9_2 (id=61) addr=B.B.B.B:31160 task=0x7c13f50 (state=0x00 nice=0 calls=2 exp= tmask=0x1 age=18h22m) txn=0x5978210 flags=0x88003000 meth=1 status=-1 req.st=MSG_DONE rsp.st =MSG_RPBEFORE req.f=0x0c blen=0 chnk=0 next=0 rsp.f=0x00 blen=0 chnk=0 next=0 si[0]=0x882c388 (state=EST flags=0x48008 endp0=CS:0x7901a20 exp= et=0x000 sub=0) si[1]=0x882c3c8 (state=CON flags=0x111 endp1=CS:0x6ca3a50 exp= et=0x008 sub=2) co0=0x67684b0 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM target=LISTENER:0x23eb040 flags=0x80243300 fd=970 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 cs=0x7901a20 csf=0x00106a00 ctx=0x7ac8650 co1=0x74cba90 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM target=SERVER:0x24cdad0 flags=0x00403370 fd=2185 fd.state=22 fd.cache=0 updt=0
