Re: problem in 1.8 with hosts going out of service

[Paul Lockaby]

This patch works for me. Thank you!


> On Jan 24, 2018, at 1:02 PM, Christopher Faulet  wrote:
> 
> Le 24/01/2018 à 17:21, Paul Lockaby a écrit :
>> Sorry, I know this list is super busy and that there are a number of other 
>> more important issues that need to be worked through but I'm hoping one of 
>> the maintainers has been able to confirm this bug?
> 
> Hi,
> 
> Sorry Paul. As you said, we are pretty busy. And you're right to ping us. So, 
> I can confirm the bug. It is a bug on threads, a deadlock, because of a typo.
> 
> Could you check the attached patch to confirm it fixes your problem ?
> 
> Thanks,
> -- 
> Christopher Faulet
> <0001-BUG-MEDIUM-threads-server-Fix-deadlock-in-srv_set_st.patch>

Re: problem in 1.8 with hosts going out of service

[PiBa-NL]

Hi Christopher,

Patch seems to work for me.
Maybe Paul can confirm as well.

Regards,
PiBa-NL / Pieter

Op 24-1-2018 om 22:02 schreef Christopher Faulet:

Le 24/01/2018 à 17:21, Paul Lockaby a écrit :
Sorry, I know this list is super busy and that there are a number of 
other more important issues that need to be worked through but I'm 
hoping one of the maintainers has been able to confirm this bug?




Hi,

Sorry Paul. As you said, we are pretty busy. And you're right to ping 
us. So, I can confirm the bug. It is a bug on threads, a deadlock, 
because of a typo.


Could you check the attached patch to confirm it fixes your problem ?

Thanks,

Re: [PATCH] CONTRIB: log: emit warning when -sf/-sd cannot parse argument

[Christopher Lane]

Ping; this is a safe change to wanr about a confusing source of error when
you don't quote the arguments to haproxy in a "soft reloading new config a
lot" type of environment.

On Thu, Dec 7, 2017 at 1:17 PM Chris Lane  wrote:

>
> Previously, -sf and -sd command line parsing used atol which cannot
> detect errors.  I had a problem where I was doing -sf "$pid1 $pid2 $pid"
> and it was sending the gracefully terminate signal only to the first pid.
> The change uses strtol and checks endptr and errno to see if the parsing
> worked.  It doesn't exit so as not to cause failures but will allow
> trouble-shooting
> to be faster.
> ---
>  src/haproxy.c | 13 -
>  1 file changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/src/haproxy.c b/src/haproxy.c
> index eb5e65b..3185a2e 100644
> --- a/src/haproxy.c
> +++ b/src/haproxy.c
> @@ -1412,13 +1412,24 @@ static void init(int argc, char **argv)
> else
> oldpids_sig = SIGTERM; /*
> terminate immediately */
> while (argc > 1 && argv[1][0] != '-') {
> +   char * endptr = NULL;
> oldpids = realloc(oldpids,
> (nb_oldpids + 1) * sizeof(int));
> if (!oldpids) {
> ha_alert("Cannot allocate
> old pid : out of memory.\n");
> exit(1);
> }
> argc--; argv++;
> -   oldpids[nb_oldpids] = atol(*argv);
> +   errno = 0;
> +   oldpids[nb_oldpids] =
> strtol(*argv, , 10);
> +   if (errno) {
> +   ha_alert("-%2s option:
> failed to parse {%s}: %s\n",
> +(char *)*argv,
> strerror(errno));
> +   } else if (endptr &&
> strlen(endptr)) {
> +   while (isspace(*endptr))
> endptr++;
> +   if (*endptr != 0)
> +   ha_alert("-%2s
> option: some bytes unconsumed in PID list {%s}\n",
> +flag,
> endptr);
> +   }
> if (oldpids[nb_oldpids] <= 0)
> usage(progname);
> nb_oldpids++;
> --
> 2.1.1
>
>
>

Re: problem in 1.8 with hosts going out of service (alive.txt+404+track)

[PiBa-NL]

Hi Paul, List,

Op 24-1-2018 om 17:21 schreef Paul Lockaby:

Sorry, I know this list is super busy and that there are a number of other more 
important issues that need to be worked through but I'm hoping one of the 
maintainers has been able to confirm this bug?
I can reproduce it indeed with your config on 1.8.3, cpu usage go's 100% 
and stats stops responding when the alive.txt is removed.
When a backend go's down completely the track works as intended though 
(stats shows both server marked down).
Hope this and below added info / smaller config helps someone track the 
issue down further in the code.


Thanks,
-Paul


On Jan 17, 2018, at 10:27 AM, Paul Lockaby  wrote:

Ok I've tracked this problem down specifically to the usage of check tracking.

That is to say, the backend "example-api" is set to track the backend 
"example-http". When that tracking is enabled and one of the servers in the backend goes 
down then all of haproxy goes down and never recovers.

So this works:
server myhost myhost.example.com:8445 ssl ca-file 
/usr/local/ssl/certs/cacerts.cert

But this does not:
server myhost myhost.example.com:8445 track example-http/myhost ssl ca-file 
/usr/local/ssl/certs/cacerts.cert

This is definitely a regression from 1.7 because I used this feature in 1.7 
without issue.



It seems a combination of 3 combined features to trigger this issue:
    option httpchk GET /alive.txt + http-check disable-on-404 + track

Backtrace keeps showing this:
(gdb) bt
#0  0x0046b59f in srv_set_stopping ()
#1  0x004a3057 in ?? ()
#2  0x004f0eaf in process_runnable_tasks ()
#3  0x004aa13c in ?? ()
#4  0x004a9a16 in main ()

I could reduce the config to this:

frontend stats-frontend
    bind *:2999
    mode http
    log global
    stats enable
    stats uri /haproxy

frontend stats-frontend
    bind *:2999
    mode http
    log global
    stats enable
    stats uri /haproxy

frontend secured
    bind *:8080
    mode http
    acl request_api hdr_beg(Host) -i api.
    use_backend example-api if request_api
    default_backend example-http

backend example-http
    mode http
    option httpchk GET /haproxy/alive.txt
    http-check disable-on-404
    server myhost vhost1.pfs.local:302 check

backend example-api
    mode http
    option httpchk GET /haproxy/alive.txt
    http-check disable-on-404
    server myhost vhost1.pfs.local:303 track example-http/myhost

Regards,

PiBa-NL / Pieter

Re: problem in 1.8 with hosts going out of service

[Christopher Faulet]

Le 24/01/2018 à 17:21, Paul Lockaby a écrit :

Sorry, I know this list is super busy and that there are a number of other more 
important issues that need to be worked through but I'm hoping one of the 
maintainers has been able to confirm this bug?



Hi,

Sorry Paul. As you said, we are pretty busy. And you're right to ping 
us. So, I can confirm the bug. It is a bug on threads, a deadlock, 
because of a typo.


Could you check the attached patch to confirm it fixes your problem ?

Thanks,
--
Christopher Faulet
>From 6436e9d934045c7cd9c41bffa036eb6213ff21ee Mon Sep 17 00:00:00 2001
From: Christopher Faulet 
Date: Wed, 24 Jan 2018 21:49:41 +0100
Subject: [PATCH] BUG/MEDIUM: threads/server: Fix deadlock in
 srv_set_stopping/srv_set_admin_flag

Because of a typo (HA_SPIN_LOCK instead of HA_SPIN_UNLOCK), there is a deadlock
in srv_set_stopping and srv_set_admin_flag when there is at least one trackers.

This patch must be backported in 1.8.
---
 src/server.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/server.c b/src/server.c
index 3901e7d8..07a6603a 100644
--- a/src/server.c
+++ b/src/server.c
@@ -976,7 +976,7 @@ void srv_set_stopping(struct server *s, const char *reason, struct check *check)
 	for (srv = s->trackers; srv; srv = srv->tracknext) {
 		HA_SPIN_LOCK(SERVER_LOCK, >lock);
 		srv_set_stopping(srv, NULL, NULL);
-		HA_SPIN_LOCK(SERVER_LOCK, >lock);
+		HA_SPIN_UNLOCK(SERVER_LOCK, >lock);
 	}
 }
 
@@ -1019,7 +1019,7 @@ void srv_set_admin_flag(struct server *s, enum srv_admin mode, const char *cause
 	for (srv = s->trackers; srv; srv = srv->tracknext) {
 		HA_SPIN_LOCK(SERVER_LOCK, >lock);
 		srv_set_admin_flag(srv, mode, cause);
-		HA_SPIN_LOCK(SERVER_LOCK, >lock);
+		HA_SPIN_UNLOCK(SERVER_LOCK, >lock);
 	}
 }
 
-- 
2.14.3

Re: [PATCH] Fix build when compiling without threads traffic

[Willy Tarreau]

On Wed, Jan 24, 2018 at 03:44:25PM +0100, Olivier Houchard wrote:
> Hi,
> 
> Commit 1605c7ae6154d8c2cfcf3b325872b1a7266c5bc2 broke building haproxy
> without threads support. The attached patch should fix it.

Oh, thank you Olivier. I didn't know that HA_SPINLOCK_T wasn't declared
without threads, since the macros that use it are transparent. Then I
can remove the __maybe_unused attribute which I placed there on purpose
to avoid a build warning.

Willy

h2 sending RSTs in response to RSTs

[klzgrad]

Hi,

I patched Chromium to close the stream immediately after seeing END_STREAM.

In testing, Chromium sends an RST (CANCEL) for this, but HAProxy
replies with an RST (STREAM_CLOSED). This is a MUST NOT (though only a
nuisance for me as Chromium will print warnings for it.):

> To avoid looping, an endpoint MUST NOT send a RST_STREAM in response to a 
> RST_STREAM frame.

I put some logging points. This branch is being triggered:

if (h2s->flags & H2_SF_RST_RCVD) {

During this, h2s is h2_closed_stream, and the "closed" stream was
previously deleted from h2_detach.

-klzgrad

Re: mworker: seamless reloads broken since 1.8.1

[Willy Tarreau]

Hi Pierre,

On Wed, Jan 24, 2018 at 03:07:54PM +0100, Pierre Cheynier wrote:
> Willy, I confirm that it works well again running the following version:
> 
> $ haproxy -v
> HA-Proxy version 1.8.3-945f4cf 2018/01/23
> 
> Added nbthread again, reloads are transparents.

Excellent, many thanks for confirming!

Willy

Re: problem in 1.8 with hosts going out of service

[Paul Lockaby]

Sorry, I know this list is super busy and that there are a number of other more 
important issues that need to be worked through but I'm hoping one of the 
maintainers has been able to confirm this bug?

Thanks,
-Paul

> On Jan 17, 2018, at 10:27 AM, Paul Lockaby  wrote:
> 
> Ok I've tracked this problem down specifically to the usage of check tracking.
> 
> That is to say, the backend "example-api" is set to track the backend 
> "example-http". When that tracking is enabled and one of the servers in the 
> backend goes down then all of haproxy goes down and never recovers.
> 
> So this works:
> 
>server myhost myhost.example.com:8445 ssl ca-file 
> /usr/local/ssl/certs/cacerts.cert
> 
> 
> But this does not:
> 
>server myhost myhost.example.com:8445 track example-http/myhost ssl 
> ca-file /usr/local/ssl/certs/cacerts.cert
> 
> 
> This is definitely a regression from 1.7 because I used this feature in 1.7 
> without issue.
> 
> 
>> On Jan 16, 2018, at 10:36 PM, Paul Lockaby  wrote:
>> 
>> I'm experiencing a problem that I can't diagnose but I can recreate pretty 
>> consistently. I have a single server that responds for example.com and 
>> api.example.com and it runs haproxy. All the names run through an SSL front 
>> door but an ACL makes it such that requests for example.com get sent to 8443 
>> where Apache runs and requests for api.example.com get sent to 8445 where 
>> the same instance of haproxy runs and does further examination of the 
>> request and sends it to an application server running on localhost.
>> 
>> This configuration works great except when I take a server out of the 
>> rotation by disabling it with disable-on-404. As soon as I take any server 
>> out of the rotation, haproxy completely stops responding to ANY requests for 
>> ANY backend even things that aren't part of the group such as the stats 
>> backend and frontend. If I put the server back in to service haproxy does 
>> not recover. I must restart haproxy on all hosts to recover. Nothing shows 
>> up in the logs and I can't figure out how to debug it such that I can 
>> provide more information but it's very consistently reproducible using the 
>> configuration below. I am running 1.8.3 and I have not tried this on 1.7 or 
>> earlier versions of 1.8.
>> 
>> Thanks for your help.
>> -Paul
>> 
>> 
>> 
>> global
>>   log /dev/log local0
>>   user nobody
>>   group nobody
>>   tune.ssl.default-dh-param 2048
>>   stats socket /var/run/haproxy.sock user nobody group nobody
>>   daemon
>> 
>> defaults
>>   timeout connect 5000ms
>>   timeout client 60ms
>>   timeout server 60ms
>> 
>>   option httplog
>>   option forwardfor
>>   option http-server-close
>>   option contstats
>> 
>> frontend stats-frontend
>>   bind *:2999
>>   mode http
>>   log global
>>   stats enable
>>   stats uri /haproxy
>> 
>> backend stats-backend
>>   mode http
>>   log global
>>   server stats /var/run/haproxy.sock check
>> 
>> frontend secured
>>   # get the list of certificate options from a list in a file
>>   bind *:443 ssl crt-list /srv/haproxy/certificates.lst
>>   mode http
>>   log global
>> 
>>   # tell backend connections what our ssl client cn is
>>   http-request set-header X-SSL-Client-Verify %[ssl_c_verify]
>>   http-request set-header X-SSL-Client-DN %{+Q}[ssl_c_s_dn]
>>   http-request set-header X-SSL-Client-CN %{+Q}[ssl_c_s_dn(cn)]
>>   http-request set-header X-SSL-Issuer-DN %{+Q}[ssl_c_i_dn]
>>   http-request set-header X-SSL-Issuer-CN %{+Q}[ssl_c_i_dn(cn)]
>> 
>>   acl server-status path_beg /server-
>>   use_backend bogus-http if server-status
>> 
>>   # connection requests for apis go to the api backends
>>   acl request_api hdr_beg(Host) -i api.
>>   use_backend example-api if request_api
>> 
>>   default_backend example-http
>> 
>> backend example-http
>>   mode http
>>   log global
>>   balance source
>>   hash-type consistent
>>   option httpchk GET /haproxy/alive.txt
>>   http-check disable-on-404
>>   server myhost myhost.example.com:8443 check ssl ca-file 
>> /usr/local/ssl/certs/cacerts.cert
>> 
>> backend bogus-http
>>   mode http
>>   errorfile 503 /netops/www/haproxy/403.http
>> 
>> backend example-api
>>   mode http
>>   log global
>>   balance roundrobin
>>   option httpchk GET /haproxy/alive.txt
>>   http-check disable-on-404
>>   server myhost myhost.example.com:8445 track example-http/myhost ssl 
>> ca-file /usr/local/ssl/certs/cacerts.cert
>> 
>> frontend localhost-api-frontend
>>   bind *:8445 ssl crt /usr/local/ssl/certs/example.com.pem
>>   mode http
>>   log global
>>   option forwardfor if-none
>>   option dontlog-normal
>> 
>>   # the alerts api backend
>>   acl alerts-api_host hdr_beg(Host) -i api.alerts
>>   use_backend localhost-api-backend-alerts if alerts-api_host
>> 
>>   default_backend bogus-http
>> 
>> backend localhost-api-backend-alerts
>>   mode http
>>   log global
>>   option forwardfor if-none
>>   option dontlog-normal
>>   server localhost 

How can I map bindings to the correct backend?

[Pieter Vogelaar]

I have the following configuration:


frontend default-tcp
  bind 192.168.52.12:5044
  bind 192.168.52.12:
  bind 192.168.52.12:5556
  bind 192.168.52.12:5672
  bind 192.168.52.13:5672
  mode tcp
  option tcplog
  use_backend %[dst_port,map(/etc/haproxy/tcp-bindings-to-backends.map,default)]


The map file (/etc/haproxy/tcp-bindings-to-backends.map) looks like:


192.168.52.12:5044 logging-logstashmsg-acc
192.168.52.12: logging-logstashmsg-acc
192.168.52.12:5556 logging-logstashmsg-acc
192.168.52.12:5672 logging-rabbitmq-acc
192.168.52.13:5672 stackstorm-rabbitmq-tst


I would like to route based on the IP address and port to the correct backend. 
At the moment I used the dst_port sample on the use_backend line. But that must 
be a concatenation of dst sample, “:” and dst_port sample.
It seems like a basic thing to do, but I just can’t figure out how to do this.

Some help is very much appreciated!


Best regards,
Pieter Vogelaar

Re: tune.h2.initial-window-size not applied to connection windows

[klzgrad]

On Tue, Jan 23, 2018 at 2:13 PM, klzgrad  wrote:
> This tuning knob is probably only useful if it's also applied to the
> overall connection windows in addition to stream windows.

This should be fairly easy to do:

--- a/src/mux_h2.c   2017-12-31 01:13:19.0 +0800
+++ b/src/mux_h2.c  2018-01-24 21:45:41.089380260 +0800
@@ -399,7 +399,7 @@ static int h2c_frt_init(struct connectio
h2c->max_id = -1;
h2c->errcode = H2_ERR_NO_ERROR;
h2c->flags = H2_CF_NONE;
-   h2c->rcvd_c = 0;
+   h2c->rcvd_c = some_initial_connection_window_size -
h2_settings_initial_window_size;
h2c->rcvd_s = 0;
h2c->nb_streams = 0;

The problem is the window size for the connection probably has to use
a different value.

Bad things happen with the current default 65535 window size for both
the connection and streams. One upload stream is able to use up the
connection window size and stall all other streams (at least this is
what Chromium does).

So the connection window size should be at least 2x the stream window
size. Things become more complicated if one considers packet losses of
WINDOW_UPDATEs:

- one WINDOW_UPDATE lost.
- X00 ms later, retransmission of WINDOW_UPDATE.
- During the X00 ms the connection flow-control prevents sending
anything because HTTP/2 says so not because TCP is actually having
problem sending stuff.

Okay, for one lost WINDOW_UPDATE we can add one or two extra round
trips to window size calculation. What about multiple losses?

At this point I wonder what is the point of having HTTP/2
connection-level flow-control on top of TCP connection flow control?
For throttling there's other knobs to use. I would use 2^31-1 for the
connection window size.

-klzgrad

[PATCH] Fix build when compiling without threads traffic

[Olivier Houchard]

Hi,

Commit 1605c7ae6154d8c2cfcf3b325872b1a7266c5bc2 broke building haproxy
without threads support. The attached patch should fix it.

Regards,

Olivier
>From 17e4494874b4a75da039f06f00f668d413038283 Mon Sep 17 00:00:00 2001
From: Olivier Houchard 
Date: Wed, 24 Jan 2018 15:41:04 +0100
Subject: [PATCH] MINOR: threads: Fix build when we're not compiling with
 threads.

Only declare the start_lock if threads are compiled in, otherwise
HA_SPINLOCK_T won't be defined.
This should be backported to 1.8 when/if
1605c7ae6154d8c2cfcf3b325872b1a7266c5bc2 is backported.
---
 src/haproxy.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/haproxy.c b/src/haproxy.c
index 8e0e30d8..b1d77653 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -2422,7 +2422,9 @@ static void *run_thread_poll_loop(void *data)
 {
struct per_thread_init_fct   *ptif;
struct per_thread_deinit_fct *ptdf;
+#ifdef USE_THREAD
static __maybe_unused HA_SPINLOCK_T start_lock;
+#endif
 
tid = *((unsigned int *)data);
tid_bit = (1UL << tid);
-- 
2.14.3

Re: mworker: seamless reloads broken since 1.8.1

[Pierre Cheynier]

On 23/01/2018 19:29, Willy Tarreau wrote:
> Pierre, please give a try to the latest 1.8 branch or the next nightly
> snapshot tomorrow morning. It addresses the aforementionned issue, and
> I hope it's the same you're facing.
>
> Cheers,
> Willy
Willy, I confirm that it works well again running the following version:

$ haproxy -v
HA-Proxy version 1.8.3-945f4cf 2018/01/23

Added nbthread again, reloads are transparents.

Thanks,

Pierre



signature.asc
Description: OpenPGP digital signature

rate-limit sessions not consistent between ssl and non ssl traffic

[Jérôme Magnin]

Hi,

I've been toying with haproxy and rate limiting lately, and noticed an odd
behavior with rate-limit sessions, or maybe I misunderstood how it is supposed
to be used.

I'm using the following config:

global
maxconn 2
log 127.0.0.1 local0
userhaproxy
chroot  /usr/share/haproxy
pidfile /run/haproxy.pid
daemon
stats socket /var/run/haproxy.sock

defaults
mode http

frontend  fe_foo
bind *:1234
bind *:1235 ssl crt /etc/haproxy/www.pem
rate-limit sessions 10
default_backend be_foo

backend be_foo
server s1 127.0.0.1:8001

I'm using ab to send traffic to the frontend.

1/ ab -c 40 -n 100 http://127.0.0.1:1234/

the output of show info shows maxconnrate 10 and maxsessrate 10.
This is coherent with the value I set for rate-limit sessions.

2/ ab -c 40 -n 100 https://127.0.0.1:1235/

the output of show info shows maxconnrate, maxsslrate, maxsessrate and
sslfrontendmaxkeyrate equal 40, 4 times the value for my rate-limit sessions.

Am I doing something wrong here ?

thanks,
Jérôme

Re: h2 DATA padding ignored in flow control

[Willy Tarreau]

Hi,

On Wed, Jan 24, 2018 at 12:13:07PM +0800, klzgrad wrote:
> Hi,
> 
> I think I found a new issue while inspecting the source code.
> 
> In h2_frt_transfer_data(), the paddings are not added to the
> connection and stream flow control counters, h2c->rcvd_c and
> h2c->rcvd_s. But it should, per RFC 7540 6.1:
> 
> > The entire DATA frame payload is included in flow control, including the 
> > Pad Length and Padding fields if present.

You're right, I think I broke it when modifying the DATA frames parser
to handle partial frames recently :-(

Does the attached patch fix the problem for you ?

Please note that it will not address your other issue (lack of accounting
of DATA frames subject to an RST) though, it will require a bit more work
to avoid breaking what currently works.

Thanks,
Willy
diff --git a/src/mux_h2.c b/src/mux_h2.c
index 7bb51ea..4196e75 100644
--- a/src/mux_h2.c
+++ b/src/mux_h2.c
@@ -2821,6 +2821,9 @@ static int h2_frt_transfer_data(struct h2s *h2s, struct 
buffer *buf, int count)
/* here we're done with the frame, all the payload (except padding) was
 * transferred.
 */
+   h2c->rcvd_c += h2c->dpl;
+   h2c->rcvd_s += h2c->dpl;
+   h2c->dpl = 0;
h2c->st0 = H2_CS_FRAME_A; // send the corresponding window update
 
/* don't send it before returning data!

Different health-check URI per server : how would you do that ?

[Pierre Cheynier]

Hi,

We have a use-case in which the health-check URI is depending on the
server-name (be reassured, only the health-check :) ).

It would be something like:

backend be_testmode http[...] option httpchk get /check
HTTP/1.1\r\nHost: test.tld default-server inter 3s fall 3 rise 2  
server srv01 10.2.3.4:31432 check port 10345   server srv02
10.2.3.5:32498 check port 18452

Except that in this case it will use `/check` on any server, I would like it to 
be something like /check/srv01 or
something highly configurable.

How would you do that? LUA, external checks (will probably cause trouble
regarding perfs), anything else? Is it worth thinking about this (pretty
rare) use-case in HAProxy itself?

Thanks in advance,

Pierre



0x6E601DDB.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature