Re: No Private Key found in '/etc/letsencrypt/live/www.mydomain.org/fullchain.pem.key

[Cyril Bonté]

Hi,

Le 07/11/2023 à 12:54, Christoph Kukulies a écrit :
(...) 
Now haproxy fails on my config (which the former version 2.4 I was 
running before, didn't)


This is the line in question:

  bind *:443 ssl crt /etc/haproxy/fullchain.pem crt ssl-skip-self-issued-ca

How do I fix this?

Put crt ssl-skip-self-issued-ca
in a separate line?

Where?


It seems you're not expert in compiling haproxy and you missed some 
options (I guess SSL is not enabled), which may also be a problem for 
you future upgrades.


Why not simply use available packages ?

See https://haproxy.debian.net/ and 
https://www.haproxy.com/blog/how-to-install-haproxy-on-ubuntu


According to your previous information, it would result in:
https://haproxy.debian.net/#distribution=Ubuntu=jammy=2.8

--
Cyril Bonté

Re: Wierd issue with OCSP updating

[Cyril Bonté]

Hi Shawn,

Le 13/07/2023 à 18:48, Shawn Heisey a écrit :
Looks like on my last edit I deleted it and didn't add it to defaults, 
so I was wrong in what I said.  It throws a different error when added 
to defaults:


elyograg@bilbo:~$ sudo haproxy -dD -c -f /etc/haproxy/haproxy.cfg
[NOTICE]   (521883) : haproxy version is 2.8.1
[NOTICE]   (521883) : path to executable is /usr/local/sbin/haproxy
[ALERT]    (521883) : config : parsing [/etc/haproxy/haproxy.cfg:32] : 
unknown keyword 'httpclient.resolvers.prefer' in 'defaults' section
[ALERT]    (521883) : config : Error(s) found in configuration file : 
/etc/haproxy/haproxy.cfg

[ALERT]    (521883) : config : Fatal errors found in configuration.


Because it should be in the global section, not the defaults one ;)

--
Cyril Bonté

Re: [PATCH] DOC: Attempt to fix dconv parsing error for tune.h2.fe.initial-window-size

[Cyril Bonté]

Hi all,

Le 20/06/2023 à 11:57, Amaury Denoyelle a écrit :

On Tue, Jun 13, 2023 at 03:18:19PM +0200, Tim Düsterhus, WoltLab GmbH wrote:

Hi
please find the patch attached.
This email address is not subscribed to the list, please keep it in Cc
when replying.


Thanks Tim, I applied all of your patches.

On a side note, I noticed on the rendered doc output that keywords in
the "See also" section are not highlighted with a link. Maybe because
quotes are missing around.


Indeed, historically, everywhere else in the documentation, keywords are 
surrounded by quotes. The parser is quite lazy and detects only such 
keywords. This also prevents false-positives.


That said, thanks for your efforts to adapt the documentation so that I 
don't have to update the converter code, as I really don't find time to 
maintain it :)


Cyril

some updates around haproxy-dconv

[Cyril Bonté]

Hi all !

Yes, I'm quite far from the mailing list, and I fear I won't be as 
active as before for some more times.


Nevertheless, here are some updates :

* Concerning the syntax for keywords with undelimited arguments 
(containing "..."), I've just left a comment here :

https://github.com/haproxy/haproxy/pull/1888#issuecomment-1280026631

* About unordered chapters, I've also committed a fix.
https://github.com/haproxy/haproxy/issues/996#issuecomment-1280024320

* When it started, https://docs.haproxy.org/ was experimental and we 
decided not to remove docs in https://cbonte.github.io/haproxy-dconv/. 
It seems it is now mature and is the official place for the 
documentation. If you're OK with that, I'll remove the docs on the 
github side to redirect user to haproxy.org.


* I've also noticed that the quick reference on docker hub still 
references https://cbonte.github.io/haproxy-dconv/. I'm not sure about 
who maintains this page but it may be a good idea to update to haproxy.org.

See https://hub.docker.com/_/haproxy

Thanks ;)
Cyril Bonté

Re: [ANNOUNCE] haproxy-2.3.0

[Cyril Bonté]

Hi all !

Le 05/11/2020 à 19:20, Willy Tarreau a écrit :

Hi,

HAProxy 2.3.0 was released on 2020/11/05. It added 33 new commits after
version 2.3-dev9. I was right to wait a few more days before releasing,
we could spot two late regressions and fix them in time!
[...]
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/


I'm sad to not find enough time to contribute to haproxy. But well, at 
least I try to read mail subjects :-/

With some delays, I've now pushed the documentation for 2.3 and 2.4-dev ;-)

--
Cyril Bonté

github template

[Cyril Bonté]

Hi all,
If I remember well, Lukas maintains the github issues tempplate.

Because of lack of time, I'm not able to read all emails since I receive 
github issues notifications. Time to time, I try to read them all but 
the current template makes the operation quite difficult because all 
issues first start with "haproxy -vv" output, hiding the important 
informations.


(Another case is when I try to follow the issue reports during vacation)

I think it could be easier and quicker by only changing the sections 
order like this :

1. Expected behavior
2. Actual behavior
3. Steps to reproduce the behavior
4. Do you have any idea what may have caused this?
5. Do you have an idea how to solve the issue?
6. What's the configuration?
7. Output of haproxy -vv and uname -a

What do you think about it ?

--
Cyril Bonté

Re: Documentation

[Cyril Bonté]

Hi all,

Le 11/07/2020 à 15:37, Lukas Tribus a écrit :

But yes, I'm sure Cyril will publish the 2.3-dev documentation
shortly, then the links on haproxy.org will work.


Indeed, 2.3-dev appeared while I was not available. I just had time to 
update the 2.2 documentation in a hurry.


Now, all versions are available, including the dev branch ;-)

--
Cyril Bonté

Re: [2.1.1] http-request replace-uri does not work

[Cyril Bonté]

Hi Willy,

Le 16/12/2019 à 22:06, Artur a écrit :

[...]
URLs like https://q.d/PPDSlide/testfile are correctly rewritten to
https://q.d/p3/PPDSlide/testfile and forwarded to the backend.

Once I switched to 2.1.1, haproxy no longer rewrites the URI and the
URIs remains unchanged while forwarded to the backend. I had to
downgrade to have the usual behaviour.

Is it a bug or something changed in normal haproxy behaviour with 2.1
release ?


I can confirm the issue.

It seems to happen with h2 requests only, since commit #30ee1efe67.
haproxy normalizes the URI but replace-uri doesn't take into account
this information. The fix should be easy for replace-uri (If someone
wants to work on it, I won't have time this week).

http://git.haproxy.org/?p=haproxy-2.1.git;a=commit;h=30ee1efe67


I'm not 100% sure it's the right approach to fix the issue. Can you 
check if this patch may fix the issue in all conditions ?


From what I've observed, it seems we can rely on the 
HTX_SL_F_NORMALIZED_URI flag to detect if the URI was normalized by 
haproxy and in that case, we should start at the path instead of the URI.


--
Cyril Bonté
diff --git a/src/http_act.c b/src/http_act.c
index d6015d362..797d75275 100644
--- a/src/http_act.c
+++ b/src/http_act.c
@@ -141,6 +141,7 @@ static enum act_return http_action_replace_uri(struct act_rule *rule, struct pro
 {
 	enum act_return ret = ACT_RET_ERR;
 	struct buffer *replace, *output;
+	struct htx_sl *sl;
 	struct ist uri;
 	int len;
 
@@ -148,7 +149,12 @@ static enum act_return http_action_replace_uri(struct act_rule *rule, struct pro
 	output  = alloc_trash_chunk();
 	if (!replace || !output)
 		goto leave;
-	uri = htx_sl_req_uri(http_get_stline(htxbuf(>req.buf)));
+
+	sl = http_get_stline(htxbuf(>req.buf));
+	uri = htx_sl_req_uri(sl);
+	if (sl->flags & HTX_SL_F_NORMALIZED_URI)
+		uri = http_get_path(uri);
+
 	if (!regex_exec_match2(rule->arg.act.p[1], uri.ptr, uri.len, MAX_MATCH, pmatch, 0))
 		goto leave;
 

Re: [2.1.1] http-request replace-uri does not work

[Cyril Bonté]

Hi Artur,

Le 16/12/2019 à 19:06, Artur a écrit :

Hello,

This is an extract of my frontend configuration working perfectly on 2.0.11.

frontend wwws
     bind 0.0.0.0:443 ssl crt /etc/haproxy/ssl/server.pem alpn
h2,http/1.1
     mode http
     acl is_dev_qd hdr(host) -i dev.q.d dev.qs.d
     acl is_qd hdr(host) -i q.d qs.d www.q.d www.qs.d
     http-request replace-uri ^/PPDSlide/(.*) /d3/PPDSlide/\1 if
is_dev_qd
     http-request replace-uri ^/PPDSlide/(.*) /p3/PPDSlide/\1 if is_qd
     

URLs like https://q.d/PPDSlide/testfile are correctly rewritten to
https://q.d/p3/PPDSlide/testfile and forwarded to the backend.

Once I switched to 2.1.1, haproxy no longer rewrites the URI and the
URIs remains unchanged while forwarded to the backend. I had to
downgrade to have the usual behaviour.

Is it a bug or something changed in normal haproxy behaviour with 2.1
release ?


I can confirm the issue.

It seems to happen with h2 requests only, since commit #30ee1efe67.
haproxy normalizes the URI but replace-uri doesn't take into account 
this information. The fix should be easy for replace-uri (If someone 
wants to work on it, I won't have time this week).


http://git.haproxy.org/?p=haproxy-2.1.git;a=commit;h=30ee1efe67


--
Cyril Bonté

[PATCH] doc: fix date and http_date keywords syntax

[Cyril Bonté]

---
 doc/configuration.txt | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index d8fe6b650..0bb7313cd 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -13245,7 +13245,7 @@ hex2i
   Converts a hex string containing two hex digits per input byte to an
   integer. If the input value cannot be converted, then zero is returned.
 
-http_date([])
+http_date([])
   Converts an integer supposed to contain a date since epoch to a string
   representing this date in a format suitable for use in HTTP header fields. If
   an offset value is specified, then it is added to the date before the
@@ -14066,7 +14066,7 @@ cpu_ns_tot : integer
   high cpu_calls count, for example when processing many HTTP chunks, and for
   this reason it is often preferred to log cpu_ns_avg instead.
 
-date([, ]) : integer
+date([],[]) : integer
   Returns the current date as the epoch (number of seconds since 01/01/1970).
 
   If an offset value is specified, then it is added to the current date before
-- 
2.24.0

Re: Timeout defaults?

[Cyril Bonté]

Le 26/10/2019 à 16:53, Troels Arvin a écrit :

Hello,

Cyril Bonté wrote:

if no "timeout tunnel" is specified in the configuration, there is
no timeout for that specific case. "timeout client" and "timeout server"
take the relay, if they're defined as well.


OK, so if we have the following timeout values, then "timeout tunnel"
becomes what?

timeout connect 10s
timeout client  13s
timeout server  37s


As already said, with this configuration there is NO tunnel timeout, you 
have a CLIENT timeout and a SERVER timeout. Those timeouts are well 
documented to understand what will happen :
- if haproxy is waiting for data from the client, "timeout client" will 
apply.
- if haproxy is waiting for data from the server, "timeout server" will 
apply.



--
Cyril Bonté

Re: Timeout defaults?

[Cyril Bonté]

Le 26/10/2019 à 09:24, Troels Arvin a écrit :

Hello,

Cyril Bonté wrote:

You will find the information at several places :

[...]

No matter how hard I look, I can't find an answer to this:

Let's say my defaults section has the following three timeout related
lines:

 timeout connect 10s
 timeout client  10s
 timeout server  10s

What value for "timeout tunnel" will HAproxy 1.5 choose, given that it's
not specified in haproxy.cfg? - I mean it has to choose some value for
"timeout tunnel", right?


No, if no "timeout tunnel" is specified in the configuration, there is 
no timeout for that specific case. "timeout client" and "timeout server" 
take the relay, if they're defined as well.


haproxy doesn't set default timeouts unless they are defined in the 
confiugration.


--
Cyril Bonté

Re: Timeout defaults?

[Cyril Bonté]

Hi,

Le 25/10/2019 à 22:28, Troels Arvin a écrit :

I'm trying to find out what the default values are for the following
parameters for HAproxy 1.5:

timeout client
timeout server
timeout tunnel

Where can I find the values? -- I don't seem to be able to find them in
the documentation at
https://cbonte.github.io/haproxy-dconv/1.5/configuration.html


You will find the information at several places :
- in the documentation : "An unspecified timeout results in an infinite 
timeout"

https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4.2-timeout%20client
https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4.2-timeout%20server
https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4.2-timeout%20connect

- in the warnings logged by the haproxy process :
[WARNING] 298/010006 (813) : config : missing timeouts for proxy ''.
   | While not properly invalid, you will certainly encounter various 
problems
   | with such a configuration. To fix this, please ensure that all 
following

   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.

--
Cyril Bonté

Re: haproxy doesn't bring failed server up

[Cyril Bonté]

Hi,

Le 06/10/2019 à 09:19, rihad a écrit :
Hi, all. This annoying bug can be experienced in 1.7-2.0 servers (while 
1.9 has added another bug of high CPU utilization - unrelated to this). 
In essence, once an external server that we forward internal requests to 
stops responding for some time and comes back to life a bit later, more 
often than not haproxy can no longer reach it.


Without any other details (logs would be helpul), I tend to think 
there's no bug here, but a configuration issue.


By default, haproxy resolves host names once, on start up. As you are 
using a host name to delcare the smtp server, which oftenly updates its 
IPs :



Configuration is very simple

[...]
     server amazon email-smtp.us-west-2.amazonaws.com:587 check inter 
30s fall 1440 rise 1




That's it. if email-smtp.us-west-2.amazonaws.com:587 fails 
intermittently and the downtime lasts more than a few 30 sec checks, it 
can then no longer be accessed via 127.0.0.2:2588 even if the external 
servers resumes normal operation, and nothing short of a reload (-sf) 
fixes the problem.


I guess that when it happens, email-smtp.us-west-2.amazonaws.com has a 
new pool of IP addresses and haproxy can't connect to the old resolved 
one. You should have a look to the documentation chapter "Server IP 
address resolution using DNS" :

https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#5.3

--
Cyril Bonté

Re: haproxy 2.0.x: another case of high CPU consumption

[Cyril Bonté]

Hi again,

Le 13/08/2019 à 11:57, Willy Tarreau a écrit :

On Tue, Aug 13, 2019 at 11:49:20AM +0200, Willy Tarreau wrote:

Excellent, I think you caught it! I can reproduce it here, except that
it doesn't last long, as soon as I get the socket error it's done. So
we indeed broke something in the connection setup.


Bug kindly brought to you by your benevolent dictator :

   commit d58f27feadbc71c947fa0810f49552a94c60dc9a (refs/bisect/bad)
   Author: Willy Tarreau 
   Date:   Mon Jun 3 10:12:22 2019 +0200

 MINOR: mux-h1: don't try to recv() before the connection is ready
 
 Just as we already do in h1_send(), if the connection is not yet ready,

 do not proceed and instead subscribe. This avoids a needless recvfrom()
 and subscription to polling for a case which will never work since the
 request was not even sent.

I think the subscription prevents us from performing a synchronous
operation somewhere, I'll check.


And I confirm that commit 5b06cb33eb (BUG/MEDIUM: mux_h1: Don't bother 
subscribing in recv if we're not connected) fixes the issue ;-)



--
Cyril Bonté

haproxy 2.0.x: another case of high CPU consumption

[Cyril Bonté]

Hi Willy,

This morning, I found another case where haproxy 2.0 abnormally consumes 
the cpu. It occured on my laptop when my network was not setup yet.


I could find a simple reproducer :
  listen test
mode http
bind :
server s1 10.0.0.1:

then, curl http://localhost:/

10.0.0.1 is unreachable and the connection stays in SYN_SENT, but 
haproxy progressively raises to 100% CPU even after stopping curl.

running a second curl will raise the CPU up to 200%, etc...

I think I'll have time to investigate tonight but maybe you'll find the 
issue and the fix before me ;)


It looks to be very specific to the 2.0 branch, I can't reproduce it in 
1.9 and 2.1.



--
Cyril Bonté

Re: [ANNOUNCE] haproxy-2.0.0

[Cyril Bonté]

Le 16/06/2019 à 21:56, Willy Tarreau a écrit :

Hi,

HAProxy 2.0.0 was released on 2019/06/16. It added 63 new commits
after version 2.0-dev7.
[...]

Please find the usual URLs below :
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/


And with more than 24h later, I've deployed the HTML documentation for 
2.0.0 and 2.1-dev. I really need to clean up/repair/rewrite my scripts ;)



--
Cyril Bonté

Re: segfault using cache with 1.9.4

[Cyril Bonté]

Le 10/04/2019 à 22:52, Lukas Tribus a écrit :

On Wed, 10 Apr 2019 at 17:07, Juan Pablo Mora
 wrote:

 acl is_static url_beg  /lgt/lgtfrontend/library/ or 
/lgt/lgtfrontend/pdfjs/ or /lgt/lgtfrontend/img/


That's not the correct syntax. That would be:

acl is_static url_beg /lgt/lgtfrontend/library/
acl is_static url_beg /lgt/lgtfrontend/pdfjs/
acl is_static url_beg /lgt/lgtfrontend/img/


And that's not the only error, url_beg is not useable in http-response :

http-response cache-store frontendcache if is_stati

It should not crash, of course.


At least, we need a (more) complete configuration to investigate 
further, as the lines provided are not valid and are not sufficient to 
reproduce the issue.



--
Cyril Bonté

[DOC] Clarification of the "abbreviated form with all-0-octets omitted" for IPv4 addresses needed

[Cyril Bonté]

Hi Matous,


Hello,

my collegues and me were highly surprised with the "abbreviated form with 
all-0-octets ommitted" [1] for IPv4 addresses. It would be good to provide better 
(or rather exact) technical explanation why it is so - either in the documentation or at 
least here in the mailing-list. Any sort of clarification would be highly appreciated for 
this is *very* uncommon and we think it is worth the effort to provide some. We can 
easily live with
it but would like to know the exact reason why it is so...  Thank you.


[1]: http://cbonte.github.io/haproxy-dconv/1.9/configuration.html#7.1.6


Well, for that kind of notation, you can refer to the inet manpage, 
where everything is explained ;-)

https://linux.die.net/man/3/inet

(I hope the reply will arrive in the right thread cause I didn't receive 
the email but found the question in mail-archive instead).


--
Cyril Bonté

Re: Wrong sha256 checksum for HAProxy 1.8 and 1.9?

[Cyril Bonté]

> De: "Tim Düsterhus" 
> À: "Cyril Bonté" , "Willy Tarreau" , "Kevin 
> Mao" 
> Cc: haproxy@formilux.org
> Envoyé: Mardi 26 Février 2019 12:12:33
> Objet: Re: Wrong sha256 checksum for HAProxy 1.8 and 1.9?
> 
> Willy,
> Cyril,
> 
> Am 26.02.19 um 10:52 schrieb Cyril Bonté:
> > Interesting, in fact the downloaded file is a gzip of the tar.gz
> > itself.
> > 
> 
> Yes. This appears to be a misconfiguration in either HAProxy or
> Apache.
> Probably Apache, because the `.tar.gz` is delivered with Content-Type
> application/x-tar which I suspect causes HAProxy to compress it once
> again:

Well, this is more a browser bug than a misconfiguration. But for now, the 
configuration requires a workaround to not use "Content-Encoding: gzip" for 
gzipped files.
https://bugzilla.mozilla.org/show_bug.cgi?id=610679
https://bugzilla.mozilla.org/show_bug.cgi?id=902503

Cyril

Re: Wrong sha256 checksum for HAProxy 1.8 and 1.9?

[Cyril Bonté]

Hi again,

> De: "Cyril Bonté" 
> À: "Willy Tarreau" , "Kevin Mao" 
> Cc: haproxy@formilux.org
> Envoyé: Mardi 26 Février 2019 09:39:08
> Objet: Re: Wrong sha256 checksum for HAProxy 1.8 and 1.9?
> 
> Using firefox, I'm observing the same issue.
> Most of the time, the downloaded file is missing the final 1308 bytes
> for haproxy-1.9.4.tar.gz. I've tried with http/https, http2
> enabled/disabled, the file is always corrupted.
> Similar truncated results occurs with the other archives.
> 
> I don't have the issue using wget/curl/chromium. I'll make some more
> tests today.

Interesting, in fact the downloaded file is a gzip of the tar.gz itself.

Cyril

Re: Wrong sha256 checksum for HAProxy 1.8 and 1.9?

[Cyril Bonté]

Hi Willy,

Le 26/02/2019 à 07:38, Willy Tarreau a écrit :

Hi Kevin,

On Tue, Feb 26, 2019 at 06:27:30AM +, Kevin Mao wrote:

Hi haproxy@,

It seems like the sha256 checksum's are wrong for the latest 1.8 and 1.9 
HAProxy versions, Can you please confirm?
https://www.haproxy.org/download/1.9/src/,
$  shasum -a 256 -c haproxy-1.9.4.tar.gz.sha256haproxy-1.9.4.tar.gz: 
FAILEDshasum: WARNING: 1 computed checksum did NOT match
$  shasum -a 256 -c haproxy-1.9.3.tar.gz.sha256haproxy-1.9.3.tar.gz: 
FAILEDshasum: WARNING: 1 computed checksum did NOT match
$  shasum -a 256 -c haproxy-1.8.19.tar.gz.sha256haproxy-1.8.19.tar.gz: 
FAILEDshasum: WARNING: 1 computed checksum did NOT match
$  shasum -a 256 -c haproxy-1.8.18.tar.gz.sha256haproxy-1.8.18.tar.gz: 
FAILEDshasum: WARNING: 1 computed checksum did NOT match


It's not what I'm seeing here using the sha256sum utility :

   $ cat haproxy-1.9.4.tar.gz.sha256
   8483fe12b30256f83d542b3f699e165d8f71bf2dfac8b16bb53716abce4ba74f  
haproxy-1.9.4.tar.gz

   $ sha256sum haproxy-1.9.4.tar.gz
   8483fe12b30256f83d542b3f699e165d8f71bf2dfac8b16bb53716abce4ba74f  
haproxy-1.9.4.tar.gz

   $ sha256sum -c haproxy-1.9.4.tar.gz.sha256
   haproxy-1.9.4.tar.gz: OK

OpenSSL agrees with it :
   $ openssl sha256 haproxy-1.9.4.tar.gz
   SHA256(haproxy-1.9.4.tar.gz)= 
8483fe12b30256f83d542b3f699e165d8f71bf2dfac8b16bb53716abce4ba74f

So I suspect that you're facing a download issue or your shasum fails
for whatever reason. The file above is 2357935 bytes long here. Please
check that matches on your side.


Using firefox, I'm observing the same issue.
Most of the time, the downloaded file is missing the final 1308 bytes 
for haproxy-1.9.4.tar.gz. I've tried with http/https, http2 
enabled/disabled, the file is always corrupted.

Similar truncated results occurs with the other archives.

I don't have the issue using wget/curl/chromium. I'll make some more 
tests today.



--
Cyril Bonté

Re: [PATCH] REG-TEST: mailers: add new test for 'mailers' section

[Cyril Bonté]

Hi all,

Le 08/01/2019 à 10:06, Willy Tarreau a écrit :

On Tue, Jan 08, 2019 at 09:31:22AM +0100, Frederic Lecaille wrote:

Indeed this script could worked with a short mailer timeout before af4021e6
commit. Another git bisect shows that 53216e7d introduced the email bombing
issue.

Note that 33a09a5f refers to 53216e7d commit.

I am not sure this can help.


Sure it helps, at least to know whom to ping :-)


Well, from what I've seen with a small test, I've a different conclusion 
about the commit which introduced the issue. It looks to have been 
introduced earlier with commit 0108bb3e4 "MEDIUM: mailers: Init alerts 
during conf parsing and refactor their processing"

http://git.haproxy.org/?p=haproxy.git;a=commit;h=0108bb3e4

Hope this helps.

--
Cyril Bonté

Re: Http HealthCheck Issue

[Cyril Bonté]

> De: "PRAVEEN UPPALAPATI" 
> À: "Cyril Bonté" 
> Cc: haproxy@formilux.org
> Envoyé: Mercredi 19 Décembre 2018 18:10:50
> Objet: RE: Http HealthCheck Issue
> 
> The pointed mistake was GET which I updated and tried. I'm not sure
> if there was something else pointed.

Then please read again the answers, talking about the missing Host header in 
your forged request.

> 
> Thanks,
> Praveen.
> 
> -Original Message-
> From: Cyril Bonté [mailto:cyril.bo...@free.fr]
> Sent: Wednesday, December 19, 2018 9:28 AM
> To: UPPALAPATI, PRAVEEN 
> Cc: haproxy@formilux.org
> Subject: Re: Http HealthCheck Issue
> 
> please, please, PLEASE ! Read the answers before telling again it's
> not working.
> Several users already told you where you made a mistake.
> 
> 
> - Mail original -
> > De: "PRAVEEN UPPALAPATI" 
> > À: "Jarno Huuskonen" 
> > Cc: haproxy@formilux.org, "THANIGAIVEL SIVANANDHAM"
> > 
> > Envoyé: Mercredi 19 Décembre 2018 15:55:34
> > Objet: RE: Http HealthCheck Issue
> > 
> > Hi Jarno,
> > 
> > Even after updating to GET it is still failing:
> > 
> > [Dec 19 06:12:41]  Health check for server
> > bk_8093_read/primary8093r
> > failed, reason: Layer7 timeout, check duration: 2001ms, status: 0/2
> > DOWN.
> > [Dec 19 06:12:44]  Health check for server
> > bk_8093_read/primary8093r
> > failed, reason: Layer7 wrong status, code: 400, info: "No Host",
> > check duration: 769ms, status: 0/2 DOWN.
> > [Dec 19 06:49:30]  Health check for server bk_8089/primary8089
> > succeeded, reason: Layer4 check passed, check duration: 0ms,
> > status:
> > 3/3 UP.
> > [Dec 19 06:49:31]  Health check for server
> > bk_5100_read/primary5100r
> > failed, reason: Layer7 wrong status, code: 400, info: "No Host",
> > check duration: 379ms, status: 0/2 DOWN.
> > [Dec 19 06:49:31]  Health check for backup server
> > bk_5100_read/backUp05100r failed, reason: Layer7 wrong status,
> > code:
> > 400, info: "No Host", check duration: 105ms, status: 0/2 DOWN.
> > [Dec 19 06:51:32]  Health check for server bk_8089/primary8089
> > succeeded, reason: Layer4 check passed, check duration: 0ms,
> > status:
> > 3/3 UP.
> > [Dec 19 06:51:32]  Health check for server
> > bk_8093_read/primary8093r
> > failed, reason: Layer7 wrong status, code: 400, info: "No Host",
> > check duration: 124ms, status: 0/2 DOWN.
> > [Dec 19 06:51:33]  Health check for backup server
> > bk_8093_read/backUp08093r failed, reason: Layer7 wrong status,
> > code:
> > 400, info: "No Host", check duration: 1ms, status: 0/2 DOWN.
> > [Dec 19 06:51:33]  Health check for backup server
> > bk_8093_read/backUp18093r failed, reason: Layer4 connection
> > problem,
> > info: "Connection refused", check duration: 63ms, status: 0/2 DOWN.
> > [Dec 19 06:51:33]  Health check for server
> > bk_8093_write/primary8093w
> > failed, reason: Layer7 invalid response, info: "<15><03><03>",
> > check
> > duration: 128ms, status: 0/2 DOWN.
> > [Dec 19 06:51:34]  Health check for server
> > bk_5100_read/primary5100r
> > failed, reason: Layer7 wrong status, code: 400, info: "No Host",
> > check duration: 269ms, status: 0/2 DOWN.
> > [Dec 19 06:51:34]  Health check for backup server
> > bk_5100_read/backUp05100r failed, reason: Layer7 wrong status,
> > code:
> > 400, info: "No Host", check duration: 20ms, status: 0/2 DOWN.
> > [haproxy@zld05596 etc]$
> > 
> > 
> > backend bk_8093_read
> > balancesource
> > http-response set-header X-Server %s
> > option log-health-checks
> > option httpchk GET
> > /nexus/v1/repository/rawcentral/com.att.swm.attpublic/healthcheck.txt
> > HTTP/1.1\r\nAuthorization:\ Basic\ 
> > server primary8093r :8093 check verify none
> > server backUp08093r :8093 check backup verify none
> > server backUp18093r :8093 check backup verify none
> > 
> > also wanted to find out if the same option httpchk will work for
> > https?
> > 
> > frontend http-5100
> > bind *:5100 ssl crt .pem
> > option httplog
> > capture request header Host len 24
> > acl is_get  method GET
> > use_backend bk_5100_read  if is_get
> > 
> > 
> > 
> > backend bk_5100_read
> > balancesource
> > http-response set-header X-Server %s
> > option log-health-checks
>

Re: Http HealthCheck Issue

[Cyril Bonté]

please, please, PLEASE ! Read the answers before telling again it's not working.
Several users already told you where you made a mistake.


- Mail original -
> De: "PRAVEEN UPPALAPATI" 
> À: "Jarno Huuskonen" 
> Cc: haproxy@formilux.org, "THANIGAIVEL SIVANANDHAM" 
> Envoyé: Mercredi 19 Décembre 2018 15:55:34
> Objet: RE: Http HealthCheck Issue
> 
> Hi Jarno,
> 
> Even after updating to GET it is still failing:
> 
> [Dec 19 06:12:41]  Health check for server bk_8093_read/primary8093r
> failed, reason: Layer7 timeout, check duration: 2001ms, status: 0/2
> DOWN.
> [Dec 19 06:12:44]  Health check for server bk_8093_read/primary8093r
> failed, reason: Layer7 wrong status, code: 400, info: "No Host",
> check duration: 769ms, status: 0/2 DOWN.
> [Dec 19 06:49:30]  Health check for server bk_8089/primary8089
> succeeded, reason: Layer4 check passed, check duration: 0ms, status:
> 3/3 UP.
> [Dec 19 06:49:31]  Health check for server bk_5100_read/primary5100r
> failed, reason: Layer7 wrong status, code: 400, info: "No Host",
> check duration: 379ms, status: 0/2 DOWN.
> [Dec 19 06:49:31]  Health check for backup server
> bk_5100_read/backUp05100r failed, reason: Layer7 wrong status, code:
> 400, info: "No Host", check duration: 105ms, status: 0/2 DOWN.
> [Dec 19 06:51:32]  Health check for server bk_8089/primary8089
> succeeded, reason: Layer4 check passed, check duration: 0ms, status:
> 3/3 UP.
> [Dec 19 06:51:32]  Health check for server bk_8093_read/primary8093r
> failed, reason: Layer7 wrong status, code: 400, info: "No Host",
> check duration: 124ms, status: 0/2 DOWN.
> [Dec 19 06:51:33]  Health check for backup server
> bk_8093_read/backUp08093r failed, reason: Layer7 wrong status, code:
> 400, info: "No Host", check duration: 1ms, status: 0/2 DOWN.
> [Dec 19 06:51:33]  Health check for backup server
> bk_8093_read/backUp18093r failed, reason: Layer4 connection problem,
> info: "Connection refused", check duration: 63ms, status: 0/2 DOWN.
> [Dec 19 06:51:33]  Health check for server bk_8093_write/primary8093w
> failed, reason: Layer7 invalid response, info: "<15><03><03>", check
> duration: 128ms, status: 0/2 DOWN.
> [Dec 19 06:51:34]  Health check for server bk_5100_read/primary5100r
> failed, reason: Layer7 wrong status, code: 400, info: "No Host",
> check duration: 269ms, status: 0/2 DOWN.
> [Dec 19 06:51:34]  Health check for backup server
> bk_5100_read/backUp05100r failed, reason: Layer7 wrong status, code:
> 400, info: "No Host", check duration: 20ms, status: 0/2 DOWN.
> [haproxy@zld05596 etc]$
> 
> 
> backend bk_8093_read
> balancesource
> http-response set-header X-Server %s
> option log-health-checks
> option httpchk GET
> /nexus/v1/repository/rawcentral/com.att.swm.attpublic/healthcheck.txt
> HTTP/1.1\r\nAuthorization:\ Basic\ 
> server primary8093r :8093 check verify none
> server backUp08093r :8093 check backup verify none
> server backUp18093r :8093 check backup verify none
> 
> also wanted to find out if the same option httpchk will work for
> https?
> 
> frontend http-5100
> bind *:5100 ssl crt .pem
> option httplog
> capture request header Host len 24
> acl is_get  method GET
> use_backend bk_5100_read  if is_get
> 
> 
> 
> backend bk_5100_read
> balancesource
> http-response set-header X-Server %s
> option log-health-checks
> option httpchk GET /v1/_ping HTTP/1.1\r\nAuthorization:\ Basic\
> 
> server primary5100r :5100 ssl check verify none
> server backUp05100r :5100 ssl check backup verify none
> 
> Thanks,
> Praveen.
> -Original Message-
> From: Jarno Huuskonen [mailto:jarno.huusko...@uef.fi]
> Sent: Wednesday, December 19, 2018 1:05 AM
> To: UPPALAPATI, PRAVEEN 
> Cc: haproxy@formilux.org; SIVANANDHAM, THANIGAIVEL 
> Subject: Re: Http HealthCheck Issue
> 
> Hi,
> 
> On Tue, Dec 18, UPPALAPATI, PRAVEEN wrote:
> > My backend config is:
> > 
> > backend bk_8093_read
> > balancesource
> > http-response set-header X-Server %s
> > option log-health-checks
> > option httpchk get
> > /nexus/v1/repository/rawcentral/com.att.swm.attpublic/healthcheck.txt
> > HTTP/1.1\r\nAuthorization:\ Basic\ 
> 
> Change get to GET, at least apache, ngingx and tomcat expect GET not
> get.
> Or test with for example netcat that your server1 accepts get.
> 
> Something like: nc server1.add.re.ss 8093
> get
> /nexus/v1/repository/rawcentral/com.att.swm.attpublic/healthcheck.txt
> HTTP/1.1
> Host: ...
> Authorization: Basic ...
> 
> > server primary8093r :8093 check verify none
> > server backUp08093r ::8093 check backup verify none
> > server backUp18093r ::8093 check backup verify none
> > 
> > Output of log:
> > 
> > [Dec 18 05:22:51]  Health check for server
> > bk_8093_read/primary8093r failed, reason: Layer7 wrong status,
> > code: 400, info: "No Host", check duration: 543ms, status: 0/2
> > DOWN.
> 
> Like Jonathan said "No Host" is telling you what's wrong.
> 

Re: [ANNOUNCE] haproxy-1.9-dev11

[Cyril Bonté]

Hi Willy,

Le 17/12/2018 à 14:13, Arnall a écrit :

Le 16/12/2018 à 23:05, Willy Tarreau a écrit :

I expected to release this week-end after running it on the haproxy.org
servers, but some annoying issues faced in production took some time to
get fixed and delayed the release.

Things have been quiet now, with 18 hours running without a glitch in
legacy mode (without HTX) and now 13 hours an counting with HTX enabled,
so things are getting much better.


Hello Willy,

don't know if it's related but haproxy.org answers with 400 status right 
now !


(Windows 10 Chrome/Firefox)


I think I have met the exact same issue when I enabled HTX on a small 
test config.


There's a bug with cookies and HTX.
When haproxy sends the headers to the backend server, it leaks 6 more 
bytes at the end of the cookie value, which is the length of the name 
"cookie".


This is due to this part of code :
http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/h2.c;h=1b784fd4aba2dbad91db156612243e93aa34e5f3;hb=HEAD#l533

http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/h2.c;h=1b784fd4aba2dbad91db156612243e93aa34e5f3;hb=HEAD#l557
htx_set_blk_value_len(blk, bs + 2 + vl);

Here, we set the new block value length, but bs is not the value length, 
it the whole block size :

http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/h2.c;h=1b784fd4aba2dbad91db156612243e93aa34e5f3;hb=HEAD#l542
 542blk = htx_add_header(htx, ist("cookie"), list[ck].v);
 543if (!blk)
 544  goto fail;
 545
 546fs = htx_free_data_space(htx);
 547bs = htx_get_blksz(blk);

I'm not sure how to fix this properly. I'd say we can store list[ck].v 
in a "bvl" variable, update this variable while looping on the cookie 
values, then set the new block value length according to this result.
Or is there a function to explicitely update the block size (kind of 
htx_set_blksz) ?


I prefer to report the bug quickly so you can fix it today. Or I can 
work on a patch, but this will not be before the end of the day ;-)


--
Cyril Bonté

Re: 'stick': unknown fetch method 'res.cook_beg'

[Cyril Bonté]

Hi,

Le 01/11/2018 à 01:33, Gibson, Brian (IMS) a écrit :

Thanks for the response James, I’ll give that a shot.

If that is indeed the case though, it seems that the documentation in 
section 7.3.6 should be reviewed.  That’s where I got the syntax I was 
using.


Well, the documentation is still valid but the syntax you used is only 
valid for acls. This is true for all fetches preceded by "ACL derivatives".


--
Cyril Bonté

Re: [PATCH 1/2] DOC: split the http-request actions in their own section

[Cyril Bonté]

Hi Willy,

Le 17/10/2018 à 04:19, Willy Tarreau a écrit :

Hi Cyril,

On Wed, Oct 17, 2018 at 12:18:49AM +0200, Cyril Bonté wrote:

Hi Willy,

this set of patches is an attempt to make the http-request/http-response
documentation more readable.
If it's OK for you, I will do the same work for tcp-request/tcp-response.


Oh that's an excellent idea, thanks for taking care of this! I'm just
wondering if we shouldn't have an "actions" section and have a compatibility
matrix between the keywords and the rule sets, given that some actions are
common between some of them. This would remove some annoying copy-paste.

For example we could have for each keyword something like this :

   track-sc 
 Supported in: tcp-request connection, tcp-request content, http-request

 Description:
   blah

I don't know what's the best way to do this, but I wanted to suggest it in
case it helps you build up an idea on it.

Regardless of this point, just let me know if you prefer that we first
merge your current patch and that we apply potential extra changes later,
or if you prefer to experiment differently. I'm open to any option.


Oops, the answer delay was not expected. Each day I said to myself "OK, 
it's late, I'll absolutely reply tomorrow" :-)


Having a new section would be a good target for this changes. But it 
will take some time : I think I'll have to adapt my doc converter to 
recognize those actions as "children" keywords.


If you're OK, I'd prefer to apply those first patches and to update the 
doc step by step. This will also avoid conflicts with doc updates from 
other users while I take monthes to upgrade the doc converter :-)


--
Cyril Bonté

Re: [PATCH 1/2] DOC: split the http-request actions in their own section

[Cyril Bonté]

Hi Willy,

this set of patches is an attempt to make the http-request/http-response 
documentation more readable.

If it's OK for you, I will do the same work for tcp-request/tcp-response.

Cheers,

Le 17/10/2018 à 00:14, Cyril Bonté a écrit :

Since http-request was first introduced, more and more actions have been
added over time. This makes the "http-request" difficult to read and some
actions were forgotten in the list.


--
Cyril Bonté

[PATCH 2/2] DOC: split the http-response actions in their own section

[Cyril Bonté]

Similarly to the "http-request" actions, this is an attempt to make the
documentation easier to read.
---
 doc/configuration.txt | 543 ++
 1 file changed, 290 insertions(+), 253 deletions(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index f8f9240a..1b2cf1d2 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -4499,6 +4499,7 @@ http-request wait-for-handshake [ { if | unless } 
 ]
   sure they are valid.
 
 
+http-response   [ { if | unless }  ]
   Access control for Layer 7 responses
 
   May be used in sections:   defaults | frontend | listen | backend
@@ -4511,291 +4512,327 @@ http-request wait-for-handshake [ { if | unless } 
 ]
   if the condition is true. Since these rules apply on responses, the backend
   rules are applied first, followed by the frontend's rules.
 
-  The first keyword is the rule's action. Currently supported actions include :
-- "allow" : this stops the evaluation of the rules and lets the response
-  pass the check. No further "http-response" rules are evaluated for the
-  current section.
-
-- "deny" : this stops the evaluation of the rules and immediately rejects
-  the response and emits an HTTP 502 error. No further "http-response"
-  rules are evaluated.
-
-- "add-header" appends an HTTP header field whose name is specified in
-   and whose value is defined by  which follows the log-format
-  rules (see Custom Log Format in section 8.2.4). This may be used to send
-  a cookie to a client for example, or to pass some internal information.
-  This rule is not final, so it is possible to add other similar rules.
-  Note that header addition is performed immediately, so one rule might
-  reuse the resulting header from a previous rule.
-
-- "set-header" does the same as "add-header" except that the header name
-  is first removed if it existed. This is useful when passing security
-  information to the server, where the header must not be manipulated by
-  external users.
-
-- "del-header" removes all HTTP header fields whose name is specified in
-  .
-
-- "replace-header" matches the regular expression in all occurrences of
-  header field  according to , and replaces them with
-  the  argument. Format characters are allowed in replace-fmt
-  and work like in  arguments in "add-header". The match is only
-  case-sensitive. It is important to understand that this action only
-  considers whole header lines, regardless of the number of values they
-  may contain. This usage is suited to headers naturally containing commas
-  in their value, such as Set-Cookie, Expires and so on.
+  The first keyword is the rule's action. The supported actions are described
+  below.
 
-  Example:
+  There is no limit to the number of http-response statements per instance.
 
-http-response replace-header Set-Cookie (C=[^;]*);(.*) \1;ip=%bi;\2
+  It is important to know that http-response rules are processed very early in
+  the HTTP processing, before "rspdel" or "rsprep" or "rspadd" rules. That way,
+  headers added by "add-header"/"set-header" are visible by almost all further
+  ACL rules.
 
-  applied to:
+  Using "rspadd"/"rspdel"/"rsprep" to manipulate request headers is discouraged
+  in newer versions (>= 1.5). But if you need to use regular expression to
+  delete headers, you can still use "rspdel". Also please use
+  "http-response deny" instead of "rspdeny".
 
-Set-Cookie: C=1; expires=Tue, 14-Jun-2016 01:40:45 GMT
+  Example:
+acl key_acl res.hdr(X-Acl-Key) -m found
 
-  outputs:
+acl myhost hdr(Host) -f myhost.lst
 
-Set-Cookie: C=1;ip=192.168.1.20; expires=Tue, 14-Jun-2016 01:40:45 GMT
+http-response add-acl(myhost.lst) %[res.hdr(X-Acl-Key)] if key_acl
+http-response del-acl(myhost.lst) %[res.hdr(X-Acl-Key)] if key_acl
 
-  assuming the backend IP is 192.168.1.20.
+  Example:
+acl value  res.hdr(X-Value) -m found
 
-- "replace-value" works like "replace-header" except that it matches the
-  regex against every comma-delimited value of the header field 
-  instead of the entire header. This is suited for all headers which are
-  allowed to carry more than one value. An example could be the Accept
-  header.
+use_backend bk_appli if { hdr(Host),map_str(map.lst) -m found }
 
-  Example:
+http-response set-map(map.lst) %[src] %[res.hdr(X-Value)] if value
+http-response del-map(map.lst) %[src] if ! value
 
-http-response replace-value Cache-control ^public$ private
+  See also : "http-request", section 3.4 about userlists and section 7 about
+ ACL usage.
 
-  applied to:
+http-response add-acl()  [ { if | unless }  ]
 
-Cache-Control: max-age=3600, public
+  This is used to add a new entry into an ACL. The ACL must be loaded from a
+  file (even a dummy empty 

[PATCH 1/2] DOC: split the http-request actions in their own section

[Cyril Bonté]

Since http-request was first introduced, more and more actions have been
added over time. This makes the "http-request" difficult to read and some
actions were forgotten in the list.

This is an attempt to make the documenation cleaner. In future steps, it
would be great to provide at least one example for each action.
---
 doc/configuration.txt | 917 +-
 1 file changed, 467 insertions(+), 450 deletions(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index 60857739..f8f9240a 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -3933,31 +3933,8 @@ http-check send-state
 
   See also : "option httpchk", "http-check disable-on-404"
 
-http-request { allow | auth [realm ] | redirect  | reject |
-  tarpit [deny_status ] | deny [deny_status ] |
-  add-header   | set-header   |
-  capture  [ len  | id  ] |
-  del-header  | set-nice  | set-log-level  |
-  replace-header|
-  replace-value|
-  set-method  | set-path  | set-query  |
-  set-uri  | set-tos  | set-mark  |
-  set-priority-class  | set-priority-offset 
-  add-acl()  |
-  del-acl()  |
-  del-map()  |
-  set-map()   |
-  set-var()  |
-  unset-var() |
-  { track-sc0 | track-sc1 | track-sc2 }  [table ] |
-  sc-inc-gpc0() |
-  sc-inc-gpc1() |
-  sc-set-gpt0()  |
-  silent-drop |
-  send-spoe-group |
-  cache-use
- }
- [ { if | unless }  ]
+
+http-request  [options...] [ { if | unless }  ]
   Access control for Layer 7 requests
 
   May be used in sections:   defaults | frontend | listen | backend
@@ -3969,519 +3946,559 @@ http-request { allow | auth [realm ] | 
redirect  | reject |
   followed by an ACL-based condition, in which case it will only be evaluated
   if the condition is true.
 
-  The first keyword is the rule's action. Currently supported actions include :
-- "allow" : this stops the evaluation of the rules and lets the request
-  pass the check. No further "http-request" rules are evaluated.
+  The first keyword is the rule's action. The supported actions are described
+  below.
 
-- "deny" : this stops the evaluation of the rules and immediately rejects
-  the request and emits an HTTP 403 error, or optionally the status code
-  specified as an argument to "deny_status". The list of permitted status
-  codes is limited to those that can be overridden by the "errorfile"
-  directive. No further "http-request" rules are evaluated.
-
-- "reject" : this stops the evaluation of the rules and immediately closes
-  the connection without sending any response. It acts similarly to the
-  "tcp-request content reject" rules. It can be useful to force an
-  immediate connection closure on HTTP/2 connections.
-
-- "tarpit" : this stops the evaluation of the rules and immediately blocks
-  the request without responding for a delay specified by "timeout tarpit"
-  or "timeout connect" if the former is not set. After that delay, if the
-  client is still connected, an HTTP error 500 (or optionally the status
-  code specified as an argument to "deny_status") is returned so that the
-  client does not suspect it has been tarpitted. Logs will report the flags
-  "PT". The goal of the tarpit rule is to slow down robots during an attack
-  when they're limited on the number of concurrent requests. It can be very
-  efficient against very dumb robots, and will significantly reduce the
-  load on firewalls compared to a "deny" rule. But when facing "correctly"
-  developed robots, it can make things worse by forcing haproxy and the
-  front firewall to support insane number of concurrent connections. See
-  also the "silent-drop" action below.
-
-- "auth" : this stops the evaluation of the rules and immediately responds
-  with an HTTP 401 or 407 error code to invite the user to present a valid
-  user name and password. No further "http-request" rules are evaluated. An
-  optional "realm" parameter is supported, it sets the authentication realm
-  that is returned with the response (typically the application's name).
+  There is no limit to the number of http-request statements per instance.
 
-- "redirect" : this performs an HTTP redirection based on a redirect rule.
-  This is exactly the same as the "redirect" statement except that it
-  inserts a redirect rule which can be processed in the middle of other
-  "http-request" rules and that these rules use the "log-format" strings.
-  See the "redirect" keyword for the rule's syntax.
+  It is important to know that http-request rules are processed very early in
+  the HTTP processing, just after "block" rules and before 

Re: Setting response headers conditionally

[Cyril Bonté]

Hi,

Le 14/10/2018 à 22:39, Ivan Kurnosov a écrit :
I have the following config, it's under the `frontend` section for tls 
connection and haproxy terminates https connections:


     acl domain-acl-host hdr(host) -i domain.tld
     rspadd X-Foo:\ bar if domain-acl-host
     rspadd X-Baz:\ baz
     http-response set-header X-Bar bar if domain-acl-host
     use_backend backend_name if domain-acl-host

The `use_backend` directive works conditionally as expected (there are 
multiple different domain names served, and they are chosen correctly)


But headers are not added/set to the response conditionally.

I expect 3 extra headers to be added there: `X-Foo`, `X-Baz`, and 
`X-Bar`, but only `X-Baz` is added:


     < HTTP/1.1 302 Found
     < Server: nginx
     < Content-Type: text/html; charset=UTF-8
     < Transfer-Encoding: chunked
     < Cache-Control: max-age=0, must-revalidate, private
     < Date: Sun, 14 Oct 2018 20:25:59 GMT
     < Location: https://domain.tld/somewhere/else
     < X-Baz: baz

I'm sure I'm missing something trivial, but reading documentation or 
google did not help.


Well, did you have a look at the warnings emitted by haproxy on startup 
saying your acl will never match for "rspadd X-Foo" and "http-response 
set-header" ? You can't manipulate response headers based on request 
headers acl (they're not in the memory buffer anymore).


You can capture the request header in a variable and modify your acl to 
use this variable instead.


Example:
http-request set-var(txn.host) hdr(host)
acl domain-acl-host var(txn.host) -i domain.tld

See the documentation for details:
- 
http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-http-request

- http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#var


PS: it's `haproxy 1.8.8`

PPS: I originally asked it at https://serverfault.com/q/935492/45086 as well

--
With best regards, Ivan Kurnosov



--
Cyril Bonté

commit #f9cc07c25b: compilation error when threads are disabled

[Cyril Bonté]

Hi William,

I won't have time to provide a patch tonight and maybe you're still 
reorganizing some code.
My compilation test scripts are reporting that commit #f9cc07c25b broke 
the compilation when threads are disabled (USE_THREAD=) :


src/haproxy.c: In function ‘mworker_loop’:
src/haproxy.c:870:6: error: lvalue required as left operand of assignment
  tid = 0;
  ^

http://git.haproxy.org/?p=haproxy.git;a=commit;h=f9cc07c25beab93700044955d97117465b4852ae

I prefer to report the issue as I'm not sure you've seen it ;-)

--
Cyril Bonté

Re: HTTP/2 issues and segfaults with current 1.9-dev [7ee465]

[Cyril Bonté]

Hi Willy,

Le 22/08/2018 à 05:42, Willy Tarreau a écrit :

On Wed, Aug 22, 2018 at 04:32:49AM +0200, Willy Tarreau wrote:

Excellent, I think I found it :

 trash.data = recv(conn->handle.fd, trash.area, trash.size,
  MSG_PEEK);
 if (trash.data < 0) {
 if (errno == EINTR)
 continue;
 if (errno == EAGAIN) {
 fd_cant_recv(conn->handle.fd);
 return 0;
 }
...

trash.data is a size_t now so it cannot be negative. Thus it's believed
that recv() never fails. This it's clearly related to the buffer changes.
I'm seeing a few other such places that require using an intermediate
variable for the test. After all it's not that bad because we've inherited
such assignments from a good decade, and it's time to clean this up as well.


So I've now addressed all those I could find (quite a bunch in fact).
I think everything's OK now regarding this. I haven't checked for the
capture yet but since it was already broken in 1.6, it can still wait
a bit ;-)


Great job ! I can't reproduce the issue anymore, even with nbthread > 1 
(I noticed too late it was easier to trigger). We can consider this as 
fixed ;-)

I'll upgrade the haproxy instance on my test server in the morning.

--
Cyril Bonté

Re: HTTP/2 issues and segfaults with current 1.9-dev [7ee465]

[Cyril Bonté]

Le 22/08/2018 à 00:40, Cyril Bonté a écrit :

Hi again Willy,

Le 21/08/2018 à 22:55, Cyril Bonté a écrit :

Thanks for the diag. I don't remember changing anything around the proxy
protocol, but it's possible that something subtle changed. Also it's not
on the regular send/receive path so maybe I overlooked certain parts and
broke it by accident when changing the buffers.

Same here, if you have a small reproducer it will really help.


I try to find a configuration that could help identify the issue, but 
currently I fail (timings seems to have a role). I let you know once I 
have a good reproducer.


OK, I have a small reproducer that triggers the issue quite often on my 
laptop:

     global
     log /dev/log len 2048 local7 debug err

     nbproc 4

     defaults
     mode http
     log global
     option log-health-checks

     listen ssl-offload-http
     bind :4443 ssl crt localhost.pem ssl no-sslv3 alpn h2,http/1.1
     bind-process 2-4

     server http abns@http send-proxy

     listen http
     bind-process 1
     bind abns@http accept-proxy name ssl-offload-http

     option forwardfor

then execute several H2 requests on the same connection, for example:
$ curl -k -d 'x=x' $(printf 'https://localhost:4443/%s ' {1..8})
503 Service Unavailable
No server is available to handle this request.

503 Service Unavailable
No server is available to handle this request.

503 Service Unavailable
No server is available to handle this request.

curl: (16) Error in the HTTP2 framing layer
503 Service Unavailable
No server is available to handle this request.

curl: (16) Error in the HTTP2 framing layer
curl: (16) Error in the HTTP2 framing layer
503 Service Unavailable
No server is available to handle this request.


In the logs, I can see:
Aug 22 00:34:19 asus-wifi haproxy[12623]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.459] http http/ 0/-1/-1/-1/0 503 212 - - 
SC-- 1/1/0/0/0 0/0 "POST /1 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12625]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.458] ssl-offload-http~ ssl-offload-http/http 
0/0/0/0/0 503 212 - -  1/1/0/0/0 0/0 "POST /1 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12623]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.459] http http/ 0/-1/-1/-1/0 503 212 - - 
SC-- 1/1/0/0/0 0/0 "POST /2 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12625]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.459] ssl-offload-http~ ssl-offload-http/http 
0/0/0/0/0 503 212 - -  1/1/0/0/0 0/0 "POST /2 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12623]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.460] http http/ 0/-1/-1/-1/0 503 212 - - 
SC-- 1/1/0/0/0 0/0 "POST /3 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12625]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.459] ssl-offload-http~ ssl-offload-http/http 
0/0/0/0/0 503 212 - -  1/1/0/0/0 0/0 "POST /3 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12623]: PROXY SIG ERROR 
X-Forwarded-For: 127.0.0.1
Aug 22 00:34:19 asus-wifi haproxy[12623]: unix:1 
[22/Aug/2018:00:34:19.460] http/ssl-offload-http: Received something 
which does not look like a PROXY protocol header
Aug 22 00:34:19 asus-wifi haproxy[12625]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.459] ssl-offload-http~ ssl-offload-http/http 
0/0/0/-1/0 -1 0 - - SD-- 1/1/0/0/0 0/0 "POST /4 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12623]: PROXY SIG ERROR unix:1 
[22/Aug/2018:00:34:19.460] http/ssl-offload-http
Aug 22 00:34:19 asus-wifi haproxy[12623]: unix:1 
[22/Aug/2018:00:34:19.461] http/ssl-offload-http: Received something 
which does not look like a PROXY protocol header
Aug 22 00:34:20 asus-wifi haproxy[12623]: 127.0.0.1:37542 
[22/Aug/2018:00:34:20.462] http http/ 0/-1/-1/-1/0 503 212 - - 
SC-- 1/1/0/0/0 0/0 "POST /5 HTTP/1.1"
Aug 22 00:34:20 asus-wifi haproxy[12625]: 127.0.0.1:37542 
[22/Aug/2018:00:34:19.460] ssl-offload-http~ ssl-offload-http/http 
0/0/1002/0/1002 503 212 - -  1/1/0/0/1 0/0 "POST /5 HTTP/1.1"
Aug 22 00:34:20 asus-wifi haproxy[12623]: PROXY SIG ERROR 
X-Forwarded-For: 127.0.0.1
Aug 22 00:34:20 asus-wifi haproxy[12623]: unix:1 
[22/Aug/2018:00:34:20.463] http/ssl-offload-http: Received something 
which does not look like a PROXY protocol header
Aug 22 00:34:20 asus-wifi haproxy[12625]: 127.0.0.1:37542 
[22/Aug/2018:00:34:20.463] ssl-offload-http~ ssl-offload-http/http 
0/0/0/-1/0 -1 0 - - SD-- 1/1/0/0/0 0/0 "POST /6 HTTP/1.1"
Aug 22 00:34:20 asus-wifi haproxy[12623]: PROXY SIG ERROR unix:1 
[22/Aug/2018:00:34:20.463] http/ssl-offload-http
Aug 22 00:34:20 asus-wifi haproxy[12623]: unix:1 
[22/Aug/2018:00:34:20.469] http/ssl-offload-http: Received something 
which does not look like a PROXY protocol header
Aug 22 00:34:20 asus-wifi haproxy[12625]: 127.0.0.1:37546 
[22/Aug/2018:00:34:20.469] ssl-offload-http~ ssl-offload-http/http 
0/0/0/-1/0 -1 0 - - SD-- 1/1/0/0/0 0/0 "POST /7 HTTP/1.1"
Aug 22 00:34:20 asus-wifi haproxy[12623]

Re: HTTP/2 issues and segfaults with current 1.9-dev [7ee465]

[Cyril Bonté]

Hi again Willy,

Le 21/08/2018 à 22:55, Cyril Bonté a écrit :

Thanks for the diag. I don't remember changing anything around the proxy
protocol, but it's possible that something subtle changed. Also it's not
on the regular send/receive path so maybe I overlooked certain parts and
broke it by accident when changing the buffers.

Same here, if you have a small reproducer it will really help.


I try to find a configuration that could help identify the issue, but 
currently I fail (timings seems to have a role). I let you know once I 
have a good reproducer.


OK, I have a small reproducer that triggers the issue quite often on my 
laptop:

global
log /dev/log len 2048 local7 debug err

nbproc 4

defaults
mode http
log global
option log-health-checks

listen ssl-offload-http
bind :4443 ssl crt localhost.pem ssl no-sslv3 alpn h2,http/1.1
bind-process 2-4

server http abns@http send-proxy

listen http
bind-process 1
bind abns@http accept-proxy name ssl-offload-http

option forwardfor

then execute several H2 requests on the same connection, for example:
$ curl -k -d 'x=x' $(printf 'https://localhost:4443/%s ' {1..8})
503 Service Unavailable
No server is available to handle this request.

503 Service Unavailable
No server is available to handle this request.

503 Service Unavailable
No server is available to handle this request.

curl: (16) Error in the HTTP2 framing layer
503 Service Unavailable
No server is available to handle this request.

curl: (16) Error in the HTTP2 framing layer
curl: (16) Error in the HTTP2 framing layer
503 Service Unavailable
No server is available to handle this request.


In the logs, I can see:
Aug 22 00:34:19 asus-wifi haproxy[12623]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.459] http http/ 0/-1/-1/-1/0 503 212 - - 
SC-- 1/1/0/0/0 0/0 "POST /1 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12625]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.458] ssl-offload-http~ ssl-offload-http/http 
0/0/0/0/0 503 212 - -  1/1/0/0/0 0/0 "POST /1 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12623]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.459] http http/ 0/-1/-1/-1/0 503 212 - - 
SC-- 1/1/0/0/0 0/0 "POST /2 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12625]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.459] ssl-offload-http~ ssl-offload-http/http 
0/0/0/0/0 503 212 - -  1/1/0/0/0 0/0 "POST /2 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12623]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.460] http http/ 0/-1/-1/-1/0 503 212 - - 
SC-- 1/1/0/0/0 0/0 "POST /3 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12625]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.459] ssl-offload-http~ ssl-offload-http/http 
0/0/0/0/0 503 212 - -  1/1/0/0/0 0/0 "POST /3 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12623]: PROXY SIG ERROR 
X-Forwarded-For: 127.0.0.1
Aug 22 00:34:19 asus-wifi haproxy[12623]: unix:1 
[22/Aug/2018:00:34:19.460] http/ssl-offload-http: Received something 
which does not look like a PROXY protocol header
Aug 22 00:34:19 asus-wifi haproxy[12625]: 127.0.0.1:37538 
[22/Aug/2018:00:34:19.459] ssl-offload-http~ ssl-offload-http/http 
0/0/0/-1/0 -1 0 - - SD-- 1/1/0/0/0 0/0 "POST /4 HTTP/1.1"
Aug 22 00:34:19 asus-wifi haproxy[12623]: PROXY SIG ERROR unix:1 
[22/Aug/2018:00:34:19.460] http/ssl-offload-http
Aug 22 00:34:19 asus-wifi haproxy[12623]: unix:1 
[22/Aug/2018:00:34:19.461] http/ssl-offload-http: Received something 
which does not look like a PROXY protocol header
Aug 22 00:34:20 asus-wifi haproxy[12623]: 127.0.0.1:37542 
[22/Aug/2018:00:34:20.462] http http/ 0/-1/-1/-1/0 503 212 - - 
SC-- 1/1/0/0/0 0/0 "POST /5 HTTP/1.1"
Aug 22 00:34:20 asus-wifi haproxy[12625]: 127.0.0.1:37542 
[22/Aug/2018:00:34:19.460] ssl-offload-http~ ssl-offload-http/http 
0/0/1002/0/1002 503 212 - -  1/1/0/0/1 0/0 "POST /5 HTTP/1.1"
Aug 22 00:34:20 asus-wifi haproxy[12623]: PROXY SIG ERROR 
X-Forwarded-For: 127.0.0.1
Aug 22 00:34:20 asus-wifi haproxy[12623]: unix:1 
[22/Aug/2018:00:34:20.463] http/ssl-offload-http: Received something 
which does not look like a PROXY protocol header
Aug 22 00:34:20 asus-wifi haproxy[12625]: 127.0.0.1:37542 
[22/Aug/2018:00:34:20.463] ssl-offload-http~ ssl-offload-http/http 
0/0/0/-1/0 -1 0 - - SD-- 1/1/0/0/0 0/0 "POST /6 HTTP/1.1"
Aug 22 00:34:20 asus-wifi haproxy[12623]: PROXY SIG ERROR unix:1 
[22/Aug/2018:00:34:20.463] http/ssl-offload-http
Aug 22 00:34:20 asus-wifi haproxy[12623]: unix:1 
[22/Aug/2018:00:34:20.469] http/ssl-offload-http: Received something 
which does not look like a PROXY protocol header
Aug 22 00:34:20 asus-wifi haproxy[12625]: 127.0.0.1:37546 
[22/Aug/2018:00:34:20.469] ssl-offload-http~ ssl-offload-http/http 
0/0/0/-1/0 -1 0 - - SD-- 1/1/0/0/0 0/0 "POST /7 HTTP/1.1"
Aug 22 00:34:20 asus-wifi haproxy[12623]: 127.0.0.1:37550 
[22/Aug/2018:00:34:20.472] http http/ 0/-1/

Re: HTTP/2 issues and segfaults with current 1.9-dev [7ee465]

[Cyril Bonté]

Le 21/08/2018 à 21:56, Willy Tarreau a écrit :

On Tue, Aug 21, 2018 at 07:00:39PM +0200, Cyril Bonté wrote:

Le 21/08/2018 à 18:36, Willy Tarreau a écrit :

Cyril,

if you want to test, please be sure to update to at least 1b13bfd as
we've just added another fix on top of this series.


OK, I was performing some tests, I've now updated with this patch.
It looks better as now I don't see anymore hangs, but some issues remain.


OK, FWIW I've finally addressed the remaining potential deadlock causes
that could happen between checks and other operations on a same server
around the rdv point. But your issues seem unrelated.


Ah great !


(...)

OK, that one is my fault, I used a "listen" section for use_backend (maybe
we'll have to secure such configuration error to prevent segfaults).


I don't see how it could be an issue, that's a completely supported
config! You can even have this :

   listen a
  use_backend b if { prefer_b }

   listen b
  use_backend a if { prefer_a }

So any reproducer would be interesting.


I could extract this minimal configuration to reproduce the issue:
  defaults
mode http

  listen proxy1
bind :9001
http-request set-header X-Forwarded-Proto http
http-request capture hdr(X-Forwarded-Proto) len 8

  listen proxy2
bind :9002
use_backend proxy1

then, curl http://localhost:9002/ will cause a segfault.


The good news : I kept a configuration with abns, but removed the proxy
protocol. Everything is working as expected. So, the proxy protocol looks
to have a main role in the buffer issue.


Thanks for the diag. I don't remember changing anything around the proxy
protocol, but it's possible that something subtle changed. Also it's not
on the regular send/receive path so maybe I overlooked certain parts and
broke it by accident when changing the buffers.

Same here, if you have a small reproducer it will really help.


I try to find a configuration that could help identify the issue, but 
currently I fail (timings seems to have a role). I let you know once I 
have a good reproducer.



--
Cyril Bonté

Re: HTTP/2 issues and segfaults with current 1.9-dev [7ee465]

[Cyril Bonté]

Le 21/08/2018 à 19:00, Cyril Bonté a écrit :

Le 21/08/2018 à 18:36, Willy Tarreau a écrit :

Cyril,

if you want to test, please be sure to update to at least 1b13bfd as
we've just added another fix on top of this series.


OK, I was performing some tests, I've now updated with this patch.
It looks better as now I don't see anymore hangs, but some issues 
remain. It seems that some data in the buffers are not processed correctly.

Some requests done in the grafana dashboard fails to receive the response.
Note that in that test, I'm in the case where SSL is performed on a 
frontend, then sent to a backend thru abns and send-proxy, the backends 
makes some captures, etc...


Time to time, it results in such logs :
Aug 21 18:52:25 intense.local haproxy[5894]: unix:2 
[21/Aug/2018:18:52:25.169] http/ssl-offload-http: Received something 
which does not look like a PROXY protocol header


To exclude send-proxy from the equation, I've replaced the line
   server http abns@http send-proxy
with
   use_backend http

Sadly, it was worst with a new segfault on header captures.
#0  0x563f9a94bee7 in http_action_req_capture (rule=, 
px=, sess=, s=, 
flags=) at src/proto_http.c:12268

12268   if (cap[h->index] == NULL)
(gdb) bt
#0  0x563f9a94bee7 in http_action_req_capture (rule=, 
px=, sess=, s=, 
flags=) at src/proto_http.c:12268
#1  0x563f9a95076b in http_req_get_intercept_rule 
(px=px@entry=0x563f9b34f440, rules=rules@entry=0x563f9b34f488, 
s=s@entry=0x563f9e49f3f0, deny_status=deny_status@entry=0x7ffd1da4e20c) 
at src/proto_http.c:2767
#2  0x563f9a956eec in http_process_req_common 
(s=s@entry=0x563f9e49f3f0, req=req@entry=0x563f9e49f400, 
an_bit=an_bit@entry=256, px=0x563f9b34f440) at src/proto_http.c:3494
#3  0x563f9a98ea3a in process_stream (t=, 
context=0x563f9e49f3f0, state=) at src/stream.c:1932

#4  0x563f9aa14978 in process_runnable_tasks () at src/task.c:381
#5  0x563f9a9c3371 in run_poll_loop () at src/haproxy.c:2386
#6  run_thread_poll_loop (data=) at src/haproxy.c:2451
#7  0x563f9a91d9de in main (argc=, 
argv=0x7ffd1da4e748) at src/haproxy.c:3053


OK, that one is my fault, I used a "listen" section for use_backend 
(maybe we'll have to secure such configuration error to prevent segfaults).


The good news : I kept a configuration with abns, but removed the proxy 
protocol. Everything is working as expected. So, the proxy protocol 
looks to have a main role in the buffer issue.



--
Cyril Bonté

Re: HTTP/2 issues and segfaults with current 1.9-dev [7ee465]

[Cyril Bonté]

Le 21/08/2018 à 18:36, Willy Tarreau a écrit :

Cyril,

if you want to test, please be sure to update to at least 1b13bfd as
we've just added another fix on top of this series.


OK, I was performing some tests, I've now updated with this patch.
It looks better as now I don't see anymore hangs, but some issues 
remain. It seems that some data in the buffers are not processed correctly.

Some requests done in the grafana dashboard fails to receive the response.
Note that in that test, I'm in the case where SSL is performed on a 
frontend, then sent to a backend thru abns and send-proxy, the backends 
makes some captures, etc...


Time to time, it results in such logs :
Aug 21 18:52:25 intense.local haproxy[5894]: unix:2 
[21/Aug/2018:18:52:25.169] http/ssl-offload-http: Received something 
which does not look like a PROXY protocol header


To exclude send-proxy from the equation, I've replaced the line
  server http abns@http send-proxy
with
  use_backend http

Sadly, it was worst with a new segfault on header captures.
#0  0x563f9a94bee7 in http_action_req_capture (rule=, 
px=, sess=, s=, 
flags=) at src/proto_http.c:12268

12268   if (cap[h->index] == NULL)
(gdb) bt
#0  0x563f9a94bee7 in http_action_req_capture (rule=, 
px=, sess=, s=, 
flags=) at src/proto_http.c:12268
#1  0x563f9a95076b in http_req_get_intercept_rule 
(px=px@entry=0x563f9b34f440, rules=rules@entry=0x563f9b34f488, 
s=s@entry=0x563f9e49f3f0, deny_status=deny_status@entry=0x7ffd1da4e20c) 
at src/proto_http.c:2767
#2  0x563f9a956eec in http_process_req_common 
(s=s@entry=0x563f9e49f3f0, req=req@entry=0x563f9e49f400, 
an_bit=an_bit@entry=256, px=0x563f9b34f440) at src/proto_http.c:3494
#3  0x563f9a98ea3a in process_stream (t=, 
context=0x563f9e49f3f0, state=) at src/stream.c:1932

#4  0x563f9aa14978 in process_runnable_tasks () at src/task.c:381
#5  0x563f9a9c3371 in run_poll_loop () at src/haproxy.c:2386
#6  run_thread_poll_loop (data=) at src/haproxy.c:2451
#7  0x563f9a91d9de in main (argc=, 
argv=0x7ffd1da4e748) at src/haproxy.c:3053



--
Cyril Bonté

Re: haproxy-1.9-dev [0c026f49e]: 100% CPU when a server goes DOWN with option redispatch

[Cyril Bonté]

> De: "Cyril Bonté" 
> À: "Willy Tarreau" 
> Cc: "HAProxy" 
> Envoyé: Mardi 21 Août 2018 16:09:55
> Objet: haproxy-1.9-dev [0c026f49e]: 100% CPU when a server goes DOWN with 
> option redispatch
> 
> Hi Willy,
> Here is another issue seen today with the current dev branch [tests
> were also made after pulling recent commit 3bcc2699b].
> 
> Since 0c026f49e, when a server status is set to DOWN and option
> redispatch is enabled, the haproxy process hits 100% CPU.
> Even more, with the latest commits, if haproxy is compiled with
> DEBUG_FULL, it will simply segfault.

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x7703f2f1 in __GI_abort () at abort.c:79
#2  0x5558040f in __spin_lock (line=, file=, func=, l=, lbl=) at 
include/common/hathreads.h:725
#3  pendconn_redistribute (s=0x55786d80) at src/queue.c:411
#4  0x555e7842 in srv_update_status () at src/server.c:4680
#5  0x555e89a1 in srv_set_stopped.part () at src/server.c:966
#6  0x555e8bd1 in srv_set_stopped (s=, reason=, check=) at src/server.c:948
#7  0x55639358 in process_chk_conn (state=, 
context=0x557871d0, t=0x55783290) at src/checks.c:2265
#8  process_chk () at src/checks.c:2304
#9  0x556a2293 in process_runnable_tasks () at src/task.c:384
#10 0x556408b9 in run_poll_loop () at src/haproxy.c:2386
#11 run_thread_poll_loop () at src/haproxy.c:2451
#12 0x55581e4b in main () at src/haproxy.c:3053
#13 0x7702ab17 in __libc_start_main (main=0x55580e50 , 
argc=3, argv=0x7fffdfa8, init=, fini=, 
rtld_fini=, stack_end=0x7fffdf98) at ../csu/libc-start.c:310
#14 0x55583d0a in _start ()


> Here is the minimal configuration for the test:
> listen crash
> bind :9000
> option redispatch
> server non-existent 127.0.0.1: check
> 
> I'll look at it later if nobody has fixed it in the mean time ;)
> Cyril
> 
> 

haproxy-1.9-dev [0c026f49e]: 100% CPU when a server goes DOWN with option redispatch

[Cyril Bonté]

Hi Willy,
Here is another issue seen today with the current dev branch [tests were also 
made after pulling recent commit 3bcc2699b].

Since 0c026f49e, when a server status is set to DOWN and option redispatch is 
enabled, the haproxy process hits 100% CPU.
Even more, with the latest commits, if haproxy is compiled with DEBUG_FULL, it 
will simply segfault.

Here is the minimal configuration for the test:
listen crash
bind :9000
option redispatch
server non-existent 127.0.0.1: check

I'll look at it later if nobody has fixed it in the mean time ;)
Cyril

HTTP/2 issues and segfaults with current 1.9-dev [7ee465]

[Cyril Bonté]

Hi Willy,

I'm afraid there's still some issues with HTTP/2 in the current dev branch :-(

This morning, I've upgraded a test server and discovered that some HTTPS sites 
did not work anymore (content hangs and is not sent to the client), I've also 
noticed some segfaults in haproxy.
As this is a test server that I've used for several years with haproxy, the 
configuration begins to be quite ugly, it won't be helpful to provide it in its 
current state.

Here is a backtrace of a recent segfault:
#0  si_cs_send (cs=0x0) at src/stream_interface.c:648
#1  0x557260d4c6cd in si_cs_io_cb (t=, ctx=, 
state=) at src/stream_interface.c:764
#2  0x557260d7d237 in process_runnable_tasks () at src/task.c:384
#3  0x557260d2bf61 in run_poll_loop () at src/haproxy.c:2386
#4  run_thread_poll_loop (data=) at src/haproxy.c:2451
#5  0x557260c869de in main (argc=, argv=0x7fff3770b8d8) at 
src/haproxy.c:3053

I could identify that it was easy to reproduce with a grafana server behind 
haproxy (loading css/js resources seems to hang).
It seems the issues began with commit d54a8ceb9 MAJOR: start to change buffer 
API.

Here is an example of configuration which allows to reproduce the hanging issue 
(I could not reproduce the segfault with that one):
defaults http
mode http
timeout connect 5s
timeout client  300s
timeout server  300s
timeout http-request 10s
timeout http-keep-alive 15s

listen http
bind :4080 name http  # OK
bind :4443 ssl crt localhost.pem ssl no-sslv3 alpn h2,http/1.1  # FAIL
bind :6443 ssl crt localhost.pem ssl no-sslv3  # OK

bind abns@http accept-proxy

server grafana 127.0.0.1:3000

listen https
bind :8443 ssl crt localhost.pem ssl no-sslv3 alpn h2,http/1.1  # FAIL

http-reuse never
server http abns@http send-proxy

>From the browser, requesting http://localhost:4080/ or 
>https://localhost:6443/, it will work.
But once HTTP/2 is used, it hangs : https://localhost:4443/ and 
http://localhost:8443/

Some details:
# haproxy -vv
HA-Proxy version 1.9-dev1-7ee465-56 2018/08/19
Copyright 2000-2018 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv 
-fno-strict-overflow -Wno-null-dereference -Wno-unused-label
  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0f  25 May 2017
Running on OpenSSL version : OpenSSL 1.1.0f  25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols markes as  cannot be specified using 'proto' keyword)
  h2 : mode=HTTP   side=FE
: mode=TCP|HTTP   side=FE|BE

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

I'll try to investigate more tonight,
Cyril

[PATCH] BUG/MINOR: lua: fix extra 500ms added to socket timeouts

[Cyril Bonté]

Since commit #56cc12509, haproxy accepts double values for timeouts. The
value is then converted to  milliseconds before being rounded up and cast
to int. The issue is that to round up the value, a constant value of 0.5
is added to it, but too early in the conversion, resulting in an
additional 500ms to the value. We are talking about a precision of 1ms,
so we can safely get rid of this rounding trick and adjust resulting
timeouts equal to 0 to a minimum of 1ms.

This patch is specific to the 1.9 branch and doesn't require to be
backported.
---
 src/hlua.c | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/src/hlua.c b/src/hlua.c
index 06c115561..65986473a 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -2545,17 +2545,19 @@ __LJMP static int hlua_socket_settimeout(struct 
lua_State *L)
 
socket = MAY_LJMP(hlua_checksocket(L, 1));
 
-   /* round up for inputs that are fractions and convert to millis */
-   dtmout = (0.5 + MAY_LJMP(luaL_checknumber(L, 2))) * 1000;
+   /* convert the timeout to millis */
+   dtmout = MAY_LJMP(luaL_checknumber(L, 2)) * 1000;
 
/* Check for negative values */
if (dtmout < 0)
WILL_LJMP(luaL_error(L, "settimeout: cannot set negatives 
values"));
 
if (dtmout > INT_MAX) /* overflow check */
-   WILL_LJMP(luaL_error(L, "settimeout: cannot set values larger 
than %d", INT_MAX));
+   WILL_LJMP(luaL_error(L, "settimeout: cannot set values larger 
than %d ms", INT_MAX));
 
tmout = MS_TO_TICKS((int)dtmout);
+   if (tmout == 0)
+   tmout++; /* very small timeouts are adjusted to a minium of 1ms 
*/
 
/* Check if we run on the same thread than the xreator thread.
 * We cannot access to the socket if the thread is different.
-- 
2.18.0

Re: [PATCH] BUG/MINOR: lua: fix extra 500ms added to socket timeouts

[Cyril Bonté]

Hi Willy,

Le 19/08/2018 à 07:34, Willy Tarreau a écrit :

Hi Cyril,

Thanks for this, however there remains one (small) corner case : if dtmout
is equal to INT_MAX (2147483647), it will be rounded up by the +0.5 then
become negative :


Well, the original comment is not valid. The behaviour is not to round 
up every values but to round values to the nearest int.



On Sun, Aug 19, 2018 at 12:27:13AM +0200, Cyril Bonté wrote:

if (dtmout > INT_MAX) /* overflow check */
-   WILL_LJMP(luaL_error(L, "settimeout: cannot set values larger than 
%d", INT_MAX));
+   WILL_LJMP(luaL_error(L, "settimeout: cannot set values larger than 
%d ms", INT_MAX));
   
Above it will pass the test.



-   tmout = MS_TO_TICKS((int)dtmout);
+   /* round up the timeout to convert it to int */
+   tmout = MS_TO_TICKS((int)(dtmout + 0.5));


and here we convert 2147483647.5 to int, which becomes -2147483648.


2147483647.5, when cast to int, should become 2147483647.


Given that the double to int conversion already rounds values, we do not
need the +0.5 so we can get rid of it. Alternatively, dtmout could be
rounded up before the checks using round() but apparently this is not
needed.

I suspect that the +0.5 might have been placed here to ensure that no
short timeout is rounded down to zero. 


Adding 0.5 is a trick to round positive values to the nearest int (it 
works in most cases). For example :

1234.1 will become 1234
1234.6 will become 1235


If that's the case we could
simply proceed like this :

if (dtmout < 0)
  ...
if (dtmout > INT_MAX)
  ...
tmout = MS_TO_TICKS((int)dtmout);
if (tmout == 0)
tmout++;  // min 1ms


We're talking about loosing 1ms of precision, so the rounding trick is 
maybe an overkill feature. I'm fine with removing the 0.5 addition and 
adding 1ms if the result is equal to 0. And I clearly prefer that 
behaviour ! Btw, this can already happen for very small values. 
Currently, sock:settimeout(0.0001) will result in a 0ms timeout.



Just let me know if you want me to adjust the patch one way or another
so that you don't need to respin one.


I'm going to send you a new patch, which actually consists in including 
what you've suggested ;-)



--
Cyril Bonté

[PATCH] BUG/MINOR: lua: fix extra 500ms added to socket timeouts

[Cyril Bonté]

Since commit #56cc12509, haproxy accepts double values for timeouts. The
value is then converted to  milliseconds before being round up and cast
to int. The issue is that to round up the value, a constant value of 0.5
is added to it, but too early in the conversion, resulting in an
additional 500ms to the value.

This patch is specific to the 1.9 branch and doesn't require to be
backported.
---
 src/hlua.c | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/src/hlua.c b/src/hlua.c
index 06c115561..73227b4ca 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -2545,17 +2545,18 @@ __LJMP static int hlua_socket_settimeout(struct 
lua_State *L)
 
socket = MAY_LJMP(hlua_checksocket(L, 1));
 
-   /* round up for inputs that are fractions and convert to millis */
-   dtmout = (0.5 + MAY_LJMP(luaL_checknumber(L, 2))) * 1000;
+   /* convert the timeout to millis */
+   dtmout = MAY_LJMP(luaL_checknumber(L, 2)) * 1000;
 
/* Check for negative values */
if (dtmout < 0)
WILL_LJMP(luaL_error(L, "settimeout: cannot set negatives 
values"));
 
if (dtmout > INT_MAX) /* overflow check */
-   WILL_LJMP(luaL_error(L, "settimeout: cannot set values larger 
than %d", INT_MAX));
+   WILL_LJMP(luaL_error(L, "settimeout: cannot set values larger 
than %d ms", INT_MAX));
 
-   tmout = MS_TO_TICKS((int)dtmout);
+   /* round up the timeout to convert it to int */
+   tmout = MS_TO_TICKS((int)(dtmout + 0.5));
 
/* Check if we run on the same thread than the xreator thread.
 * We cannot access to the socket if the thread is different.
-- 
2.18.0

[PATCH] BUG/MEDIUM: lua: socket timeouts are not applied

[Cyril Bonté]

Sachin Shetty reported that socket timeouts set in LUA code have no effect.
Indeed, connect timeout is never modified and is always set to its default,
set to 5 seconds. Currently, this patch will apply the specified timeout
value to the connect timeout.
For the read and write timeouts, the issue is that the timeout is updated but
the expiration dates were not updated.

This patch should be backported up to the 1.6 branch.
---
 src/hlua.c | 9 +
 1 file changed, 9 insertions(+)

diff --git a/src/hlua.c b/src/hlua.c
index c29a7cc4e..06c115561 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -2574,10 +2574,19 @@ __LJMP static int hlua_socket_settimeout(struct 
lua_State *L)
si = appctx->owner;
s = si_strm(si);
 
+   s->sess->fe->timeout.connect = tmout;
s->req.rto = tmout;
s->req.wto = tmout;
s->res.rto = tmout;
s->res.wto = tmout;
+   s->req.rex = tick_add_ifset(now_ms, tmout);
+   s->req.wex = tick_add_ifset(now_ms, tmout);
+   s->res.rex = tick_add_ifset(now_ms, tmout);
+   s->res.wex = tick_add_ifset(now_ms, tmout);
+
+   s->task->expire = tick_add_ifset(now_ms, tmout);
+   task_queue(s->task);
+
xref_unlock(>xref, peer);
 
lua_pushinteger(L, 1);
-- 
2.18.0

Re: lua socket settimeout has no effect

[Cyril Bonté]

Hi Willy and Sachin,

Le 14/08/2018 à 01:27, Cyril Bonté a écrit :
Thanks for the feedback. I've made some tests with the lines of code 
you suggested and it looks to work fine. I'll prepare the patches tomorrow.


OK, some updates on the patch.
It was working well with haproxy 1.7, the same with haproxy 1.8.
But when I prepared the patch for the 1.9 branch, I discovered a strange 
behaviour, where the timeout was approximaterly delayed by 500ms. I 
spent some time tonight, until I discovered it was under my eyes some 
lines earlier :)


The culprit is commit #56cc12509 [1] where the rounding calculation is 
wrong :

/* round up for inputs that are fractions and convert to millis */
dtmout = (0.5 + MAY_LJMP(luaL_checknumber(L, 2))) * 1000;

Well, now it's identified, I'll (really) send the patch tomorrow and 
will see in another patch how we can fix #56cc12509.



[1] 
http://git.haproxy.org/?p=haproxy.git;a=blobdiff;f=src/hlua.c;h=60cf8f948437677b72adabc50602ca99d80349bf;hp=633841c6d7aecce3e5cf893e01b5eed1cfe64e0f;hb=56cc12509;hpb=7741c854cd908dd4947325c36a6feb8203748d16



--
Cyril Bonté

Re: Trying to get logging above 1024 characters

[Cyril Bonté]

Hi Shawn,

Le 14/08/2018 à 22:53, Shawn Heisey a écrit :

I'm trying with 1.8.13 to get full logging of requests that would push
the syslog message beyond 1024 characters.  I'm not having very good luck.

I have this config in global:

   log 127.0.0.1   len 65535 format rfc5424 local0
   log 127.0.0.1   len 65535 format rfc5424 local1 notice

In some of the backends, I have this:

   no log
   log 127.0.0.1   len 65535 format rfc5424 local0 notice err

Can't remember precisely why I did that, but I came up with that config
(minus the len and format parameters) a while back after discussing
something with this mailing list.

Here's a logging message from a test with a long URL path:

Aug 14 14:25:48 smeagol haproxy[39296] 209.63.XXX.YYY:30626
[14/Aug/2018:14:25:48.365] web-443~ be-ssl-purg-4001/gollum 0/0/1/37/43
404 18221 - - --VN 2/2/0/1/0 0/0 {REDACTED} "GET
/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/eee/ddd/rrr/fff/111/ee"

The actual URL path that I tried to access is about half again as long
as what got logged.  Notice that there is a quote at the end of the
message, telling me that haproxy truncated the request and put quotes
around it.
[...]
Is there any config that will successfully log the full request?


Please read the documentation about the length option for the log 
keyword, particularly the part about tune.http.logurilen ;-)

http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#log
http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#tune.http.logurilen

--
Cyril Bonté

Re: lua socket settimeout has no effect

[Cyril Bonté]

Hi Willy,

Le 13/08/2018 à 09:28, Willy Tarreau a écrit :

Hi Cyril,

On Sun, Aug 12, 2018 at 10:49:13PM +0200, Cyril Bonté wrote:

2. hlua_socket_settimeout() initializes rto/wto values, maybe it should also
compute the rex/wex values :
socket->s->req.rex = tick_add_ifset(now_ms, tmout);
socket->s->req.wex = tick_add_ifset(now_ms, tmout);
socket->s->res.rex = tick_add_ifset(now_ms, tmout);
socket->s->res.wex = tick_add_ifset(now_ms, tmout);


You're absolutely right, it's indeed incorrect to set {r,w}to without
updating the expire date. It only works when the timeout is extended,
but not when it's shortened.


3. It may require to wake up the task if a new timeout is set after a first
one was already set (in your case the task doesn't wake up after 2 secondes
because a first timeout was set to 3 seconds) :
task_wakeup(socket->s->task, TASK_WOKEN_OTHER);


Normally it should not be needed, however it needs to be requeued to
take care of the fact that the timeout might have been shortened :

socket->s->task->expire = tick_add_ifset(now_ms, tmout);
task_queue(socket->s->task);


At least, it seems to fix the issue but before sending a patch, I want to be
sure that's how we should fix this.


If you can just validate that the two lines above are enough, that would
be great. Avoiding to call process_stream() just to update a timeout is
desirable. If for any reason it doesn't work or seems more complicated,
then the wakeup as you proposed should indeed work.


Thanks for the feedback. I've made some tests withe the lines of code 
you suggested and it looks to work fine. I'll prepare the patches tomorrow.


--
Cyril Bonté

Re: lua socket settimeout has no effect

[Cyril Bonté]

Le 12/08/2018 à 18:21, Sachin Shetty a écrit :

Hi Cyril,

I have created a very simple config to reproduce this. This config 
always read timesout in 9 seconds.


I think there are 3 issues.


[...]
function get_from_gds(key)
[...] 
     local sock = core.tcp()

     -- Connect timeout after patch
     sock:settimeout(3)
     local result = DOMAIN_NOT_FOUND
     local status, error = sock:connect(gds_host, gds_port)
     if not status then
     core.Alert("Error in connecting:" .. key .. ":" .. error)
     return "Error", "Error: " .. error
     end
     sock:settimeout(2)
     sock:send(key .. "\r\n")
     while true do
     local s, status, partial = sock:receive("*l")


1. The first one is in the LUA code, where you don't check the return 
code after calling sock:receive(). In this case, you enter in an 
"infinite" loop, adding an extra time to the response almost equal to 
tune.lua.session-timeout (4s by default).

You may want to add this :
if not s then
core.Alert("Error reading:" .. status)
return "Error", status
end

Now, 2 other issues seems to be in haproxy, but I'm not sure if it's the 
right way to fix this (I add Willy and Thierry to the thread) :
2. hlua_socket_settimeout() initializes rto/wto values, maybe it should 
also compute the rex/wex values :

socket->s->req.rex = tick_add_ifset(now_ms, tmout);
socket->s->req.wex = tick_add_ifset(now_ms, tmout);
socket->s->res.rex = tick_add_ifset(now_ms, tmout);
socket->s->res.wex = tick_add_ifset(now_ms, tmout);
3. It may require to wake up the task if a new timeout is set after a 
first one was already set (in your case the task doesn't wake up after 2 
secondes because a first timeout was set to 3 seconds) :

task_wakeup(socket->s->task, TASK_WOKEN_OTHER);

At least, it seems to fix the issue but before sending a patch, I want 
to be sure that's how we should fix this.


--
Cyril Bonté

Re: lua socket settimeout has no effect

[Cyril Bonté]

Hi,

Le 12/08/2018 à 08:41, Sachin Shetty a écrit :

Hi Cyril,

Any idea how I can deterministically set the readtimeout as well?


Well, currently I can't reproduce this at all. Can you provide some more 
details ? or even a full configuration providing the test case ?

From the tests I've made, read and write timeouts work as expected.




Thanks
Sachin

On Fri, Jul 27, 2018 at 1:23 PM, Sachin Shetty <mailto:sshe...@egnyte.com>> wrote:


Thankyou Cyril, your patch fixed the connect issue.

Read timeout still seems a bit weird though, at settimeout(1),
readtimeout kicks in at about 4 seconds, and at settimeout(2),
readtimeout kicks in at about 8 seconds.

is that expected? I couldn't find read timeout explicitly set
anywhere in the same source file.

Thanks
Sachin

On Fri, Jul 27, 2018 at 5:18 AM, Cyril Bonté mailto:cyril.bo...@free.fr>> wrote:

Hi,

Le 26/07/2018 à 19:54, Sachin Shetty a écrit :

Hi,

We are using a http-req lua action to dynamically set some
app specific metadata headers. The lua handler connects to a
upstream memcache like service over tcp to fetch additional
metadata.

Functionally everything works ok, but I am seeing that
socket.settimeout has no effect. Irrespective of what I set
in settimeout if the upstream service is unreachable,
connect always timesout at 5 seconds, and read timeout
around 10 seconds. It seems like  settimeout has no effect
and it always picks defaults of 5 seconds for connect
timeout and 10 seconds for read timeout.


For the connect timeout, it seems this is a hardcoded default
value in src/hlua.c:
   socket_proxy.timeout.connect = 5000; /* By default the
timeout connection is 5s. */

If it's possible, can you try the patch attached (for the 1.7.x
branch) ? But please don't use it in production yet ;-)


Haproxy conf call:

http-request lua.get_proxy

Lua code sample:

function get_proxy(txn)
  local sock = core.tcp()
  sock:settimeout(2)
  status, error = sock:connect(gds_host, gds_port)
  if not status then
  core.Alert("1 Error in connecting:" .. key .. ":"
.. error)
  return result, "Error: " .. error
  end
  sock:send(key .. "\r\n")
  
  


core.register_action("get_proxy", { "http-req" }, get_proxy)

Haproxy version:

HA-Proxy version 1.7.8 2017/07/07
Copyright 2000-2017 Willy Tarreau mailto:wi...@haproxy.org> <mailto:wi...@haproxy.org
<mailto:wi...@haproxy.org>>>


Build options :
    TARGET  = linux2628
    CPU = generic
    CC  = gcc
    CFLAGS  = -O2 -g -fno-strict-aliasing
-Wdeclaration-after-statement -fwrapv -DTCP_USER_TIMEOUT=18
    OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1
USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
    maxconn = 2000, bufsize = 16384, maxrewrite = 1024,
maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.2
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
    epoll : pref=300,  test result OK
     poll : pref=200,  test result OK
   select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
  [COMP] compression
      [TRACE] trace
  [SPOE] spoe



Thanks
Sachin



-- 
Cyril Bonté







--
Cyril Bonté

Re: haproxy doesn't reuse server connections

[Cyril Bonté]

Hi Alessandro,

Le 27/07/2018 à 17:50, Alessandro Gherardi a écrit :

Hi,
I'm running haproxy 1.8.12 on Ubuntu 14.04. For some reason, haproxy 
does not reuse connections to backend servers. For testing purposes, I'm 
sending the same HTTP request multiple times over the same TCP connection.


The servers do not respond with Connection: close and do not close the 
connections. The wireshark capture shows haproxy RST-ing the 
connections  a few hundred milliseconds after the servers reply. The 
servers send no FIN nor RST to haproxy.


I tried various settings (http-reuse always, option http-keep-alive, 
both at global and backend level), no luck.


The problem goes away if I have a single backend server, but obviously 
that's not a viable option in real life.


Here's my haproxy.cfg:

global
         #daemon
         maxconn 256

defaults
         mode http
         timeout connect 5000ms
         timeout client 5ms
         timeout server 5ms

         option http-keep-alive
         timeout http-keep-alive 30s
         http-reuse always

frontend http-in
         bind 10.220.178.236:80
         default_backend servers

backend servers
         server server1 10.220.178.194:80 maxconn 32
         server server2 10.220.232.132:80 maxconn 32

Any suggestions?


Well, you've not configured any persistence option nor any load 
balancing algorithm. So, the default is to do a roundrobin between the 2 
backend servers. If there's no traffic, it's very likely that there's no 
connection to reuse when switching to the second server for the second 
request.




Thanks in advance,
Alessandro



--
Cyril Bonté

Re: lua socket settimeout has no effect

[Cyril Bonté]

Hi,

Le 26/07/2018 à 19:54, Sachin Shetty a écrit :

Hi,

We are using a http-req lua action to dynamically set some app specific 
metadata headers. The lua handler connects to a upstream memcache like 
service over tcp to fetch additional metadata.


Functionally everything works ok, but I am seeing that socket.settimeout 
has no effect. Irrespective of what I set in settimeout if the upstream 
service is unreachable, connect always timesout at 5 seconds, and read 
timeout around 10 seconds. It seems like  settimeout has no effect and 
it always picks defaults of 5 seconds for connect timeout and 10 seconds 
for read timeout.


For the connect timeout, it seems this is a hardcoded default value in 
src/hlua.c:
  socket_proxy.timeout.connect = 5000; /* By default the timeout 
connection is 5s. */


If it's possible, can you try the patch attached (for the 1.7.x branch) 
? But please don't use it in production yet ;-)




Haproxy conf call:

http-request lua.get_proxy

Lua code sample:

function get_proxy(txn)
     local sock = core.tcp()
     sock:settimeout(2)
     status, error = sock:connect(gds_host, gds_port)
     if not status then
     core.Alert("1 Error in connecting:" .. key .. ":" .. error)
     return result, "Error: " .. error
     end
     sock:send(key .. "\r\n")
     
     


core.register_action("get_proxy", { "http-req" }, get_proxy)

Haproxy version:

HA-Proxy version 1.7.8 2017/07/07
Copyright 2000-2017 Willy Tarreau <mailto:wi...@haproxy.org>>


Build options :
   TARGET  = linux2628
   CPU = generic
   CC  = gcc
   CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement 
-fwrapv -DTCP_USER_TIMEOUT=18
   OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 
USE_LUA=1 USE_PCRE=1


Default settings :
   maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")

Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.2
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND


Available polling systems :
   epoll : pref=300,  test result OK
    poll : pref=200,  test result OK
  select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
     [COMP] compression
     [TRACE] trace
     [SPOE] spoe



Thanks
Sachin



--
Cyril Bonté
diff --git a/src/hlua.c b/src/hlua.c
index 874a4acc..10cf845e 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -2286,6 +2286,7 @@ __LJMP static int hlua_socket_settimeout(struct lua_State *L)
 	if (tmout < 0)
 		WILL_LJMP(luaL_error(L, "settimeout: cannot set negatives values"));
 
+	socket->s->sess->fe->timeout.connect = tmout;
 	socket->s->req.rto = tmout;
 	socket->s->req.wto = tmout;
 	socket->s->res.rto = tmout;

Re: Missing SRV cookie

[Cyril Bonté]

Hi Norman,

Le 23/07/2018 à 18:36, Norman Branitsky a écrit :

My client’s environment had 3 HAProxy servers.

Due to a routing issue, my client’s users could only see the old HAProxy 
1.5 server when connecting from their data center.

They could not see the 2 new HAProxy 1.7 servers.

The routing issue was resolved last week and they could now see the 2 
new HAProxy servers, as well the old server.


They started getting quick disconnects from their Java application –

the SEVERE error indicated that they had arrived at the wrong server and 
had no current session.

[...]
New HAProxy servers configuration:

backend ssl_backend-vr 
     balance roundrobin

     stick-table type ip size 20k peers mypeers
     stick on src


Here you are using stick tables for session persistence.


[...]
 cookie SRV insert indirect nocache httponly secure
     server i-067c94ded1c8e212c 10.241.1.138:9001 check cookie 
i-067c94ded1c8e212c
     server i-07035eca525e56235 10.241.1.133:9001 check cookie 
i-07035eca525e56235


But here, you are using cookies for the same purpose.

I realized that the cookie mechanism was different so I shut down the 
old HAProxy server and the problem appeared to be resolved.


This morning that client is complaining that the problem has returned – 
disconnects resulting in the user being kicked out to the login screen.


Checking with multiple browsers, I can see both the old JSESSIONID 
cookie (with the machine name appended) and the new SRV cookie.


Checking with multiple browsers, my colleagues can *NOT* see the new SRV 
cookie from any browser in this office –


but they can see the SRV cookie when browsing from a virtual PC in our 
Atlanta data center!
Even more puzzling, though my client cannot see the SRV cookie (either 
in the F12 cookies sent list, or in the browser’s cookies folder)

he *never* experiences an unexpected disconnect.

Suggestions, please?


You have to make a choice, either you use stick tables, either you use 
cookies, but don't mix both otherwise you'll have the situation you are 
describing.



--
Cyril Bonté

Re: HAProxy 1.8.x not serving errorfiles with H2

[Cyril Bonté]

Hi,

Le 11/06/2018 à 16:39, J. Casalino a écrit :

Trying again – has anyone else seen this?

*From:* J. Casalino [mailto:casal...@adobe.com]
*Sent:* Tuesday, June 5, 2018 12:27 PM
*To:* haproxy@formilux.org
*Subject:* HAProxy 1.8.x not serving errorfiles with H2

We are in the process of testing HAProxy 1.8.x with ALPN and H2 on some 
of our servers. We have default 502 and 503 errorfiles defined (ex. 
errorfile 503 /etc/haproxy/errors/503.http), but we’ve noticed that 
these errorfiles are not served to the user’s browser when the error 
occurs (for instance, if the backend is down, a user should get the 503 
errorfile).


Chrome returns "ERR_SPDY_PROTOCOL_ERROR”, Curl [1] returns “curl: (92) 
HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)”, and 
Firefox shows “The connection to  was interrupted while the 
page was loading.”


With debug logging turned on, I can see that HAProxy is recognizing a 
503 if the back-end server is down [2], but it doesn’t seem to pass that 
error through to the client browser. If the backend is up and a 502 is 
generated, users do not receive the errorfile either. If we turn off H2 
and drop back to HTTP/1.1, the errorfiles are displayed properly (though 
via HTTP/0.9)


If it looks like HTTP/0.9, I tend to think that your errorfile is not 
properly set. Such files must contain the status line, the headers and 
the response body.


And indeed, from a quick test, if I remove the status line, I get the 
same error with a HTTP/2 request. Once the file is correctly defined, 
everything is OK.


Can you provide the content of your errorfile(s) ?


This has been observed in both 1.8.4 and 1.8.9. Our platform is Amazon 
Linux, using openssl-1.0.2k-12.109.amzn1.x86_64.


Thanks in advance for any thoughts you might have –

[1]

Curl verbose (curl -I) output:

*   Trying ...

* TCP_NODELAY set

* Connected to  () port 443 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* Cipher selection: 
ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH


* successfully set certificate verify locations:

*   CAfile: /etc/ssl/cert.pem

   CApath: none

* TLSv1.2 (OUT), TLS handshake, Client hello (1):

* TLSv1.2 (IN), TLS handshake, Server hello (2):

* TLSv1.2 (IN), TLS handshake, Certificate (11):

* TLSv1.2 (IN), TLS handshake, Server key exchange (12):

* TLSv1.2 (IN), TLS handshake, Server finished (14):

* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

* TLSv1.2 (OUT), TLS change cipher, Client hello (1):

* TLSv1.2 (OUT), TLS handshake, Finished (20):

* TLSv1.2 (IN), TLS change cipher, Client hello (1):

* TLSv1.2 (IN), TLS handshake, Finished (20):

* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384

* ALPN, server accepted to use h2

* Server certificate:

*  subject: [removed]

*  start date: Mar 20 00:00:00 2017 GMT

*  expire date: Mar 24 12:00:00 2020 GMT

*  subjectAltName: host "" matched cert's ""

*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA

*  SSL certificate verify ok.

* Using HTTP2, server supports multi-use

* Connection state changed (HTTP/2 confirmed)

* Copying HTTP/2 data in stream buffer to connection buffer after 
upgrade: len=0


* Using Stream ID: 1 (easy handle 0x7fbded005400)

 > HEAD /libs/cq/core/content/welcome.html HTTP/2

 > Host: 

 > User-Agent: curl/7.54.0

 > Accept: */*

 >

* Connection state changed (MAX_CONCURRENT_STREAMS updated)!

* HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

* Closing connection 0

* TLSv1.2 (OUT), TLS alert, Client hello (1):

curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

[2] haproxy[19803]: :63832 [05/Jun/2018:15:36:24.202] 
incoming_https~ local_author_app_http/ 0/-1/-1/-1/0 503 1441 - - 
SCDN 3/1/0/0/0 0/0 "GET /libs/cq/core/content/welcome.html HTTP/1.1"





--
Cyril Bonté

Re: Backup server takes too long to go active

[Cyril Bonté]

Le 24/04/2018 à 23:07, Shawn Heisey a écrit :

I sent this query to the list previously nine days ago.  I got no
response.  Trying again.


Next time reply to the thread you created, it's better to track which 
ones had responses.




==

Kernel info:

root@lb1:/etc/haproxy# uname -a
Linux lb1 3.13.0-52-generic #86-Ubuntu SMP Mon May 4 04:32:59 UTC 2015
x86_64 x86_64 x86_64 GNU/Linux


Here's the haproxy version output:

root@lb1:/etc/haproxy# haproxy -vv
HA-Proxy version 1.5.12 2015/05/02
Copyright 2000-2015 Willy Tarreau <w...@1wt.eu>

Build options :
   TARGET  = linux2628
   CPU = native
   CC  = gcc
   CFLAGS  = -O2 -march=native -g -fno-strict-aliasing
   OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE= USE_PCRE_JIT=1

Default settings :
   maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.2j  26 Sep 2016
Running on OpenSSL version : OpenSSL 1.0.2j  26 Sep 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (libpcre build without JIT?)
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
   epoll : pref=300,  test result OK
poll : pref=200,  test result OK
  select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.



The configuration I had is with a backend that has two servers, one of
them tagged as backup. This is the actual config that I had active when
I saw the problem:

backend be-cdn-9000
 description Back end for the thumbs CDN
 cookie MSDSRVHA insert indirect nocache
 server planet 10.100.2.123:9000 weight 100 cookie planet track
chk-cdn-9000/planet
 server hollywood 10.100.2.124:9000 weight 100 backup cookie
hollywood track chk-cdn-9000/hollywood


Well, you don't provide any information about the tracked servers 
chk-cdn-9000/planet and chk-cdn-9000/hollywood.



My check interval is ten seconds and the check timeout is 9990 milliseconds.

==

If I shut down the primary server, eventually haproxy notices this.  No
problem so far.

The problem is that about ten seconds pass between the time haproxy
notices the primary going down and the time that the backup server
changes to active.  During that time, clients get "no server available"
errors.

It is my expectation that as soon as the primary server goes down,
haproxy will promote the backup server(s) with the highest weight to
active and send requests there.


Without any information about the 2 tracked server, I'd say the 
behaviour is expected. A backup server is promoted only if it is UP 
itself. What is the state of chk-cdn-9000/hollywood during that time ? 
It looks like it's not UP yet.



I know that I'm running an out of date version and can't expect any
changes in 1.5.x.  I have not yet tried this with a newer version of
haproxy.

As a workaround, I have removed the "backup" keyword.  For this backend
that's an acceptable solution, but I am using that configuration
elsewhere in cases where I only want the traffic to go to one server as
long as that server is up, and I need the backup server to take over
quickly when the primary fails.

Thanks,
Shawn




--
Cyril Bonté

Re: Haproxy 1.8.4 crashing workers and increased memory usage

[Cyril Bonté]

Hi Robin,

> De: "Robin Geuze" 
> À: "Willy Tarreau" 
> Cc: haproxy@formilux.org
> Envoyé: Lundi 9 Avril 2018 10:24:43
> Objet: Re: Haproxy 1.8.4 crashing workers and increased memory usage
> 
> Hey Willy,
> 
> So I made a build this morning with libslz and re-enabled compression
> and within an hour we had the exit code 134 errors, so zlib does not
> seem to be the problem here.

I have spent some times on this issue yesterday, without being able to 
reproduce it.
I suspect something wrong with pending connections (without any clue, except 
there's an abort() in the path), but couldn't see anything wrong in the code.

There's still something missing in this thread (maybe I missed it), but can you 
provide the output of "haproxy -vv" ?
Also, are you 100% sure you're running the version you compiled ? I prefer to 
ask, as it sometimes happens ;-)

Thanks,
Cyril

Re: Grrrr.... warnings

[Cyril Bonté]

Le 01/04/2018 à 23:24, Willy Tarreau a écrit :

Can someone tell me how I'm supposed to work around this one ?


It seems a patch will be available at midnight :)



gcc -Iinclude -Iebtree -Wall  -O2 -march=native -g -fno-strict-aliasing 
-Wdeclaration-after-statement -fwrapv -fno-strict-overflow-Wno-unused-label  -DBUFSIZE=8030 
-DMAXREWRITE=1030 -DSO_MARK=36-DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY 
-DCONFIG_HAP_CRYPT -DUSE_ZLIB  -DENABLE_POLL -DENABLE_EPOLL -DUSE_CPU_AFFINITY 
-DASSUME_SPLICE_WORKS -DUSE_ACCEPT4 -DNETFILTER -DUSE_THREAD -DUSE_OPENSSL  -DUSE_SYSCALL_FUTEX 
-DUSE_PCRE -I/usr/include  -DCONFIG_HAPROXY_VERSION=\"1.9-dev0-495298-299\" 
-DCONFIG_HAPROXY_DATE=\"2018/03/21\" \
   -DBUILD_TARGET='"linux2628"' \
   -DBUILD_ARCH='""' \
   -DBUILD_CPU='"native"' \
   -DBUILD_CC='"gcc"' \
   -DBUILD_CFLAGS='"-O2 -march=native -g -fno-strict-aliasing 
-Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label 
-DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36"' \
   -DBUILD_OPTIONS='"USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1"' \
-c -o src/haproxy.o src/haproxy.c

src/haproxy.c:1:1: warning: gcc has detected the presence of C code in this 
file. This language is potentially unsafe as it lets the developer do things 
that the compiler doesn't understand. Please upgrade to a new language, support 
will be deprecated in gcc-9. [-Wc-deprecated].

Thanks,
Willy




--
Cyril Bonté

Re: Logging actual fetched URL after request is re-written

[Cyril Bonté]

Hi,

Le 27/03/2018 à 18:49, Franks Andy (IT Technical Architecture Manager) a 
écrit :

Hi all,

   Logging with HTTP as standard, the %{+Q}r log variable records the 
requested URL in the logs. I’d like to also record the URL that’s 
actually fetch after an http-request set-path directive is given (for 
debugging purposes). It’s linked to an application that provides next to 
no debugging, and tcpdump isn’t much help either – having it in haproxy 
logs would be really useful.


Can I do this or am I thinking too much outside the box? I tried setting 
a dynamic variable and then setting that in the frontend log-format, but 
it didn’t seem to record anything even despite populating the variable.


You should share your configuration to help diagnose what was wrong in it.
Something like this should work :
http-request set-path /rewritten
http-request set-var(txn.path_rewrite) path

log-format %{+Q}r\ %{+Q}[var(txn.path_rewrite)]


--
Cyril Bonté

[PATCH] DOC: log: more than 2 log servers are allowed

[Cyril Bonté]

Since commit 0f99e3497, loggers are not limited to 2 instances anymore.
---
 doc/configuration.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index d6f8b8d38..c2e44354d 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -818,7 +818,7 @@ group 
   See also "gid" and "user".
 
 log  [len ] [format ]  [max level [min 
level]]
-  Adds a global syslog server. Up to two global servers can be defined. They
+  Adds a global syslog server. Several global servers can be defined. They
   will receive logs for starts and exits, as well as all logs from proxies
   configured with "log global".
 
-- 
2.11.0

[PATCH] BUG/MINOR: force-persist and ignore-persist only apply to backends

[Cyril Bonté]

>From the very first day of force-persist and ignore-persist features,
they only applied to backends, except that the documentation stated it
could also be applied to frontends.

In order to make it clear, the documentation is updated and the parser
will raise a warning if the keywords are used in a frontend section.

This patch should be backported up to the 1.5 branch.
---
 doc/configuration.txt | 8 
 src/cfgparse.c| 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index a914c416c..992e18acc 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -2039,7 +2039,7 @@ errorloc  X  X
 X X
 errorloc302   X  X X X
 -- keyword -- defaults - frontend - listen -- backend -
 errorloc303   X  X X X
-force-persist -  X X X
+force-persist -  - X X
 filter-  X X X
 fullconn  X  - X X
 grace X  X X X
@@ -2052,7 +2052,7 @@ http-response -  X
 X X
 http-reuseX  - X X
 http-send-name-header -  - X X
 id-  X X X
-ignore-persist-  X X X
+ignore-persist-  - X X
 load-server-state-from-file   X  - X X
 log  (*)  X  X X X
 log-formatX  X X -
@@ -3503,7 +3503,7 @@ email-alert to 
 force-persist { if | unless } 
   Declare a condition to force persistence on down servers
   May be used in sections:defaults | frontend | listen | backend
-  no   |yes   |   yes  |   yes
+  no   |no|   yes  |   yes
 
   By default, requests are not dispatched to down servers. It is possible to
   force this using "option persist", but it is unconditional and redispatches
@@ -4825,7 +4825,7 @@ id 
 ignore-persist { if | unless } 
   Declare a condition to ignore persistence
   May be used in sections:defaults | frontend | listen | backend
-  no   |yes   |   yes  |   yes
+  no   |no|   yes  |   yes
 
   By default, when cookie persistence is enabled, every requests containing
   the cookie are unconditionally persistent (assuming the target server is up
diff --git a/src/cfgparse.c b/src/cfgparse.c
index 27d7eee7b..c40e71dc7 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -4068,7 +4068,7 @@ int cfg_parse_listen(const char *file, int linenum, char 
**args, int kwm)
goto out;
}
 
-   if (warnifnotcap(curproxy, PR_CAP_FE|PR_CAP_BE, file, linenum, 
args[0], NULL))
+   if (warnifnotcap(curproxy, PR_CAP_BE, file, linenum, args[0], 
NULL))
err_code |= ERR_WARN;
 
if (strcmp(args[1], "if") != 0 && strcmp(args[1], "unless") != 
0) {
-- 
2.11.0

Re: What is a nice way to bypass the maintenance mode for certain IP's?

[Cyril Bonté]

Hi Willy,

Le 02/03/2018 à 06:32, Willy Tarreau a écrit :

You're right, both are set in the same loop. Usually I prefer to adapt
the code to make it match the doc, but here I don't see a reasonable
way to do it, so I think we'll instead update the code to emit a warning
and update the doc. Any objection ?


With some delay, no objection at all. This is probably the safest option 
as it used to work like this since the beginning. I'm working on the patch.


--
Cyril Bonté

[PATCH] BUG/MEDIUM: fix a 100% cpu usage with cpu-map and nbthread/nbproc

[Cyril Bonté]

Krishna Kumar reported a 100% cpu usage with a configuration using
cpu-map and a high number of threads,

Indeed, this minimal configuration to reproduce the issue :
  global
nbthread 40
cpu-map auto:1/1-40 0-39

  frontend test
bind :8000

This is due to a wrong type in a shift operator (int vs unsigned long int),
causing an endless loop while applying the cpu affinity on threads. The same
issue may also occur with nbproc under FreeBSD. This commit addresses both
cases.

This patch must be backported to 1.8.
---
 src/haproxy.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/haproxy.c b/src/haproxy.c
index 8785b9f94..0c823c497 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -2837,7 +2837,7 @@ int main(int argc, char **argv)
CPU_ZERO();
while ((i = ffsl(cpu_map)) > 0) {
CPU_SET(i - 1, );
-   cpu_map &= ~(1 << (i - 1));
+   cpu_map &= ~(1UL << (i - 1));
}
ret = cpuset_setaffinity(CPU_LEVEL_WHICH, 
CPU_WHICH_PID, -1, sizeof(cpuset), );
}
@@ -3037,7 +3037,7 @@ int main(int argc, char **argv)
 
while ((j = ffsl(cpu_map)) > 0) {
CPU_SET(j - 1, );
-   cpu_map &= ~(1 << (j - 1));
+   cpu_map &= ~(1UL << (j - 1));
}
pthread_setaffinity_np(threads[i],
   sizeof(cpuset), );
-- 
2.11.0

Re: Idle HAProxy 1.8 spins at 100% in user space

[Cyril Bonté]

Le 12/03/2018 à 14:11, Krishna Kumar (Engineering) a écrit :

Hi Cyril,

Thanks, this patch fixes it, it is now back to 0%. Confirmed it a few
times, and undid the patch, back to 100%, and re-added the patch,
back to 0%. Fixes perfectly.


Thanks for confirming, I'm preparing the patch with a reasonable commit 
message. It should be ready in a few minutes ;-)



--
Cyril Bonté

Re: Idle HAProxy 1.8 spins at 100% in user space

[Cyril Bonté]

Hi Krishna and Willy, 

- Mail original -
> De: "Krishna Kumar (Engineering)" 
> À: "HAProxy" 
> Envoyé: Lundi 12 Mars 2018 07:48:50
> Objet: Idle HAProxy 1.8 spins at 100% in user space
> 
> As an aside, could someone also post a simple configuration file to
> enable 40 listeners (thread)?
> 
> I get 100% cpu util when running high number (>30, on a 48 core
> system)
> of threads, I have tried both these versions:
> 
> HA-Proxy version 1.8.4-1ppa1~xenial 2018/02/10: Installed via .deb
> file
> HA-Proxy version 1.8.4-1deb90d 2018/02/08: Built from source
> http://www.haproxy.org/download/1.8/src/haproxy-1.8.4.tar.gz
> 
> 1. Distro/kernel: Ubuntu 16.04.1 LTS, 4.4.0-36-generic
> 
> 
> 2. Top:
> # top -d 1 -b | head -12
> top - 11:59:06 up 4 days, 41 min, 1 user, load average: 1.00, 1.00,
> 2.14
> Tasks: 492 total, 2 running, 464 sleeping, 0 stopped, 26 zombie
> %Cpu(s): 0.5 us, 0.2 sy, 0.0 ni, 99.2 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0
> st
> KiB Mem : 13191999+total, 9520 free, 1222684 used, 52917792
> buff/cache
> KiB Swap: 0 total, 0 free, 0 used. 12986652+avail Mem
> 
> 
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 87994 haproxy 20 0 896624 14996 1468 R 100.0 0.0 3:09.60 haproxy
> 1 root 20 0 38856 7088 4132 S 0.0 0.0 0:08.69 systemd
> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.08 kthreadd
> 3 root 20 0 0 0 0 S 0.0 0.0 4:05.79 ksoftirqd/0
> 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
> 
> 
> 3. As to what it is doing:
> %Cpu0 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
> st
> 
> 
> %Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
> st
> %Cpu2 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
> st
> 
> 
> 4. Minimal configuration file to reproduce this (using this blog:
> https://www.haproxy.com/blog/multithreading-in-haproxy/ ):
> 
> 
> 
> global
> daemon
> nbproc 1
> nbthread 40
> cpu-map auto:1/1-40 0-39

I confirm I can reproduce the issue once 32 (and more) threads are used : the 
main process enters an endless loop.
I think the same issue may occur with nbproc on FreeBSD (the same code in an 
#ifdef FreeBSD__).

Can you try the patch attached ? I'll send a clean one later.

> 
> 
> frontend test-fe
> mode http
> bind 10.33.110.118:80 process all/all
> use_backend test-be
> 
> 
> backend test-be
> mode http
> server 10.33.5.62 10.33.5.62:80 weight 255
> 
> 
> 5. Problem disappears when " cpu-map auto:1/1-40 0-39" is commented
> out.
> Same strace output, so it is in user space as shown by 'top' above.
> 
> 
> 6. Version/build (gcc version 5.4.0 20160609 (Ubuntu
> 5.4.0-6ubuntu1~16.04.2))
> 
> 
> 
> # haproxy -vv
> HA-Proxy version 1.8.4-1deb90d 2018/02/08
> Copyright 2000-2018 Willy Tarreau < wi...@haproxy.org >
> 
> 
> Build options :
> TARGET = linux2628
> CPU = generic
> CC = gcc
> CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
> -fwrapv -Wno-unused-label
> OPTIONS = USE_ZLIB=yes USE_OPENSSL=1 USE_PCRE=1
> 
> 
> Default settings :
> maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents =
> 200
> 
> 
> Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
> Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
> Built with transparent proxy support using: IP_TRANSPARENT
> IPV6_TRANSPARENT IP_FREEBIND
> Encrypted password support via crypt(3): yes
> Built with multi-threading support.
> Built with PCRE version : 8.38 2015-11-23
> Running on PCRE version : 8.38 2015-11-23
> PCRE library supports JIT : no (USE_PCRE_JIT not set)
> Built with zlib version : 1.2.8
> Running on zlib version : 1.2.8
> Compression algorithms supported : identity("identity"),
> deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
> Built with network namespace support.
> 
> 
> Available polling systems :
> epoll : pref=300, test result OK
> poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 3 (3 usable), will use epoll.
> 
> 
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [TRACE] trace
> 
> 
> 7. Strace of the process:
> 88033 11:57:18.946030 <... epoll_wait resumed> [], 200, 1000) = 0
> <1.001144>
> 88032 11:57:18.946046 <... epoll_wait resumed> [], 200, 1000) = 0
> <1.001149>
> 88033 11:57:18.946078 epoll_wait(47, 
> 88034 11:57:18.946092 epoll_wait(48, 
> 88032 11:57:18.946104 epoll_wait(46, 
> 88031 11:57:18.946115 <... epoll_wait resumed> [], 200, 1000) = 0
> <1.001153>
> 88030 11:57:18.946128 <... epoll_wait resumed> [], 200, 1000) = 0
> <1.001154>
> 88029 11:57:18.946140 <... epoll_wait resumed> [], 200, 1000) = 0
> <1.001155>
> 88028 11:57:18.946152 <... epoll_wait resumed> [], 200, 1000) = 0
> <1.001216>
> 88031 11:57:18.946169 epoll_wait(44, 
> 88027 11:57:18.946181 <... epoll_wait resumed> [], 200, 1000) = 0
> <1.001183>
> 88030 11:57:18.946196 epoll_wait(43, 
> 88029 11:57:18.946208 

Re: What is a nice way to bypass the maintenance mode for certain IP's?

[Cyril Bonté]

Hi Pieter and Willy,

Le 01/03/2018 à 16:09, Pieter Vogelaar a écrit :

Hi Willy,

Yes I'm absolutely certain that the cookie is present in the browser request 
when I get the 503.
I changed the JSESSIONID line to "cookie SERVERID insert indirect nocache", but 
that didn't make a difference.

Log line when both servers in backend are in maintenance mode:

172.30.214.130:56887 [01/Mar/2018:15:37:24.928] default-https~ pieter-tomcat-tst/ 
0/-1/-1/-1/0 503 3576 - - SCDN 1/0/0/0/0 0/0 
{tst-pieter.example.org||nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7||Mozilla/5.0 (Macintosh; Intel Mac 
OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36} {|||} 
"GET /hello-world/ HTTP/1.1"

Log line when both servers in backend are active:

172.30.214.130:56451 [01/Mar/2018:15:37:09.734] default-https~ 
pieter-tomcat-tst/pieter-tomcat-01t:8080 0/0/1/2/3 200 5243 - - --VN 2/1/0/0/0 0/0 
{tst-pieter.example.org||nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7||Mozilla/5.0 (Macintosh; 
Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 
Safari/537.36} {text/html;charset=ISO-8859-1|||} "GET /hello-world/ HTTP/1.1"


Well, I think your issue will be resolved by moving "force-persist" on 
the backend side instead of the frontend one.


The issue seems to exist from the first day of "force-persist", where 
the code and the documentation don't agree about the sections where it 
can be used.


From the documentation and the config parser, it can be used in a 
frontend, but the code only loop on the s->be to evaluate the

persistence options.

See commit 
http://git.haproxy.org/?p=haproxy.git;a=commit;h=4de9149f876cc0c63495b71a2c7a3aefc722c9c0 
for details.


The same issue was also introduced with "ignore-persist".




Best regards,
Pieter Vogelaar
  


Op 01-03-18 15:32 heeft Willy Tarreau <w...@1wt.eu> geschreven:

 On Thu, Mar 01, 2018 at 02:29:57PM +, Pieter Vogelaar wrote:
 > Hi Willy,
 >
 > We use Memcached Session Manager that stores the Tomcat sessions to a Couchbase 
cluster. It suffixes the session ID with "-n1" like:
 >
 > JSESSIONID=s01~1C7985929CDF981D9ACC79EBD8A3293D-n1
 >
 > Could this JSESSIONID format somehow have impact on HAProxy?
 
 No, that's fine, but my point is : are you certain it is present in your

 browser's request when you get the 503 ? If the 3 char of the termination
 flags in the logs indicates '-' or 'N', it means it's absent. Or if you
 can run haproxy in debug mode (with -d), you will see all requests on the
 screen and it will be easy to tell whether the cookie is there or not.
 
 Willy
 




--
Cyril Bonté

Re: Using "-f" to load config files with duplicate sections from directory

[Cyril Bonté]

Hi,

Le 03/02/2018 à 12:01, garb...@gmx.de a écrit :

I thought about simplifying my haproxy configurations. I will end up with several servers 
running similar setups. The idea is to split the single "haproxy.cfg" into 
several files and combining common parts into separate files and adding differential 
config files where appropriate.
Running "haproxy -f /var/opt/haproxyconfigfiles" works fine already, but not 
all of my config settings are activated. Some seem to get ignored.

Here is a simplified example of what I am doing:

* 001_globalsettings.cfg:
global
 user ...
 group ...

* 002_defaultssettings.cfg:
defaults
 mode http
 ...

* 003_frontend.cfg:
frontend http_front
bind *:8080
stats uri /haproxy?stats
default_backend http_back

* 020_default_stats.cfg:
defaults
 stats enable
 stats hide-version
 stats refresh 30s
 stats show-node
 stats auth admin:password
 stats uri  /haproxystats

I excluded the backend definitions because I don't think they count here. 
Hopefully this example is concise enough to explain what I am doing.

Reading the docs I didn't find information about in which order haproxy reads the config 
files. Is this a random order or is the order I specified by prefixing with 
"001" considered ?


No need to look at the source code (as said below), all the answers are 
already in the documentation ;-)


About the order, this is explained in the "management" documentation :
http://cbonte.github.io/haproxy-dconv/1.8/management.html
"[...] If  is a directory, all the files (and only files) it 
contains are added in lexical order (using LC_COLLATE=C) to the list of 
configuration files to be loaded[...]"




As you can see I have the "stats uri" multiple times, but only "/haproxy?stats" 
is used. This could be explained by a random order.
But "stats auth" is not considered at all and version information is shown dispite of "stats 
hide-versions". This could mean that "defaults" from one file is being overwritten completely 
by a different file.


That's right, a "defaults" section is designed to reset the values of 
any previous "defaults sections. And those values will only apply to the 
*following* sections, not to the whole configuration :

http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4

"[...]A "defaults" section sets default parameters for all other 
sections following its declaration. Those default parameters are reset 
by the next "defaults" section.[...]"


It works the same, using one configuration file or many.


Can someone explain to me how haproxy is working ? I would rather not sift 
through the sourcecode ;-)


--
Cyril Bonté

Re: redispatch still cause error response

[Cyril Bonté]

Hi,

Le 31/01/2018 à 03:00, flamese...@yahoo.co.jp a écrit :

Hello,

What exactly does option redispatch do?


It gives a chance to retry on another server if the connection can't be 
established on the one chosen by the loadbalancing algorithm. It's 
important to understand that it concerns only the connection 
establishment step. See below for details.


I have a haproxy in front of two web servers, if one web server get 
killed when haproxy is forwarding a request to that server,


will this request get re-forwarded to another server?

So I had an test.

In one server, I run

ruby -run -ehttpd . -p8001

and

ruby -run -ehttpd . -p8000

and i start haproxy

then do a ab

ab -n 2 -c 10 http://127.0.0.1:8080/

while the ab is running, I killed 8001 ruby, then after some time I 
started it again.


here is the ab result:

Complete requests:  2
Failed requests:    5
    (Connect: 0, Receive: 0, Length: 5, Exceptions: 0)


And haproxy.log

grep -c GET haproxy.log
2

grep GET haproxy.log | grep -v 200
Jan 31 10:48:12 localhost haproxy[5948]: 127.0.0.1:45516 
[31/Jan/2018:10:48:12.386] web1 app1/b 0/0/0/-1/5 -1 0 - - SD-- 
10/10/8/3/0 0/0 "GET / HTTP/1.0"
Jan 31 10:48:12 localhost haproxy[5948]: 127.0.0.1:45538 
[31/Jan/2018:10:48:12.391] web1 app1/b 0/0/1/-1/1 -1 0 - - SD-- 
10/10/8/4/0 0/0 "GET / HTTP/1.0"
Jan 31 10:48:12 localhost haproxy[5948]: 127.0.0.1:45528 
[31/Jan/2018:10:48:12.389] web1 app1/b 0/0/1/-1/3 -1 0 - - SD-- 
9/9/8/3/0 0/0 "GET / HTTP/1.0"
Jan 31 10:48:12 localhost haproxy[5948]: 127.0.0.1:45524 
[31/Jan/2018:10:48:12.388] web1 app1/b 0/0/1/-1/4 -1 0 - - SD-- 
8/8/7/2/0 0/0 "GET / HTTP/1.0"
Jan 31 10:48:12 localhost haproxy[5948]: 127.0.0.1:45544 
[31/Jan/2018:10:48:12.392] web1 app1/b 0/0/0/-1/1 -1 0 - - SD-- 
7/7/6/0/0 0/0 "GET / HTTP/1.0"


Here, your logs indicate that the connection was already established and 
the request was also already sent to the server app1/b, but the 
connection was dropped in the middle (the server was stopped in the 
middle as you previously said). It's too late to redispatch a 
connection, so the behaviour is expected.




Here is the config:

global
     maxconn 5000
log 127.0.0.1 local2 debug
     daemon
defaults
     log global
     option httplog
     retries 3
     option redispatch
     timeout client  30s
     timeout connect 30s
     timeout server  30s
     timeout http-request 30s
     timeout http-keep-alive 30s
frontend web1
     bind *:8080
     mode http
     default_backend app1
backend app1
     balance roundrobin
     mode http
     server a 127.0.0.1:8000 check
     server b 127.0.0.1:8001 check

I have tried v1.8.3 and v1.7.10

And I missing something?

Thanks



--
Cyril Bonté

Re: Re[4]: How to parse custom PROXY protocol v2 header for custom routing in HAProxy configuration?

[Cyril Bonté]

Hi,

- Mail original -
> De: "Aleksandar Lazic" 
> À: haproxy@formilux.org
> Envoyé: Lundi 22 Janvier 2018 13:34:33
> Objet: Re[4]: How to parse custom PROXY protocol v2 header for custom routing 
> in HAProxy configuration?
> 
> Hi.
> 
> Have anyone a Idea how haproxy can handle the custom TLV in the proxy
> protocol v2

Currently, it can't. Only PP2_TYPE_NETNS is supported.
But some work can be done to, at least, support some other predefined fields, 
or even better, to provide a generic way to capture any type of field.

You can have a look at the function conn_recv_proxy() in src/connection.c :
http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/connection.c;h=0f8acb02dbdbc0a70cdd99830f8a0c9256f731e8;hb=HEAD#l604

Cyril

Re: [BUG] 100% cpu on each threads

[Cyril Bonté]

Hi all,

- Mail original -
> De: "Willy Tarreau" 
> À: "Emmanuel Hocdet" 
> Cc: "haproxy" 
> Envoyé: Vendredi 12 Janvier 2018 15:24:54
> Objet: Re: [BUG] 100% cpu on each threads
> 
> On Fri, Jan 12, 2018 at 12:01:15PM +0100, Emmanuel Hocdet wrote:
> > When syndrome appear, i see such line on syslog:
> > (for one or all servers)
> > 
> > Server tls/L7_1 is DOWN, reason: Layer4 connection problem, info:
> > "Bad file descriptor", check duration: 2018ms. 0 active and 1
> > backup servers left. Running on backup. 0 sessions active, 0
> > requeued, 0 remaining in queue.
> 
> So I tried a bit but found no way to reproduce this. I'll need some
> more info like the type of health-checks, probably the "server" line
> settings, stuff like this. Does it appear quickly or does it take a
> long time ? Also, does it recover from this on subsequent checks or
> does it stay stuck in this state ?

Im' not sure you saw Samuel Reed's mail.
He reported a similar issue some hours ago (High load average under 1.8 with 
multiple draining processes). It would be interesting to find a common 
configuration to reproduce the issue, so I add him to the thread.

Cyril

Re: disable-on-404 functionality change in 1.8

[Cyril Bonté]

Hi Paul,

I add Emeric to the thread.

Le 23/12/2017 à 02:41, Paul Lockaby a écrit :

Hello, I recently upgraded to 1.8.1 and noticed a pretty big functional 
difference that I can't attribute to any listed change in the changelog. Here 
is my configuration:


global
 log /dev/log local0
 user nobody
 group nobody

defaults
 # set timeout to ten minutes
 timeout connect 5000ms
 timeout client 60ms
 timeout server 60ms

 option httplog
 option forwardfor
 option http-server-close
 option contstats

frontend unsecured
 bind *:80
 mode http
 log global
 redirect scheme https code 301 if !{ ssl_fc }

frontend secured
 bind *:443 ssl crt /usr/local/ssl/certs/example.com.pem
 mode http
 log global
 default_backend example-http

backend example-http
 mode http
 log global
 balance source
 hash-type consistent
 option httpchk GET /haproxy/alive.txt
 http-check disable-on-404
 server example01 example01.com:8443 check ssl ca-file 
/usr/local/ssl/certs/cacerts.cert
 server example02 example02.com:8443 check ssl ca-file 
/usr/local/ssl/certs/cacerts.cert
 server example03 example03.com:8443 check ssl ca-file 
/usr/local/ssl/certs/cacerts.cert


 From 1.7 to 1.8 the functionality in "http-check disable-on-404" seems to have 
changed.

On 1.7.9, this configuration would cause a backend server to become unavailable if the path 
"/haproxy/alive.txt" returned a 404. The host would be marked "active or backup SOFT 
STOPPED for maintenance" and it would no longer be given new requests.

On 1.8.1, when "/haproxy/alive.txt" returns a 404 the host STAYS in the rotation and continues to 
receive new requests! I am able to work around it by removing "http-check disable-on-404" and then 
when the file disappears the host just goes to "active or backup DOWN". But the change from 1.7 to 
1.8 was jarring and surprising and I did not see anything in the changelog indicating that this is supposed 
to have changed like this. Was I doing it wrong from the beginning?


It looks to be a code regression.

Emeric, can you have a look at commit 5a1335110c ? It seems there was an 
unwanted change in the function call : srv_set_stopping() was replaced 
by srv_set_running()

[...]
 /* Marks the check  as valid and tries to set its server into 
stopping mode

@@ -406,7 +371,7 @@ static void check_notify_stopping(struct check *check)
 	if ((s->agent.state & CHK_ST_ENABLED) && (s->agent.health < 
s->agent.rise))

return;

-	srv_set_stopping(s, (!s->track && !(s->proxy->options2 & 
PR_O2_LOGHCHKS)) ? check_reason_string(check) : NULL);
+	srv_set_running(s, NULL, (!s->track && !(s->proxy->options2 & 
PR_O2_LOGHCHKS)) ? check : NULL);

 }
[...]

Thanks ;-)

--
Cyril Bonté

Re: Traffic delivered to disabled server when cookie persistence is enabled after upgrading to 1.8.1

[Cyril Bonté]

Hi all,

Le 21/12/2017 à 15:25, Willy Tarreau a écrit :

On Thu, Dec 21, 2017 at 02:53:07PM +0100, Emeric Brun wrote:

Hi All,

This bug should be fixed using this patch (patch on dev, abd should be 
backported in 1.8).


now applied to both branches,, thanks!
Willy


And after performing the same tests with the patch applied, I can 
confirm I don't reproduce the issue anymore ;-)


--
Cyril Bonté

Re: Traffic delivered to disabled server when cookie persistence is enabled after upgrading to 1.8.1

[Cyril Bonté]

Hi Greg,

Le 20/12/2017 à 22:42, Greg Nolle a écrit :

Hi Andrew,

Thanks for the info but I’m afraid I’m not seeing anything here that 
would affect the issue I’m seeing, and by the way the docs don’t 
indicate that the cookie names have to match the server names.


First, don't worry about the configuration, there is nothing wrong in it ;-)

That being said, I tried using your settings and am still seeing the 
issue (see below for new full config). And like I say, this is only an 
issue with v1.8.1, it works as expected in v1.7.9.


I won't be able to look further tonight, but at least I could identify 
when the regression occured : it's caused by the work done to prepare 
multi-threading, more specifically by this commit : 
http://git.haproxy.org/?p=haproxy.git;a=commitdiff;h=64cc49cf7


I add Emeric to the thread, maybe he'll be able to provide a fix faster 
than me (I'll won't be very available for the next days).


--
Cyril Bonté

[PATCH] BUG: MINOR: http: don't check http-request capture id when len is provided

[Cyril Bonté]

Randomly, haproxy could fail to start when a "http-request capture"
action is defined, without any change to the configuration. The issue
depends on the memory content, which may raise a fatal error like :
  unable to find capture id '' referenced by http-request capture
rule

Commit fd608dd2 already prevents the condition to happen, but this one
should be included for completeness and to reclect the code on the
response side.

The issue was introduced recently by commit 29730ba5 and should only be
backported to haproxy 1.8.
---
 src/proto_http.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/src/proto_http.c b/src/proto_http.c
index 1330ac864..481688fdf 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -12149,7 +12149,6 @@ enum act_parse_ret parse_http_req_capture(const char 
**args, int *orig_arg, stru
 
rule->action   = ACT_CUSTOM;
rule->action_ptr   = http_action_req_capture;
-   rule->check_ptr= check_http_req_capture;
rule->arg.cap.expr = expr;
rule->arg.cap.hdr  = hdr;
}
-- 
2.11.0

Re: [PATCH] Minor : converter: Add len converter

[Cyril Bonté]

Le 14/12/2017 à 19:10, Willy Tarreau a écrit :

If you want to make a patch for this one, in parallel I'm checking
sample_parse_expr().


Yes, I was ready to provide the patch but I failed to reproduce the 
issue today. After some hours, I realized that there was a recent (well, 
10 days ago) patch from Christopher that seems to prevent the condition 
to happen :

http://git.haproxy.org/?p=haproxy.git;a=commit;h=fd608dd2d212928e585da645d7d0c7089424a0de

My patch is less useful now but I think it should be applied too to be 
complete (this will also align the request code with the response one).


--
Cyril Bonté

Re: [PATCH] Minor : converter: Add len converter

[Cyril Bonté]

Le 14/12/2017 à 19:24, Willy Tarreau a écrit :

So I think we have two action plans for now :
   1) find another (ideally short) name for "len" which doesn't match "len"
  nor "id" nor "table" nor "if" nor "unless". Etienne initially proposed
  "strlen" but I thought that it's a bit long for such a common use case.
  Ideas welcome.


Unfortunately, for now we have to rename it. But I'm fine with "strlen", 
which follows the description in the documentation.



   2) kill this outdated way of unreliably parsing expressions among multiple
  words for 1.9. It even prevents us from improving the sample expressions
  language (ie: I'd love to have a stack-based language but we can do
  nothing if we can't even add reliable operators).


Yes, it was difficult to understand the goal of each part of this 
function and I fear that any minimal modification inside it can 
introduce side effects on one feature or another ;-/


--
Cyril Bonté

Re: [PATCH] Minor : converter: Add len converter

[Cyril Bonté]

Hi Willy,

Le 14/12/2017 à 14:36, Willy Tarreau a écrit :

On Wed, Dec 13, 2017 at 02:56:34PM +0100, Etienne Carrière wrote:

Hi,

You will find attached a patch to add a converter that can returns the
len of a string.


Thank you Etienne, now applied.


It's embarrassing, I was working on a patch for a nasty bug I've seen 
several times last week (randomly haproxy won't start), but now I've 
updated to the current master branch, it's even worse with this specific 
patch.


Example :
  listen capture
bind :9000
mode http
http-request capture hdr(X-Forwarded-Proto) len 8

With the new "len" converter, it fails with :
  parsing [capture.cfg:4] : error detected in proxy 'capture' while 
parsing 'http-request capture' rule : expects 'len' or 'id', found '8'.


It seems that we have to fix sample_parse_expr(). Currently, be careful 
if you want to backport it.


Otherwise, my original bug concerns the exact same configuration and 
looks to exist for a long time : there is an unwanted "check_ptr" 
assignment in parse_http_req_capture() when the "len" parameter is set 
instead of "id".


diff --git a/src/proto_http.c b/src/proto_http.c
index 1330ac864..481688fdf 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -12149,7 +12149,6 @@ enum act_parse_ret parse_http_req_capture(const 
char **args, int *orig_arg, stru


rule->action   = ACT_CUSTOM;
rule->action_ptr   = http_action_req_capture;
-   rule->check_ptr= check_http_req_capture;
rule->arg.cap.expr = expr;
    rule->arg.cap.hdr  = hdr;
}


--
Cyril Bonté

[PATCH] BUG: MAJOR: lb_map: server map calculation broken

[Cyril Bonté]

Adrian Williams reported that several balancing methods were broken and
sent all requests to one backend. This is a regression in haproxy 1.8 where
the server score was not correctly recalculated.

This fix must be backported to the 1.8 branch.
---
 src/lb_map.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/lb_map.c b/src/lb_map.c
index ecab4de13..a1e1d3492 100644
--- a/src/lb_map.c
+++ b/src/lb_map.c
@@ -122,7 +122,7 @@ void recalc_server_map(struct proxy *px)
}
}
px->lbprm.map.srv[o] = best;
-   HA_ATOMIC_ADD(>wscore, tot);
+   HA_ATOMIC_SUB(>wscore, tot);
}
 }
 
-- 
2.11.0

Re: BUG: 1.8.x Several balancing methods only send requests to one backend.

[Cyril Bonté]

Le 14/12/2017 à 12:13, Adrian Williams a écrit :

Can confirm that it has fixed my problem. static-rr is working as
expected after patching.


Thanks for confirming it fixed the issue, I'm preparing the final patch 
in order to include it in 1.9 and 1.8 branches.




On 14 December 2017 at 19:53, Cyril Bonté <cyril.bo...@free.fr> wrote:

Hi Adrian,

Le 14/12/2017 à 00:08, Adrian Williams a écrit :


Several backend balancing methods seem to be broken with the 1.8
releases. All traffic is directed to one backend even when multiple
are listed. This happens on both 1.8 and 1.8.1. Changing the balance
to roundrobin distributes requests as expected.

Balancing modes affected: source, URL, static-rr



I can confirm the issue, it seems there was a typo when HA_ATOMIC_* macros
were introduced. This small patch should fix the issue :
diff --git a/src/lb_map.c b/src/lb_map.c
index ecab4de13..a1e1d3492 100644
--- a/src/lb_map.c
+++ b/src/lb_map.c
@@ -122,7 +122,7 @@ void recalc_server_map(struct proxy *px)
 }
 }
 px->lbprm.map.srv[o] = best;
-   HA_ATOMIC_ADD(>wscore, tot);
+   HA_ATOMIC_SUB(>wscore, tot);
 }
  }

Do you have the possibility to test it ?

--
Cyril Bonté



--
Cyril Bonté

cache issue : some data skipped in the middle of the content

[Cyril Bonté]

Hi all,

Late this night, I had an issue I didn't met during my previous tests, related 
to the cache feature : sometimes, I had some corrupted images in the browser.
After adding some debug messages, it seems there's an issue in 
cache_store_http_forward_data() when there is not enough contiguous space 
available.
For example, with an image of 6532 bytes :
- first, the headers are captured = 287 bytes
- then, it tries to fill the cache but can only read 2609 bytes, skipping the 
first 287 bytes to ignore headers
- cache_store_http_forward_data() has to be called a second time, that's where 
I think the issue occurs : it will bypass 287 bytes of headers again, while it 
shouldn't => only the last 3636 bytes will be read instead of the remaining 
3923 ones :
[code]
/* Skip remaining headers to fill the cache */
b_adv(msg->chn->buf, st->hdrs_len);
ret = shctx_row_data_append(shctx,
st->first_block,
(unsigned char 
*)bi_ptr(msg->chn->buf),

MIN(bi_contig_data(msg->chn->buf), len - st->hdrs_len));
/* Rewind the buffer to forward all data */
b_rew(msg->chn->buf, st->hdrs_len);
[/code]

I won't be able to make more tests before this evening, but I hope there is 
enough details ;-)

Also, I've not checked yet but I was wondering : is it specific to the cache 
filter or can we find the same issue in other ones (compression for example) ?

Cyril

[PATCH] DOC: cache: update sections and fix some typos

[Cyril Bonté]

Cache sections were not defined as the others, preventing them to be
correctly parsed by the HTML converter. Also, the "Cache" subsections
where not added to the summary.

This patch should be backported to the 1.8 branch.
---
 doc/configuration.txt | 26 +++---
 doc/management.txt|  2 +-
 2 files changed, 16 insertions(+), 12 deletions(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index d71ba3049..36a904e25 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -109,6 +109,10 @@ Summary
 9.3.  Stream Processing Offload Engine (SPOE)
 
 10.   Cache
+10.1. Limitation
+10.2. Setup
+10.2.1. Cache section
+10.2.2. Proxy section
 
 1. Quick reminder about HTTP
 
@@ -16990,13 +16994,13 @@ HAProxy provides a cache, which was designed to 
perform cache on small objects
 RAM.
 
 The cache is based on a memory which is shared between processes and threads,
-this memory is splitted in blocks of 1k.
+this memory is split in blocks of 1k.
 
 If an object is not used anymore, it can be deleted to store a new object
 independently of its expiration date. The oldest objects are deleted first
 when we try to allocate a new one.
 
-The cache use a hash of the host header and the URI as the key.
+The cache uses a hash of the host header and the URI as the key.
 
 It's possible to view the status of a cache using the Unix socket command
 "show cache" consult section 9.3 "Unix Socket commands" of Management Guide
@@ -17005,8 +17009,8 @@ for more details.
 When an object is delivered from the cache, the server name in the log is
 replaced by "".
 
-10.1 Limitation

+10.1. Limitation
+
 
 The cache won't store and won't deliver objects in these cases:
 
@@ -17022,16 +17026,16 @@ The cache won't store and won't deliver objects in 
these cases:
 
 Caution!: Due to the current limitation of the filters, it is not recommended
 to use the cache with other filters. Using them can cause undefined behavior
-if they modify the response (compression for exemple).
+if they modify the response (compression for example).
 
-10.2 Setup
---
+10.2. Setup
+---
 
 To setup a cache, you must define a cache section and use it in a proxy with
 the corresponding http-request and response actions.
 
-10.2.1 Cache section
-
+10.2.1. Cache section
+-
 
 cache 
   Declare a cache section, allocate a shared cache memory named , the
@@ -17048,8 +17052,8 @@ max-age 
   seconds, which means that you can't cache an object more than 60 seconds by
   default.
 
-10.2.2 Proxy section
-
+10.2.2. Proxy section
+-
 
 http-request cache-use 
   Try to deliver a cached object from the cache . This directive is also
diff --git a/doc/management.txt b/doc/management.txt
index 2161b018f..3240505b4 100644
--- a/doc/management.txt
+++ b/doc/management.txt
@@ -1755,7 +1755,7 @@ show cli sockets
  [::1]: operator 2
 
 show cache
-  List the configurated caches and the objects stored in each cache tree.
+  List the configured caches and the objects stored in each cache tree.
 
   $ echo 'show cache' | socat stdio /tmp/sock1
   0x7f6ac6c5b03a: foobar (shctx:0x7f6ac6c5b000, available blocks:3918)
-- 
2.11.0

haproxy 1.8-rc4: cache and HTTP/1.1 response for HTTP/1.0 request

[Cyril Bonté]

Hi Willy and William,

I ran some tests with the cache filter.

In http_action_store_cache(), the code indicates that only HTTP/1.1 is 
cached. This explains why I failed on my first tests with apachebench :)
The protocol version is checked on the request side. Can't we rely on 
the response side instead ?


Btw, here are some first results for those who are interested :

* WITHOUT cache

- ab -n10 -c100 http://localhost/image.gif
Time taken for tests:   59.127 seconds
Requests per second:1691.27 [#/sec] (mean)
Time per request:   59.127 [ms] (mean)
Time per request:   0.591 [ms] (mean, across all concurrent requests)
Transfer rate:  1344.43 [Kbytes/sec] received

- h2load -n10 -c100 https://localhost/image.gif
finished in 60.06s, 1664.91 req/s, 1.11MB/s

* WITH cache :

- ab -n10 -c100 http://localhost/image.gif
Same results as before, but once patched to rely on the response, we get 
more interesting results :

Time taken for tests:   1.801 seconds
Requests per second:55539.79 [#/sec] (mean)
Time per request:   1.801 [ms] (mean)
Time per request:   0.018 [ms] (mean, across all concurrent requests)
Transfer rate:  44149.79 [Kbytes/sec] received

- h2load -n10 -c100 https://localhost/image.gif
finished in 1.49s, 67210.04 req/s, 44.80MB/s

for some details :
- image.gif = 510 bytes
- haproxy runs locally (the backend is in the same network) with this 
configuration :

global
nbthread 4
tune.ssl.default-dh-param 2048
log /dev/log local7 info err
stats socket /tmp/proxy.socket level admin

defaults
timeout client 300s
timeout server 300s
timeout connect 5s
timeout http-keep-alive 5s

listen proxy
mode http

bind :80
bind :443 ssl crt localhost.pem alpn h2,http/1.1

option httplog

http-response cache-store proxy
http-request cache-use proxy

server real backend:80

cache proxy
total-max-size 4


--
Cyril Bonté

Re: log-format in defaults section in 1.7

[Cyril Bonté]

Hi Thayne,

Le 02/11/2017 à 23:08, Thayne McCombs a écrit :
So, I looked into using `no log` in non http frontends. But that isn't 
sufficient.


For example, if I have:

global
   log-tag "test"
   log localhost:514 len 65535 local2 info info

defaults
   mode http
   timeout connect 100
   timeout server 3
   timeout client 3
   log-format "%Tq"

listen mine
   log global
   bind :80
   server localhost localhost:8080

listen health_url
   bind :27000
   mode health
   option httpchk
   no log


I still get [ALERT] 305/160229 (21975) : Parsing [test.cfg:10]: failed 
to parse log-format : format variable 'Tq' is reserved for HTTP mode.


You can specify several "defaults" sections in your configuration : one 
for http, and one for tcp frontends.


global
  log-tag "test"
  log localhost:514 len 65535 local2 info info

defaults
  mode http
  timeout connect 100
  timeout server 3
  timeout client 3
  log-format "%Tq"

listen mine
  log global
  bind :8080
  server localhost localhost:80

# ...
# Other HTTP frontends
# ...

defaults
  mode tcp
  timeout connect 100
  timeout server 3
  timeout client 3

listen health_url
  bind :27000
  mode health
  option httpchk

# ...
# Other TCP frontends
# ...


However, if I add `log-format "GARBAGE"` to the health_url listener, 
then the error goes away.


Or you can specify "option tcplog" in your "health_url" section (or any 
other tcp sections).



--
Cyril Bonté

Re: [ANNOUNCE] haproxy-1.8-rc1 : the last mile

[Cyril Bonté]

Hi all,

Le 01/11/2017 à 00:20, Willy Tarreau a écrit :

[...]
So what do we have here ? A *preview* of what the final 1.8 will be like.
[...]
Please find the usual URLs below :
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/


This announcement was exciting enough to take some time to regenerate an 
up to date HTML documentation ! 1.8-rc1 is now available : 
http://cbonte.github.io/haproxy-dconv/1.8/configuration.html



Have fun,
Willy -- feeling exhausted like a marathoner :-)


Great job ! Now it's time to test and track nasty bugs before the final 
1.8 release ;-)



--
Cyril Bonté

Re: cppcheck finding

[Cyril Bonté]

Hi all,

Le 15/09/2017 à 18:40, Willy Tarreau a écrit :

I once had an interesting discussion with PHK who proposed to extend
the varnish test program to also cover haproxy so that we could write
various test cases, as he wrote this tool to address exactly the same
issue. It could be an option, but I didn't have time (nor do I now) to
look into that.


I also wanted to explore that possibility, but unfortunately I'm not 
able to find time yet, too :-/


I had some ideas to automatically generate test configurations, 
randomizing keyword orders for example, to validate the config parser, 
the expected behaviour and to ensure there is no regression between 
versions. The only issue is that it requires some time :)



--
Cyril Bonté

Re: Error on "tcp-check connect port 3389 ssl" in config

[Cyril Bonté]

Hi Thomas,

> De: "Thomas Schweikle" <tschwei...@gmail.com>
> À: haproxy@formilux.org
> Envoyé: Jeudi 31 Août 2017 12:48:39
> Objet: Error on "tcp-check connect port 3389 ssl" in config
> 
> Hi!
> 
> Trying to configure haproxy to act as a connection broker and load
> balancer for RDP (aka Microsoft Terminal Services).
> 
> While configuring the backend, I found haproxy choke on
> 
> tcp-check connect port 3389 ssl
> 
> with message:
> 
> [ALERT] 242/124152 (6066) : parsing [/etc/haproxy/haproxy.cfg:33] :
> 'tcp-check connect' expects 'comment', 'port', 'send-proxy' or but
> got
> 'ssl' as argument.
> 
> Within the handbook I find (section "tcp-check connect"):
> 
> tcp-check connect port 443 ssl
> 
> If it chokes on this -- is it a bug? Or anything else?
> 
> # haproxy -v
> HA-Proxy version 1.7.9 2017/08/18
> Copyright 2000-2017 Willy Tarreau <wi...@haproxy.org>

I guess that you compiled haproxy yourself and forgot to enable the SSL support.
If you run the following command :
  haproxy -vv
it will probably show you that haproxy was compiled without SSL support.

Cyril Bonté

Re: AW: Enable SSL Forward Secrecy

[Cyril Bonté]

> De: "Julian Zielke" <jzie...@next-level-integration.com>
> À: "Cyril Bonté" <cyril.bo...@free.fr>
> Cc: haproxy@formilux.org
> Envoyé: Mercredi 30 Août 2017 15:11:47
> Objet: AW: Enable SSL Forward Secrecy
> 
> Hi Cyril,
> 
> tired it without success. Maybe HaProxy isn't just capable of doing
> this.

Oh well, indeed the "!kECDHE" excludes the ciphers from the list.
You should retry without it (with or without RFC names in the ciphers list)

> > ssl-default-bind-ciphers
> > TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA:TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA:
> > TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA:AES256+EECDH:AES256+EDH:TLSv1+HIGH
> > :!aNULL:!eNULL:!3DES:!RC4:!CAMELLIA:!DH:!kECDHE:@STRENGTH:!DHE

Cyril Bonté

Re: Enable SSL Forward Secrecy

[Cyril Bonté]

Hi Julian,

> De: "Julian Zielke" <jzie...@next-level-integration.com>
> Hi,
> 
> I’m struggeling with enabling SSL forward secrecy in my haproxy 1.7
> setup.
> 
> So far the global settings look like:
> 
> tune.ssl.default-dh-param 2048 # tune shared secred to 2048bits

> ssl-default-bind-options force-tlsv12 no-sslv3
> ssl-default-bind-ciphers
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA:TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA:TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA:AES256+EECDH:AES256+EDH:TLSv1+HIGH:!aNULL:!eNULL:!3DES:!RC4:!CAMELLIA:!DH:!kECDHE:@STRENGTH:!DHE

Please retry by replacing the RFC names with the openssl ones.
Look at this page for details : 
https://wiki.openssl.org/index.php/Manual:Ciphers(1)

For example with :
ssl-default-bind-ciphers 
ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256+EECDH:AES256+EDH:TLSv1+HIGH:!aNULL:!eNULL:!3DES:!RC4:!CAMELLIA:!DH:!kECDHE:@STRENGTH:!DHE

I think that with this ciphers list, ECHDE ones should now be available.

Cyril Bonté

Re: Removed health check in combination with load-server-state-from-file (Bug)

[Cyril Bonté]

> De: "Julien Laffaye" 
> Hello Cyril,
> 
> 
> Well, you can also think of this use case which I find legit:
> - you don't have probe
> - you want to have probe so you add one and reload haproxy
> - you realize your probe does not do want you intended, your backends
> are turning down
> - you panic, you decide to rollback to the previous working
> configuration and reload haproxy
> - it still does not work
> - you panic :)
> 
> 
> How can we make this rollback-to-previous-configuration works in this
> case ?

Isn't the state file part of the configuration? :-)

Re: Removed health check in combination with load-server-state-from-file (Bug)

[Cyril Bonté]

> De: "Arnaud Jost" <arn...@jost.me>
> À: haproxy@formilux.org
> Envoyé: Mercredi 30 Août 2017 10:01:37
> Objet: RE: Removed health check in combination with 
> load-server-state-from-file (Bug)
> 
> 
> 
> 
> Hello Cyril,
> 
> 
> This also can be achieved by using 'disabled' keyword on server, and
> update CLI to enable it. Are you sure that using server-state file
> to keep server DOWN from previous health check is the expected
> behaviour ?

I am sure ;-)

> May be i'm wrong, but when i have a server in DOWN state because of
> health check, if a disable health check on it, i expected the server
> to be UP.

Well health checks or not, the state file reflects the state you want.
You can absolutely imagine cases where health checks were never activated, and 
someone used the CLI to set a server DOWN. If you save the servers state and 
reload haproxy, You'd certainly want to have that server still DOWN (which is a 
different state than DOWN for maintenance, the one set by the "disabled" 
keyword).

Please avoid top posting ;-)

> 
> 
> 
> 
> 
> 
> 
> 
> De : Cyril Bonté <cyril.bo...@free.fr>
> Envoyé : mardi 29 août 2017 00:12
> À : Julien Laffaye; Tim Düsterhus
> Cc : haproxy@formilux.org
> Objet : Re: Removed health check in combination with
> load-server-state-from-file (Bug)
> 
> Hi Julien and Tim,
> 
> Le 28/08/2017 à 10:32, Julien Laffaye a écrit :
> > Hello,
> > 
> > I am experiencing the same problem.
> > Is this the expected behaviour ? Or is it a bug ?
> 
> Yes, it's expected.
> One use case is to start all servers in a DOWN state then
> programmatically set one or several of them UP once everything is
> initialized in the architecture, using the CLI command "set server
> / health up".
> 
> > 
> > Regards,
> > Julien
> > 
> > On Sat, Aug 26, 2017 at 2:55 AM, Tim Düsterhus <t...@bastelstu.be
> > < mailto:t...@bastelstu.be >> wrote:
> > 
> > Hi
> > 
> > as I did not receive any reply at all to my email from Aug 13 I
> > thought
> > I resend it (Quoted below). Can anyone at least verify that my bug
> > report is valid? :-)
> > 
> > Tim
> > 
> > Am 13.08.2017 um 13:19 schrieb Tim Düsterhus:
> > > Hi
> > > 
> > > I run haproxy with 'load-server-state-from-file'. Before
> > > reloading
> > > haproxy I dump the state using:
> > > 
> > > echo show servers state |nc -U admin.sock >
> > > /etc/haproxy/state/global
> > > 
> > > I noticed a buggy behaviour with this:
> > > 
> > > 1. Check that the backend is 'DOWN'.
> > > 2. Dump the state using the command above (the 'DOWN' state is
> > written
> > > into the file).
> > > 3. Remove the health check of the backend.
> > > 4. Reload haproxy.
> > > 5. The backend will now be 'DOWN' forever, as the initial state
> > > taken
> > > from the file is 'DOWN' and no health checks are running.
> > > 
> > > I attached an example configuration and an example state file. To
> > > reproduce the issue:
> > > 
> > > 1. Start haproxy.
> > > 2. Open the Stats page.
> > > 3. Place the state file.
> > > 4. Remove the 'check' from the configuration.
> > > 5. Reload haproxy.
> > > 6. Start the backend.
> > > 7. Reload the Stats page and notice that the backend still is
> > > 'DOWN'.
> > > 
> > > Tim
> > > 
> > 
> > 
> 
> 
> --
> Cyril Bonté
> 
> 

Re: Removed health check in combination with load-server-state-from-file (Bug)

[Cyril Bonté]

Hi Julien and Tim,

Le 28/08/2017 à 10:32, Julien Laffaye a écrit :

Hello,

I am experiencing the same problem.
Is this the expected behaviour ? Or is it a bug ?


Yes, it's expected.
One use case is to start all servers in a DOWN state then 
programmatically set one or several of them UP once everything is 
initialized in the architecture, using the CLI command "set server 
/ health up".




Regards,
Julien

On Sat, Aug 26, 2017 at 2:55 AM, Tim Düsterhus <t...@bastelstu.be 
<mailto:t...@bastelstu.be>> wrote:


Hi

as I did not receive any reply at all to my email from Aug 13 I thought
I resend it (Quoted below). Can anyone at least verify that my bug
report is valid? :-)

Tim

Am 13.08.2017 um 13:19 schrieb Tim Düsterhus:
 > Hi
 >
 > I run haproxy with 'load-server-state-from-file'. Before reloading
 > haproxy I dump the state using:
 >
 > echo show servers state |nc -U admin.sock > /etc/haproxy/state/global
 >
 > I noticed a buggy behaviour with this:
 >
 > 1. Check that the backend is 'DOWN'.
 > 2. Dump the state using the command above (the 'DOWN' state is
written
 > into the file).
 > 3. Remove the health check of the backend.
 > 4. Reload haproxy.
 > 5. The backend will now be 'DOWN' forever, as the initial state taken
 > from the file is 'DOWN' and no health checks are running.
 >
 > I attached an example configuration and an example state file. To
 > reproduce the issue:
 >
 > 1. Start haproxy.
 > 2. Open the Stats page.
 > 3. Place the state file.
 > 4. Remove the 'check' from the configuration.
 > 5. Reload haproxy.
 > 6. Start the backend.
 > 7. Reload the Stats page and notice that the backend still is 'DOWN'.
 >
 > Tim
 >





--
Cyril Bonté

Re: req.cook_cnt() broken?

[Cyril Bonté]

Hi Daniel,

Le 23/08/2017 à 11:50, Daniel Schneller a écrit :

Kindly bumping this during the summer vacation time for potentially new 
recipients :)



On 21. Aug. 2017, at 21:14, Daniel Schneller 
<daniel.schnel...@centerdevice.com> wrote:
Hi!

According to the documentation

  req.cook_cnt([]) : integer
  Returns an integer value representing the number of occurrences of the cookie
   in the request, or all cookies if  is not specified.

it should be possible to do something like this to reject a request if it contains 
more than  cookies total. I do not know the cookie names in advance. I am 
trying to reject malicious requests with hundreds or thousands of cookies, trying to 
exhaust memory in my backend servers. Tomcat has a maximum number of cookies per 
request setting, but I’d like to reject these before they even get to the backends.

I thought this would work (for n=2):

frontend fe-test
bind 0.0.0.0:8070
http-request deny deny_status 400 if { req.cook_cnt() gt 2 }
http-request auth realm tomcat
default_backend be-test


However, it does not work. The count is always 0, hence the ACL always pass >> 
[...]
When I change the ACL to include a cookie name, it works:

http-request deny deny_status 400 if { req.cook_cnt("C1") gt 2 }
[...]
I tried to figure out what the code does, to see if I am doing something wrong 
and found this in proto_http.c:

--
/* Iterate over all cookies present in a request to count how many occurrences
* match the name in args and args->data.str.len. If  is non-null, then
* multiple cookies may be parsed on the same line. The returned sample is of
* type UINT. Accepts exactly 1 argument of type string.
*/
static int
smp_fetch_cookie_cnt(const struct arg *args, struct sample *smp, const char 
*kw, void *private)
{
struct http_txn *txn;
struct hdr_idx *idx;
struct hdr_ctx ctx;
const struct http_msg *msg;
const char *hdr_name;
int hdr_name_len;
int cnt;
char *val_beg, *val_end;
char *sol;

if (!args || args->type != ARGT_STR)
return 0;
--

So without being very C-savvy, this appears to exit early when there is no 
parameter of type string passed in.

I hope someone can shed some light on this. :)


You're right. currently, the code and the documentation don't say the 
same things.


Can you try the attached patch ?

--
Cyril Bonté
diff --git a/src/proto_http.c b/src/proto_http.c
index 2b9b0d01e..0935f0145 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -11318,8 +11318,8 @@ extract_cookie_value(char *hdr, const char *hdr_end,
 		 * its value between val_beg and val_end.
 		 */
 
-		if (att_end - att_beg == cookie_name_l &&
-		memcmp(att_beg, cookie_name, cookie_name_l) == 0) {
+		if (cookie_name_l == 0 || (att_end - att_beg == cookie_name_l &&
+		memcmp(att_beg, cookie_name, cookie_name_l) == 0)) {
 			/* let's return this value and indicate where to go on from */
 			*value = val_beg;
 			*value_l = val_end - val_beg;
@@ -11617,7 +11617,7 @@ smp_fetch_cookie_cnt(const struct arg *args, struct sample *smp, const char *kw,
 	char *val_beg, *val_end;
 	char *sol;
 
-	if (!args || args->type != ARGT_STR)
+	if (!args || (args->type != ARGT_STR && args->type != ARGT_STOP))
 		return 0;
 
 	CHECK_HTTP_MESSAGE_FIRST();

Re: requests are loadbalanced to servers in maintainance mode

[Cyril Bonté]

Hi,

Le 22/08/2017 à 15:39, Stefan Sticht a écrit :

Hi,


if I place backend servers into maintainance mode (using hatop CLI ) 
they will still get connections. I believe this only happens with nbproc>1.


Can anyone tell me what I am doing wrong? HAProxy Version: 
1.7.9-1ppa1~xenial


You gave the answer, the reason is that you're using nbproc 40.




Aug 22 15:31:30 w-fw1 haproxy[30821]: Server webserver/web1-ip1 is going 
DOWN for maintenance. 11 active and 0 backup servers left. 0 sessions 
active, 0 requeued, 0 remaining in queue.
Aug 22 15:31:34 w-fw1 haproxy[30821]: Server webserver/web1-ip2 is going 
DOWN for maintenance. 10 active and 0 backup servers left. 0 sessions 
active, 0 requeued, 0 remaining in queue.


Here, process 30821 received the CLI request and set the backend servers 
into maintenance for its own instance. Processes don't share any 
information between them.


Aug 22 15:31:37 w-fw1 haproxy[30820]: 212.29.0.220:45714 
[22/Aug/2017:15:31:37.890] w~ webserver/web1-ip1 0/0/0/6/7 200 23812 - - 
 0/0/0/0/0 0/0 "GET /n/main_app.php HTTP/1.1"
Aug 22 15:31:37 w-fw1 haproxy[30820]: 212.29.0.220:45714 
[22/Aug/2017:15:31:37.890] w~ webserver/web1-ip1 0/0/0/6/7 200 23812 - - 
 0/0/0/0/0 0/0 "GET /n/main_app.php HTTP/1.1"
Aug 22 15:31:40 w-fw1 haproxy[30820]: 212.29.0.220:45716 
[22/Aug/2017:15:31:40.268] w~ webserver/web1-ip2 0/0/0/6/7 200 23812 - - 
 0/0/0/0/0 0/0 "GET /n/main_app.php HTTP/1.1"
Aug 22 15:31:40 w-fw1 haproxy[30820]: 212.29.0.220:45716 
[22/Aug/2017:15:31:40.268] w~ webserver/web1-ip2 0/0/0/6/7 200 23812 - - 
 0/0/0/0/0 0/0 "GET /n/main_app.php HTTP/1.1"
Aug 22 15:31:41 w-fw1 haproxy[30819]: 212.29.0.220:45718 
[22/Aug/2017:15:31:41.984] w~ webserver/web1-ip1 0/0/0/6/6 200 23812 - - 
 0/0/0/0/0 0/0 "GET /n/main_app.php HTTP/1.1"
Aug 22 15:31:41 w-fw1 haproxy[30819]: 212.29.0.220:45718 
[22/Aug/2017:15:31:41.984] w~ webserver/web1-ip1 0/0/0/6/6 200 23812 - - 
 0/0/0/0/0 0/0 "GET /n/main_app.php HTTP/1.1"
Aug 22 15:31:43 w-fw1 haproxy[30820]: 212.29.0.220:45720 
[22/Aug/2017:15:31:43.511] w~ webserver/web2-ip1 0/0/0/6/7 200 23812 - - 
 0/0/0/0/0 0/0 "GET /n/main_app.php HTTP/1.1"
Aug 22 15:31:43 w-fw1 haproxy[30820]: 212.29.0.220:45720 
[22/Aug/2017:15:31:43.511] w~ webserver/web2-ip1 0/0/0/6/7 200 23812 - - 
 0/0/0/0/0 0/0 "GET /n/main_app.php HTTP/1.1"
Aug 22 15:31:45 w-fw1 haproxy[30820]: 212.29.0.220:45722 
[22/Aug/2017:15:31:45.708] w~ webserver/web2-ip2 0/0/0/6/7 200 23812 - - 
 0/0/0/0/0 0/0 "GET /n/main_app.php HTTP/1.1"


But here, processes 30820 and 30819 received the requests.


global
 nbproc 40


If you want to use nproc 40 and use the CLI, you'll have to define 40 
CLI sockets, bound to each process and send the command to each one.




backend webserver
 mode http
 balance roundrobin
 http-reuse always
 option redispatch
 server web1-ip1 192.168.2.11:80 check source 192.168.2.113 
non-stick inter 30s maxconn 102400
 server web1-ip2 192.168.2.21:80 check source 192.168.2.114 
non-stick inter 30s maxconn 102400
 server web2-ip1 192.168.2.12:80 check source 192.168.2.115 
non-stick inter 30s maxconn 102400
 server web2-ip2 192.168.2.22:80 check source 192.168.2.116 
non-stick inter 30s maxconn 102400

[...]

Thanks!
Stefan



--
Cyril Bonté

Re: 3 HA proxy instances on 3 ECS clusters hang on and off

[Cyril Bonté]

Hi,

Le 03/08/2017 à 19:17, Ransika Desilva a écrit :

Hello,

Thanks for a wonderful product. We have an issue as off now, hoping that 
you will be able to help us.


We are having 3 clusters (dev/staging/prod) based on AWS ECS and we 
deploy the HA Proxy as docker containers on them. Each cluster has 1 
instance of the HA Proxy running.


We have noticed that even during low volume, all the 3 clusters getting 
hang. The instances are running but traffic is not forwarded. A simple 
restart works. We have added aws resolvers to handle the LB IP address 
changes.


The issue is some what similar to 
https://discourse.haproxy.org/t/haproxy-crashes-on-3-nodes-at-exactly-the-same-time/1039, 
but we are not using FreeBSD.


I have attached the HA Proxy config for your kind reference.

Thanks and looking forward to hear from you soon.


The RTF file is a pain, please send the configuration as plain text in 
your mail next time ;-)


The issue is on your server lines :
  server foo-a fs-sim-alb-external-XX.com:39997 resolvers awsvpc
You have configured a resolver but you didn't enable health checks. This 
is (currently) mandatory to name updates :

http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#resolvers%20(Server%20and%20default-server%20options)


--
Cyril Bonté

Re: fields vs word converter, unexpected "0" result

[Cyril Bonté]

Hi,

Le 01/08/2017 à 17:37, Daniel Schneller a écrit :

Any idea on the difference between “word” and “field”, though?


"field" and "word" are similar, except that "word" will ignore 
consecutive delimiters without any word.


Example with "x//y/z" :
  word(1,/)  => returns "x"
  word(2,/)  => returns "y"
  word(3,/)  => returns "z"
whereas :
  field(1,/) => returns "x"
  field(2,/) => returns ""
  field(3,/) => returns "y"


--
Cyril Bonté

Re: help for configuration between http and tcp mode

[Cyril Bonté]

Hi,

Le 09/07/2017 à 17:58, M a écrit :

Hi,

It seems the error is related to acl and I don’t yet understand why > [...]
frontend https_influxdb
   bind 192.168.246.17:8086 ssl crt /data/ssl_certs no-sslv3 ciphers 
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:DES-CBC3-SHA:!NULL:!aNULL:!RC4:!RC2:!MEDIUM:!LOW:!EXPORT:!DES:!MD5:!PSK:!3DES
   mode http
[...]
   acl host_influxdb-drp.example.net hdr(host) -i influxdb-drp.example.net
   use_backend influxdb-drp.example.net if host_influxdb-drp.example.net
[...]
#curl -G https://influxdb-drp.example.net:8086/query -u admin:'xxx' --data-urlencode 
"q=SHOW DATABASES"
503 Service Unavailable
No server is available to handle this request.


Jul  9 15:46:16 kalinga haproxy[50375]: 192.168.246.17:57242 
[09/Jul/2017:15:46:16.665] https_influxdb~ https_influxdb/ -1/-1/135 212 
SC 4/0/0/0/0 0/0

The acl is not matching under this frontend :-(
[...] 
Why acl is matching only on frontend https and not on frontend https_influxdb?


Because your Host header is certainly "influxdb-drp.example.net:8086", 
not "influxdb-drp.example.net". You can verify this with this acl instead :
  acl host_influxdb-drp.example.net hdr(host) -i 
influxdb-drp.example.net:8086


Or you can even capture the header in your logs, it's quite useful to 
debug acls ;-)



--
Cyril Bonté

Re: HAProxy 1.7.5 cookie JSESSIONID prefix not working

[Cyril Bonté]

Hi Norman,

Le 30/05/2017 à 23:39, Norman Branitsky a écrit :

I modified the server line thus:

server id-dv-dcavr-01 10.90.10.53:9001 check cookie id-dv-dcavr-01



Now the server name appears as a prefix with a "~" separator.

(It used to appear as a suffix with a "." separator.)


No, appsession never did that. It doesn't modify the cookie value. If a 
suffix was added, it was done by the application server, I guess the 
jvmRoute parameter in your case. I suspect you have also modified the 
configuration of you app servers and didn't set this parameter during 
the switch from haproxy 1.5 to 1.7.




JSESSIONID=id-dv-dcavr-01~E8C5E4A2; path=/le5;
domain=cadca-vr.irondatacorp.com; Secure; HttpOnly



I can now successfully login to the 2 different servers.



You ask

“Also, why not use a dedicated cookie for haproxy, instead of humping
JSESSIONID?”

Frankly, this never occurred to me.

When I started with HAProxy 1.5, 4 years ago,

I looked for example configurations for fronting JBoss and Tomcat.

The documentation always referred to:

appsession JSESSIONID len 52 timeout 3h



Are you suggesting I do something like this instead?

cookie HAPROXYID insert nocache



-Original Message-
From: Lukas Tribus [mailto:lu...@gmx.net]
Sent: May-30-17 5:00 PM
To: Norman Branitsky <norman.branit...@micropact.com>; haproxy@formilux.org
Subject: Re: HAProxy 1.7.5 cookie JSESSIONID prefix not working



Hello Norman,





Am 30.05.2017 um 18:06 schrieb Norman Branitsky:






The server’s identifier is not added to the cookie.








Did you specify the cookie value on the server line [1], as per [2]:




The value of the cookie will be the value indicated after the



"cookie<https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#



>" keyword in a "server





<https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#server>"
statement. If no cookie is declared for a given server, the cookie is
not set.





Also, why not use a dedicated cookie for haproxy, instead of humping
JSESSIONID?

Do you have clients so broken they support only one single cookie?







Regards,

Lukas





[1] https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4.2-cookie

[2] https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.2-cookie




--
Cyril Bonté

Re: [PATCH] BUILD: ssl: fix SSL_OP_NO_SSLv3 with LibreSSL >= 2.3.0

[Cyril Bonté]

Hi Emmanuel,

Sorry for the delay and for the lack of details in my previous mail. 
During this week-end I was in a world without computer :)


Le 22/05/2017 à 14:31, Emmanuel Hocdet a écrit :

Hi Cyril,

This patch should fix the build issue



Can you check it’s your case?


It didn't but now I understand why. The issue was on my test box under 
Debian testing.
There was a mismatch between the libssl packages. I had both 
libssl1.0.0, libssl1.0.2 and libssl1.1 but the dev package was 
libssl1.0-dev.


Once libssl-dev has been installed (1.1.0e-2), everything worked as 
expected. Sorry for the noise ;-)


--
Cyril Bonté

Re: [Patches] TLS methods configuration reworked

[Cyril Bonté]

Hi all,

Le 12/05/2017 à 15:13, Willy Tarreau a écrit :

Hi guys,

On Tue, May 09, 2017 at 11:21:36AM +0200, Emeric Brun wrote:

It seems to do what we want, so we can merge it.


So the good news is that this patch set now got merged :-)


Commit 5db33cbdc4 [1] seems to have broken the compilation when 
OPENSSL_NO_SSL3 is defined : SSLv3_server_method() and 
SSLv3_client_method() won't exist in this case.
Previously there was a condition to verify this, which has disappeared 
with this patch set.




Thanks for your time and efforts back-and-forth on this one!
Willy



[1] 
http://www.haproxy.org/git?p=haproxy.git;a=commit;h=5db33cbdc4f2952cbd3c140edce0eda84e1447b4


--
Cyril Bonté

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

[Cyril Bonté]

Hi Willy,

> De: "Willy Tarreau" 
> À: "jaseywang" 

[...]

> > Below is the data during benchmark:
> > *maxsock = *136; *maxconn = *50; *maxpipes = *0
> > current conns = 7488; current pipes = 0/0; conn rate = 322/sec
> > Running tasks: 7366/7449; idle = 0 %
> 
> This one tends to rule out the task_is_expired() bug because the idle
> time is null, so you're running at 100% CPU. But 100% CPU for 322
> conn/s
> seems almost impossible to me. Even my ALOHA-pocket powered over USB
> and running on a 400 MHz fanless MIPS does 3 or 5 times more.

Just a quick note : I remember having read that the configuration is running 
with dh-param 4096. It can require a lot more CPU than 2048 dh-params.

[...]