Re: Weird stick-tables / peers behaviour

[Sylvain Faivre]

On 03/24/2016 04:07 PM, Christian Ruppert wrote:

Hi all,

I've just upgraded some hosts to 1.6.4 (from 1.5) and immediately got a


[...]


and two for doing some "curl -Lvs http://127.0.0.1:8080; by hand.
If you do some on the first and some on the second host you'll notice
different values on one side. Also the counter may e.g. double while the
other side has the correct/actual value. This results into several
thousands of requests on our prod. systems but according to the logs it
can't be correct.
Does anybody else have similar weirdness or can you guys confirm false
values?
The *_cnt values seem to be ok but the *_rate ones seem to be false in
some cases.



Hi,

This is a know bug in the 1.6 series.
Willy spoke about it in a few posts on this list.
It is not fixed yet.

Regards.
Sylvain

Re: Protecting against slow HTTP POST queries

[Sylvain Faivre]

On 02/04/2016 12:06 PM, Baptiste wrote:

On Thu, Feb 4, 2016 at 9:44 AM, Sylvain Faivre
<sylvain.fai...@reservit.com> wrote:

Hi,

Is there a timeout setting in HAproxy that can help protect against slow
HTTP POST queries ?


>> [...]



please run the same test against HAProxy 1.6 and enable "option
buffer-http-request":
http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#4-option%20http-buffer-request

Then your timeout http-request will also match the POSTed data.



Hi Baptiste,

This did the trick, thanks.

Sylvain

Protecting against slow HTTP POST queries

[Sylvain Faivre]

Hi,

Is there a timeout setting in HAproxy that can help protect against slow 
HTTP POST queries ?


I'm not talking about "slow loris" type attacks (where the client sleeps 
between request headers) but "slow HTTP POST" (where the client sleeps 
between POST data lines).


Here is an example :

- Test 1 :

root@proxy1>: telnet localhost 85
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
POST /test HTTP/1.1
Host: host.domain.com
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Content-Type: application/json; charset=utf-8
Content-Length: 1234

test

<>

HTTP/1.1 408 Request Time-out
Date: Wed, 03 Feb 2016 13:03:30 GMT
Content-Length: 223
Content-Type: text/html; charset=iso-8859-1



408 Request Time-out

Request Time-out
Server timeout waiting for the HTTP request from the client.

Connection closed by foreign host.


- Test 2 :

Here we send the POST body very slowly (line by line, wait 10 seconds 
between each line).


root@proxy1>: telnet localhost 85
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
POST /test HTTP/1.1
Host: host.domain.com
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Content-Type: application/json; charset=utf-8
Content-Length: 1234

test
<>
test
<>
test
<>
test
<>
test
<>
test
<>
test
<>
Connection closed by foreign host.


In each case, HAproxy log shows termination flags "SD--" which means the 
application server closed the connection.


So, the app server timeouts after 30 seconds, but this duration is reset 
each time the client sends data in the POST body.


Is there an option to set a timeout on this part of the request ? It 
should be similar to "timeout http-request" but work against the request 
body, instead of the request headers.


We already have these settings, but none of them seems to act against 
HTTP POST content (I was able to stay connected while sending HTTP POST 
content for 5+ minutes) :


timeout connect 5s
timeout http-request12s
timeout queue   180s
timeout client  180s
timeout server  180s
timeout http-keep-alive 10s
timeout tarpit  30s


Best regards,

Sylvain

stick table replication problem

[Sylvain Faivre]

Hi,

We just had a strange replication problem on our staging environment.
We have 2 HAproxy servers. They were running for 2 weeks now.
At the beginning, I checked that the stick tables were properly synced.

Today, stick tables were not synced, for example :

root@proxy1>: echo "show table front_jsessionid" | socat stdio 
/usr/share/haproxy/mysocket

# table: front_jsessionid, type: string, size:10485760, used:3
0xd1d944: key=MkJm-rbcE3V5EJg1RmTRAw__ use=0 exp=3474723 server_id=1
0xd762c4: key=Z2rxYufqv7B0C2gIgztSfA__ use=0 exp=3571578 server_id=2
0xea9484: key=aUuiOrlUDb7RnvcFMLK9oA__ use=0 exp=2879968 server_id=2

root@proxy2>: echo "show table front_jsessionid" | socat stdio 
/usr/share/haproxy/mysocket

# table: front_jsessionid, type: string, size:10485760, used:5
0x1bc9104: key=D8EWzwrGr2UK3btCwfpweQ__ use=0 exp=3552238 server_id=1
0x1c3ba54: key=Z2rxYufqv7B0C2gIgztSfA__ use=0 exp=3315450 server_id=1
0x1cbecb4: key=aUuiOrlUDb7RnvcFMLK9oA__ use=0 exp=2624851 server_id=2
0x1c49664: key=lmqFlaNhdHXYwH4M4QQjqQ__ use=0 exp=3510898 server_id=2
0x1cbf074: key=zpE6NHQJr~aStxzmJTVEgA__ use=0 exp=2587567 server_id=2

--

I tried :
echo "set table front_jsessionid key toto data.server_id 1" | socat 
stdio /usr/share/haproxy/mysocket


The new entry was not replicated on proxy2, and tcpdump showed no 
outgoing traffic from proxy1 to proxy2.


Then, I reloaded the haproxy service on proxy2, and tables were synced, 
including old data set in proxy1 before the reload.

New data replicates all right too.

Should this situation happen again, is there anything I can do to debug 
this further ?


--

Here is netstat output from proxy1 while replication was broken :

root@proxy1>: netstat -anp |grep 9421
tcp0  0 [proxy1]:94210.0.0.0:*LISTEN 
  1163/haproxy
tcp   35  0 [proxy1]:9421[proxy2]:41205   CLOSE_WAIT 
 -
tcp1  0 [proxy1]:20512   [proxy2]:9421CLOSE_WAIT 
 1163/haproxy
tcp1  0 [proxy1]:35662   [proxy2]:9421CLOSE_WAIT 
 1163/haproxy
tcp   35  0 [proxy1]:9421[proxy2]:41257   CLOSE_WAIT 
 -
tcp0  0 [proxy1]:54443   [proxy2]:9421FIN_WAIT2 
  1163/haproxy
tcp1  0 [proxy1]:52254   [proxy2]:9421CLOSE_WAIT 
 1163/haproxy
tcp1  0 [proxy1]:54249   [proxy2]:9421CLOSE_WAIT 
 1163/haproxy
tcp   35  0 [proxy1]:9421[proxy2]:41160   CLOSE_WAIT 
 -
tcp   35  0 [proxy1]:9421[proxy2]:41110   CLOSE_WAIT 
 -
tcp1  0 [proxy1]:40469   [proxy2]:9421CLOSE_WAIT 
 1163/haproxy


--

Here are relevant parts from our setup :

peers prod
peer  proxy1[proxy1]:9421
peer  proxy2[proxy2]:9421

backend front
stick on urlp(jsessionid),url_dec table front_jsessionid
stick on urlp(jsessionid,;),url_dec table front_jsessionid
stick on cookie(JSESSIONID) table front_jsessionid
stick store-response cookie(JSESSIONID) table front_jsessionid

backend front_jsessionid
stick-table type string len 24 size 10m expire 1h peers prod

--

We are running HAproxy 1.6.2 from the vbernat PPA on Ubuntu 12.04, with 
nbproc = 1.


Thanks in advance.

Sylvain

Re: Email checks in defaults section

[Sylvain Faivre]

On 11/01/2015 06:34 PM, Tommy Atkinson wrote:

I want to enable email alerts for all my backends so I added the
"email-alert" options to the defaults section and a mailers section at
the top level. The documentation indicates this is supported but it
doesn't seem to work. HAProxy connects to the mail server but doesn't
actually send anything. Copy/pasting the options to a backend works.

1.6.1 on Linux




Hi,

This seems to be a bug, I had the same problem.
Copying & pasting the "email-alert" settings in all the backend sections 
did the trick.


When email-alert options are set only in the defaults section, just like 
Tommy said, HAproxy connects to the mail server but doesn't send anything.

Here are the postfix logs from the problem :
Nov 30 09:31:01 host1 postfix/smtpd[2502]: connect from localhost[127.0.0.1]
Nov 30 09:31:03 host1 postfix/smtpd[2502]: lost connection after CONNECT 
from localhost[127.0.0.1]
Nov 30 09:31:03 host1 postfix/smtpd[2502]: disconnect from 
localhost[127.0.0.1]


With fixed config :
Nov 30 10:03:58 host1 postfix/smtpd[8335]: connect from localhost[127.0.0.1]
Nov 30 10:03:58 host1 postfix/smtpd[8335]: 79A28402E7: 
client=localhost[127.0.0.1]
Nov 30 10:03:58 host1 postfix/cleanup[8338]: 79A28402E7: 
message-id=<20151130090358.79A28402E7@host1>
Nov 30 10:03:58 host1 postfix/smtpd[8335]: disconnect from 
localhost[127.0.0.1]
Nov 30 10:03:58 host1 postfix/qmgr[1346]: 79A28402E7: 
from=, size=600, nrcpt=1 (queue active)
Nov 30 10:03:58 host1 postfix/smtp[8339]: 79A28402E7: 
to=, relay=x:25, delay=0.14, 
delays=0.06/0.03/0.04/0, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 
962E420D9C)

Nov 30 10:03:58 host1 postfix/qmgr[1346]: 79A28402E7: removed


(using 1.6.2 Ubuntu package)

Best regards,
Sylvain

Re: appsession replacement in 1.6

[Sylvain Faivre]

Hi Willy,

On 11/18/2015 09:31 PM, Willy Tarreau wrote:

Hi Sylvain,

On Tue, Nov 17, 2015 at 03:09:49PM +0100, Sylvain Faivre wrote:

option http-buffer-request

maybe you should stick on the header ;-)


OK I added "option http-buffer-request", it will help for sure !

What do you mean "stick on the header" ?


Oh well, it seems like I forgot to stick on the request URL parameter,
when passed with a question mark.


What Aleks meant is that you don't need http-buffer-request as it's
only used to process POST data which isn't your case.


Oh OK. Well, I did not talk about http-buffer-request in the beginning, 
so I thought Aleks recommended to use it.

I can see that it is not useful in my case, so I removed it again.




So I just added this to my config :
  stick on urlp(JSESSIONID) table back_jsessionid
  stick store-request urlp(JSESSIONID) table back_jsessionid

It should cover all cases now.


These two are partially redundant, since "stick on" is already a
shortcut for "stick match" + "stick store-request". So you end up
having this :

   stick match urlp(JSESSIONID) table back_jsessionid
   stick store-request urlp(JSESSIONID) table back_jsessionid
   stick store-request urlp(JSESSIONID) table back_jsessionid

Thus you can safely remove the "stick store-request" rule, you won't
notice the difference.



OK, I did that, and also changed urlp(jsessionid) to lower case, since 
our application server passes it in lower case when in URL parameters.


For future reference, here is my latest config. It has been running for 
a few hours on our staging servers without a problem :


backend front
[stick-table for another purpose]
[other stuff]
# sticky session on JSESSIONID
stick on urlp(jsessionid) table front_jsessionid
stick on urlp(jsessionid,;) table front_jsessionid
stick on cookie(JSESSIONID) table front_jsessionid
stick store-response cookie(JSESSIONID) table front_jsessionid

backend front_jsessionid
stick-table type string len 24 size 10m expire 1h peers prod


Best regards,
Sylvain

Re: appsession replacement in 1.6

[Sylvain Faivre]

Replying to myself :

On 11/17/2015 10:58 AM, Sylvain Faivre wrote:

Hi Aleks, and thanks again for your help.

Concerning this point :
On 11/16/2015 11:16 PM, Aleksandar Lazic wrote:


As described here

http://git.haproxy.org/?p=haproxy-1.6.git;a=blob;f=doc/configuration.txt;h=45d1aacfbe0d2d53193f7956a0dd03e5f8151ea6;hb=HEAD#l5043



option http-buffer-request

maybe you should stick on the header ;-)


OK I added "option http-buffer-request", it will help for sure !

What do you mean "stick on the header" ?


Oh well, it seems like I forgot to stick on the request URL parameter, 
when passed with a question mark.


So I just added this to my config :
 stick on urlp(JSESSIONID) table back_jsessionid
 stick store-request urlp(JSESSIONID) table back_jsessionid

It should cover all cases now.


I currently have the following config, I think it will stick on
JSESSIONID, either in the cookie (so request header, also capturing the
cookie in reply header) or as a request URL parameter. Am I missing
something ?

backend front
 [stick-table for another purpose]
 [other stuff]
 # sticky session on JSESSIONID
 option http-buffer-request
 stick on urlp(JSESSIONID,;) table front_jsessionid
 stick on cookie(JSESSIONID) table front_jsessionid
 stick store-response cookie(JSESSIONID) table front_jsessionid
 stick store-request cookie(JSESSIONID) table front_jsessionid
 stick store-request urlp(JSESSIONID,;) table front_jsessionid

backend front_jsessionid
 stick-table type string len 24 size 10m expire 1h peers prod

Re: appsession replacement in 1.6

[Sylvain Faivre]

Hi Aleks, and thanks again for your help.

Concerning this point :
On 11/16/2015 11:16 PM, Aleksandar Lazic wrote:


As described here

http://git.haproxy.org/?p=haproxy-1.6.git;a=blob;f=doc/configuration.txt;h=45d1aacfbe0d2d53193f7956a0dd03e5f8151ea6;hb=HEAD#l5043


option http-buffer-request

maybe you should stick on the header ;-)


OK I added "option http-buffer-request", it will help for sure !

What do you mean "stick on the header" ?
I currently have the following config, I think it will stick on 
JSESSIONID, either in the cookie (so request header, also capturing the 
cookie in reply header) or as a request URL parameter. Am I missing 
something ?


backend front
[stick-table for another purpose]
[other stuff]
# sticky session on JSESSIONID
option http-buffer-request
stick on urlp(JSESSIONID,;) table front_jsessionid
stick on cookie(JSESSIONID) table front_jsessionid
stick store-response cookie(JSESSIONID) table front_jsessionid
stick store-request cookie(JSESSIONID) table front_jsessionid
stick store-request urlp(JSESSIONID,;) table front_jsessionid

backend front_jsessionid
stick-table type string len 24 size 10m expire 1h peers prod

Re: appsession replacement in 1.6

[Sylvain Faivre]

Hi Aleks,

On 11/10/2015 10:56 PM, Aleksandar Lazic wrote:

Dear Sylvain Faivre,

Am 10-11-2015 12:48, schrieb Sylvain Faivre:

On 11/10/2015 12:00 AM, Aleksandar Lazic wrote:

Hi Sylvain Faivre.

Am 09-11-2015 17:31, schrieb Sylvain Faivre:


[snipp]


So, I've got this so far :

backend http

  stick-table type string len 24 size 10m expire 1h peers prod

  stick on urlp(JSESSIONID,;)
  stick on cookie(JSESSIONID)


Does this seem right ?
The help for "stick on" tells it defines a request pattern, so I guess
this would not match JSESSIONID cookie ou url parameter set in the
reply ?


I have no java server here to test this commands but with this commands
haproxy does not warn you about some config errors ;-).

###
backend dest01
   mode http

   stick-table type string len 24 size 10m expire 1h peers prod

   stick on urlp(JSESSIONID,;)
   stick on cookie(JSESSIONID)

   stick store-response cookie(JSESSIONID)
#  stick store-response res.hdr(JSESSIONID,;)

   stick store-request cookie(JSESSIONID)
   stick store-request urlp(JSESSIONID,;)

   server srv_dest01 dest01.com:80
###

I have not seen a good option to read the JSESSIONID from the response
header in case it is not in a cookie.
Have anyone I idea?!

Please can you post a full response header which was created from the
app or appserver when the app or appserver have detected that the client
does not allow cookies.


[snipp]


In fact, the server sends the JSESSIONID as a cookie even if the
client does not support cookies, *and* it adds the JSESSIONID as a URL
parameter in all links, so this should be all right.


This would be helpfully to see the full response.
Maybe some appserver behaves different.


As far as I know, there is no way for the server to detect if the client 
has cookies enabled, by looking only at the first request from that client.


According to a quick Google search, the most common ways to detect 
cookies support are either to use Javascript (so client-side check) or 
to reply with a redirect response with the cookie set, then when 
processing the redirected URL, check if the client sent the cookie along 
with the request (so this case will be covered by the proposed HAproxy 
settings).


I don't feel comfortable giving our application server version on a 
public list, but I will send it to you in private.


Here are the headers from a client request and server reply, with a 
brand new profile on the client (with cookies disabled) :


- request :
GET /front/url1.do?m=booking=FR HTTP/1.1
Host: redacted.host.domain
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:42.0) 
Gecko/20100101 Firefox/42.0

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive

- reply :
HTTP/1.1 200 OK
Date: Mon, 16 Nov 2015 15:25:41 GMT
Content-Type: text/html;charset=ISO-8859-15
Set-Cookie: JSESSIONID=uNmYNvgUME5-8LYPzimsCg__.8d15fc; Path=/front
Vary: Accept-Encoding
Content-Encoding: gzip
Transfer-Encoding: chunked

And here is a URL sample from the body of the reply. You will notice 
that the jsessionid is there twice, first one after a semicolon and 
second one after a question mark. I am not sure if this comes from the 
application server of from our custom code.


<tt>src="<a  rel="nofollow" href="https://redacted.host.domain/front/url2.do;jsessionid=uNmYNvgUME5-8LYPzimsCg__.8d15fc?jsessionid=uNmYNvgUME5-8LYPzimsCg__.8d15fc&langcode=FR&quot">https://redacted.host.domain/front/url2.do;jsessionid=uNmYNvgUME5-8LYPzimsCg__.8d15fc?jsessionid=uNmYNvgUME5-8LYPzimsCg__.8d15fc&langcode=FR&quot</a>; 
</tt><tt>language="JavaScript">






I just tried your config on a test server, and the session IDs are
rightly recorded in the table, whether the client accepts cookies or
not.


Perfect.


I still have some test cases to run, I will check this next week and
report back if needed.


Oh yes please tell us the results so that we can add this as migration
example for appsession.


OK, I will. This will not go into production yet, we still need to run 
it on a test environment for at least 3 weeks...


Best regards.
Sylvain

Re: appsession replacement in 1.6

[Sylvain Faivre]

On 11/10/2015 12:00 AM, Aleksandar Lazic wrote:

Hi Sylvain Faivre.

Am 09-11-2015 17:31, schrieb Sylvain Faivre:

Hi,

Sorry I'm late on this discussion, following this thread :
https://marc.info/?l=haproxy=143345620219498=2

We are using appsession with HAproxy 1.5 like this :


Thanks ;-)


backend http
appsession JSESSIONID len 24 timeout 1h request-learn

We would like to be able to do the same thing with HAproxy 1.6.
If it is possible, we'd like to catch the JSESSIONID in cookies and
URL parameters, either in the request or in the response.

I tried to use the info posted previously by Aleksandar, but I
encountered several problems :

- using "stick on" in frontend section fails with :

'stick' ignored because frontend 'web' has no backend capability.


- using "stick store-response urlp(JSESSIONID,;)" in backend section
fails with :

'stick': fetch method 'urlp' extracts information from 'HTTP request
headers', none of which is available for 'store-response'.



So, I've got this so far :

backend http

  stick-table type string len 24 size 10m expire 1h peers prod

  stick on urlp(JSESSIONID,;)
  stick on cookie(JSESSIONID)


Does this seem right ?
The help for "stick on" tells it defines a request pattern, so I guess
this would not match JSESSIONID cookie ou url parameter set in the
reply ?


I have no java server here to test this commands but with this commands
haproxy does not warn you about some config errors ;-).

###
backend dest01
   mode http

   stick-table type string len 24 size 10m expire 1h peers prod

   stick on urlp(JSESSIONID,;)
   stick on cookie(JSESSIONID)

   stick store-response cookie(JSESSIONID)
#  stick store-response res.hdr(JSESSIONID,;)

   stick store-request cookie(JSESSIONID)
   stick store-request urlp(JSESSIONID,;)

   server srv_dest01 dest01.com:80
###

I have not seen a good option to read the JSESSIONID from the response
header in case it is not in a cookie.
Have anyone I idea?!

Please can you post a full response header which was created from the
app or appserver when the app or appserver have detected that the client
does not allow cookies.

cheers
Aleks



Thank you Aleks.

In fact, the server sends the JSESSIONID as a cookie even if the client 
does not support cookies, *and* it adds the JSESSIONID as a URL 
parameter in all links, so this should be all right.


I just tried your config on a test server, and the session IDs are 
rightly recorded in the table, whether the client accepts cookies or not.


I still have some test cases to run, I will check this next week and 
report back if needed.


Regards,
Sylvain

appsession replacement in 1.6

[Sylvain Faivre]

Hi,

Sorry I'm late on this discussion, following this thread :
https://marc.info/?l=haproxy=143345620219498=2

We are using appsession with HAproxy 1.5 like this :

backend http
appsession JSESSIONID len 24 timeout 1h request-learn

We would like to be able to do the same thing with HAproxy 1.6.
If it is possible, we'd like to catch the JSESSIONID in cookies and URL 
parameters, either in the request or in the response.


I tried to use the info posted previously by Aleksandar, but I 
encountered several problems :


- using "stick on" in frontend section fails with :
> 'stick' ignored because frontend 'web' has no backend capability.

- using "stick store-response urlp(JSESSIONID,;)" in backend section 
fails with :
> 'stick': fetch method 'urlp' extracts information from 'HTTP request 
headers', none of which is available for 'store-response'.



So, I've got this so far :

backend http

  stick-table type string len 24 size 10m expire 1h peers prod

  stick on urlp(JSESSIONID,;)
  stick on cookie(JSESSIONID)


Does this seem right ?
The help for "stick on" tells it defines a request pattern, so I guess 
this would not match JSESSIONID cookie ou url parameter set in the reply ?



Regards,
Sylvain

Re: use several gpc's ?

[Sylvain Faivre]

On 11/05/2015 03:30 PM, Baptiste wrote:

On Thu, Nov 5, 2015 at 2:48 PM, Sylvain Faivre
<sylvain.fai...@reservit.com> wrote:

Hi,

Is there a way to use several gpc's ?

I already use gpc0 to track client IPs generating too many errors, and I
need to use another counter to track client IPs requesting some pages too
fast.

Here are the relevant parts of my current setup :

frontend web
 stick-table type ip size 500k expire 5m store gpc0
 tcp-request content track-sc1 src
 http-request deny if !i_internal { sc1_get_gpc0 gt 0 }

backend front
 stick-table type ip size 100k expire 5m store http_err_rate(10s)
 tcp-request content track-sc2 src
 acl error_rate_abuse sc2_http_err_rate gt 10
 acl mark_as_abuser sc1_inc_gpc0 gt 0
 reqtarpit . if error_rate_abuse !whitelist mark_as_abuser

And I'm trying to add something like this to the frontend :

   stick-table type ip size 50k expire 24h store gpc0_rate(60s)
   acl pages_info path_sub -i info.php
   acl too_many_info_requests sc0_gpc0_rate() gt 50
   acl mark_seen_pages_info sc0_inc_gpc0 gt 0
   tcp-request content track-sc0 src if pages_info
   http-request deny if mark_seen_pages_info too_many_info_requests

But I'm afraid that I will not be able to distinguish the info stored in
gpc0 for the error count and for the requests count...
What am I missing here ?




Hi Sylvain,

Which version of HAProxy are you using?
With 1.6, there are some converters that may be used to get rid of
using gpc while counting errors.
It means you would store abuser client IP in a dedicated table and
simply check if the IP is there:
  http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#in_table

I have on my TODO to write such type of article on the blog. Some kind
of DDOS protection with HAProxy 1.6.

Baptiste



We are using HAproxy 1.5, upgrading to 1.6 shouldn't be a huge problem.

I guess I'll wait for your article, since I'm not sure I understand 
everything about all this table stuff.


So, with HAproxy 1.5, one cannot have two types of DDOS protection at 
the same time ? (against flag offenders who send too many requests, and 
those whose requests cause too many errors)

log SSL/TLS protocol version

[Sylvain Faivre]

Hello,

We use Haproxy in front of HTTP servers, SSL termination is done on HAproxy.

Is there a way to have HAproxy log the SSL or TLS protocol version (TLS 
1.0 / 1.1 / 1.2) or specific cipher that was used for requests ?


I know this is negociated between each client and the HAproxy server, 
but I would like to know which clients use outdated protocols.


Thanks.

add header or query parameter when redirecting

[Sylvain Faivre]

Hello,

I use the redirect directive to redirect users from old sites to a new 
site, eg:

redirect prefix http://new-site.com code 301 if old-site

I would like to redirect requests from many old sites to the same new 
site, so I need a way to add info about the old host in the new 
redirected request.


I'm looking for a way to add a header to the redirected request to 
identify the host, for example :

X-Orig-Site: old-site-123.com

Is this possible ?

I guess I can't add a header to the request with HAproxy, since HAproxy 
only sends a new Location header to the browser, and the browser sets 
the headers.


So, is there a way to alter the location sent in the redirect, to 
include « orig-site=old-site-123.com » ?


I think I'm missing something here.
Should I user « http-request redirect » instead of « redirect prefix » ?


By the way, I tried to use the set-cookie option for this, but it was a 
bad idea :
redirect prefix http://new-site.com code 301 set-cookie 
ORIG=%[hdr(host)] if old_site


This doesn't work for two reasons :

1. The « %[hdr(host)] » part is send literally in the request :
Set-Cookie: ORIG=%[hdr(host)]; path=/;

2. The request sent to new-site.com doesn't seem to include this cookie


Sylvain

Re: add header or query parameter when redirecting

[Sylvain Faivre]

On 06/03/2015 04:02 PM, Baptiste wrote:

On Wed, Jun 3, 2015 at 11:58 AM, Sylvain Faivre
sylvain.fai...@reservit.com wrote:

Hello,

I use the redirect directive to redirect users from old sites to a new site,
eg:
 redirect prefix http://new-site.com code 301 if old-site

I would like to redirect requests from many old sites to the same new site,
so I need a way to add info about the old host in the new redirected
request.

I'm looking for a way to add a header to the redirected request to identify
the host, for example :
X-Orig-Site: old-site-123.com

Is this possible ?

I guess I can't add a header to the request with HAproxy, since HAproxy only
sends a new Location header to the browser, and the browser sets the
headers.

So, is there a way to alter the location sent in the redirect, to include «
orig-site=old-site-123.com » ?

I think I'm missing something here.
Should I user « http-request redirect » instead of « redirect prefix » ?


By the way, I tried to use the set-cookie option for this, but it was a bad
idea :
 redirect prefix http://new-site.com code 301 set-cookie
ORIG=%[hdr(host)] if old_site

This doesn't work for two reasons :

1. The « %[hdr(host)] » part is send literally in the request :
Set-Cookie: ORIG=%[hdr(host)]; path=/;

2. The request sent to new-site.com doesn't seem to include this cookie


Sylvain



Hi Sylvain,

The only good way to do what you want to achieve, is to use a query
string parameter and http-request and http-response rules coupled to a
few sections...
Basically, haproxy is not able to modify the headers sent by a
redirect rule. So the trick here, is to perform the redirect in a
dummy frontend section used as a server in a dedicated backend and
insert a header in the response, like this:

backend be_redirect
  http-request capture req.hdr(host),word(1,:),lower len 32
  http-response replace-value Location (.*)
\1orig-site=%[capture.req.hdr(0)] if { res.hdr(Location) -m sub ? }
  http-response replace-value Location (.*)
\1?orig-site=%[capture.req.hdr(0)] if !{ res.hdr(Location) -m sub ? }
   server dummy_redirect 127.0.0.1:8001

frontend fe_dummy_redirect
  bind 127.0.0.1:8001
  http-request redirect prefix http://new-site.com code 301


Note that this configuration needs HAProxy 1.6 (latest snapshot).

Baptiste




Hi Baptiste,

Unfortunately, we are not willing to upgrade to HAproxy 1.6 just yet, so 
we are going to use another solution for this redirect (change DNS 
records to resolve old hostnames to the new web server).


Thank you for the info anyway, it may be useful for another time.

Sylvain

understanding HAproxy stats

[Sylvain Faivre]

Hello,

I have trouble understanding stats reports from our HAproxy servers, can 
anyone please shed some light on this ?



1. On a backend with only one server, scur(BACKEND)  scur(server). How 
can this be ?
# 
pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,

backend_b,server_a,0,0,1,1,1,92433,167970344,127389583,,0,,0,0,0,0,UP,1,1,0,512,1,20,1,,6373,backend_a/server_a,2,3,,290,92432,0,0,0,0,00,0,0,,,12858,2,530,13826,
backend_b,BACKEND,48,142,49,143,410,92481,167970344,127389583,0,0,,0,0,0,0,UP,1,1,0,,0,88182,0,,1,20,0,,6373,,1,0,,650,92432,0,0,0,0,0,0,0,0,0,0,183,,,12858,2,530,13826,


2. Same HAproxy instance, on another backend, smax(BACKEND)  
slim(server_a) + slim(server_b). Again, how is this possible ? It seems 
like smax(BACKEND) = smax(server_a) + smax(server_b) + qmax(BACKEND) ?
# 
pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,

backend_c,server_a,0,0,0,3,3,95541,232232183,241897167,,0,,0,0,0,0,UP,1,1,0,512,1,19,1,,95507,backend_a/server_a,2,0,,50,95541,0,0,0,0,01,0,2,,,0,1,706,2028,
backend_c,server_b,0,0,0,3,3,95587,232144919,241789273,,0,,0,0,0,0,UP,1,1,0,512,1,19,2,,95546,backend_a/server_b,2,0,,50,95587,0,0,0,0,00,0,3,,,0,2,675,2082,
backend_c,BACKEND,0,3,0,9,410,191128,464377102,483686440,0,0,,0,0,0,0,UP,2,2,0,,0,89074,0,,1,19,0,,191053,,1,0,,100,191128,0,0,0,0,1,0,0,0,0,0,2,,,0,1,678,2059,


3. qmax stays at zero on each backend server, even though qmax(BACKEND) 
 0. Are queue stats not maintained for each server ?
# 
pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,

backend_c,server_a,0,0,0,3,3,95541,232232183,241897167,,0,,0,0,0,0,UP,1,1,0,512,1,19,1,,95507,backend_a/server_a,2,0,,50,95541,0,0,0,0,01,0,2,,,0,1,706,2028,
backend_c,server_b,0,0,0,3,3,95587,232144919,241789273,,0,,0,0,0,0,UP,1,1,0,512,1,19,2,,95546,backend_a/server_b,2,0,,50,95587,0,0,0,0,00,0,3,,,0,2,675,2082,
backend_c,BACKEND,0,3,0,9,410,191128,464377102,483686440,0,0,,0,0,0,0,UP,2,2,0,,0,89074,0,,1,19,0,,191053,,1,0,,100,191128,0,0,0,0,1,0,0,0,0,0,2,,,0,1,678,2059,


4. On another HAproxy instance, on a backend with two servers, we always 
have scur=slim on both servers, but qcur stays at zero.
The application servers tell us their threads are ready (zero busy 
thread). Is there any way to show more info about the sessions which 
HAproxy thinks are in use ?
# 
pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,

backend_a,server_a,0,0,12,12,12,90,114944,68086,,0,,0,0,0,0,UP,1,1,0,18,6,67872,1015,128,1,9,1,,88,,2,0,,8,L7OK,200,5,0,78,0,0,0,0,00,0,501099,OK,,3,1,42,55,
backend_a,server_b,0,0,12,12,12,72,87204,47593,,0,,0,0,0,0,UP,1,1,0,25,6,67842,1064,128,1,9,2,,72,,2,0,,7,L7OK,200,4,0,60,0,0,0,0,00,0,501461,OK,,0,1,28,36,
backend_a,BACKEND,0,3,24,27,410,181,223139,119707,0,0,,5,0,0,0,UP,2,2,0,,6,67872,862,,1,9,0,,160,,1,0,,150,138,0,0,19,0,5,0,0,0,0,0,501099,,,3,1,66,86,


BTW, we are running HAproxy 1.5.9 from the vbernat Ubuntu PPA.

Thanks in advance.
Sylvain

Send client to a specific backend if header found in previous reply from server

[Sylvain Faivre]

Hello,

This is a followup to my post and Baptiste's answer on 20141120.
Unfortunately, I couldn't get this to work.

I am trying to achieve the following : when a response from the 
application server contains a header named X-test, send the following 
requests from the client IP to another backend. The goal is to send 
clients who abuse the servers to a slower queue.


Here is my current config :

In the frontend:
stick-table type ip size 50k expire 5m store gpc0
tcp-request content track-sc2 src
#use_backend slow if { sc2_get_gpc0 gt 0 }

In the backend:
acl mark_as_CPUusage sc2_inc_gpc0 gt 0
acl unmark_as_CPUusage sc2_clr_gpc0 gt 0
tcp-response content accept if { res.hdr(X-test) -m found } 
mark_as_CPUusage
#tcp-response content accept if ! { res.hdr(X-test) -m found } 
unmark_as_CPUusage



I used capture response header to confirm that the application server 
sends the X-test header for some clients.
By querying the stats through unix socket commands, I get gpc0=0 for 
every client IP, so this doesn't seem to work.


Any idea about what I'm missing ?

Sylvain.

Send client to a specific backend if header found in previous reply from server

[Sylvain Faivre]

Hi,

I am trying to achieve the following : when a response from the 
application server contains a header named X-test, send the following 
requests from the client IP to another backend for 5 minutes. The goal 
is to send clients who abuse the servers to a slower queue.


Here is what I got so far :

In the frontend:
stick-table type ip size 100k expire 5m store gpc1
tcp-request content track-sc1 src
use_backend slow if { sc1_get_gpc1 gt 0 }

In the backend:
acl mark_as_high_usage sc1_inc_gpc1 gt 0
??? if res.hdr_cnt(X-test) mark_as_high_usage

Does this look good so far ?
I am wondering what to use in place of the ???, because no action is to 
be taken in the backend, this serves only as a way to use the ACL and 
mark the IP using gpc1 so that the frontend sends its further 
connections to another backend.


Thanks in advance.

Sylvain Faivre.