Re: Consistent hashing based on cookie - across multiple HAProxy boxes

[Alex Davies]

For those interested, this turned out to be some wierdness in haproxy - one
of the servers was only storing the first 24 (despite being configured for
32, and the other identical binary/config working fine).

The way to debug was the following on both servers for a given cookie
value:

echo show table webservers | socat /var/lib/haproxy/stats - | fgrep
4start_of_session

From my config, it was also important to store text not binary data in the
table.

-Alex


On Sat, Mar 2, 2013 at 5:32 PM, Alex Davies a...@davz.net wrote:

 I was trying to troubleshoot this with a packet dump on the peer
 traffic. The raw tcpdump does not mean anything to me, and Wireshark is
 decoding it as Java RMI traffic which isnt much use, and it looks like a
 binary protocol.

 So I can report that 'peer' traffic is certainly working because when I
 restart one of the haproxy processes there are a bunch of different packets
 sent (Wireshark doesent even attempt to decode these). I was hoping to be
 able to grep for one of the offending session IDs to see if that gave me a
 hint.

 Does anybody have a working configuration for multi peer persistence based
 on a cookie that I could test, by chance?

 Thanks,

 -Alex




-- 
Alex Davies

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the sender immediately by e-mail and delete this e-mail permanently.

[PATCH] DOCS: Add explanation of intermediate certs to crt paramater

[Alex Davies]

This change makes the crt block of the documentation easier to use
for those not clear on what needs to go in what file, specifically for
those using CAs that require intermediate certificates.
---
 doc/configuration.txt |   44 +---
 1 files changed, 29 insertions(+), 15 deletions(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index 398ce87..a94fe21 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -7107,23 +7107,37 @@ crl-file crlfile

 crt cert
   This setting is only available when support for OpenSSL was built in.
-  It designates a PEM file from which to load both a certificate and the
-  associated private key. This file can be built by concatenating both PEM
-  files into one. If the OpenSSL used supports Diffie-Hellman, parameters
-  present in this file are also loaded. If a directory name is used
instead of a
-  PEM file, then all files found in that directory will be loaded. This
-  directive may be specified multiple times in order to load certificates
from
-  multiple files or directories. The certificates will be presented to
clients
-  who provide a valid TLS Server Name Indication field matching one of
their CN
-  or alt subjects. Wildcards are supported, where a wildcard character '*'
is
-  used instead of the first hostname component (eg: *.example.org matches
-  www.example.org but not www.sub.example.org). If no SNI is provided by
the
-  client or if the SSL library does not support TLS extensions, or if the
client
-  provides and SNI which does not match any certificate, then the first
loaded
-  certificate will be presented. This means that when loading certificates
from
-  a directory, it is highly recommended to load the default one first as a
file.
+  It designates a PEM file containing both the required certificates and
any
+  associated private keys.  This file can be built by concatenating
multiple
+  PEM files into one (e.g. cat cert.pem key.pem  combined.pem). If your CA
+  requires an intermediate  certificate, this can also be concatenated into
+  this file.
+
+  If the OpenSSL used supports Diffie-Hellman, parameters present in this
file
+  are loaded.
+
+  If a directory name is used instead of a PEM file, then all files found
in
+  that directory will be loaded. This directive may be specified multiple
times
+  in order to load certificates from multiple files or directories. The
certificates
+  will be presented to clients who provide a valid TLS Server Name
Indication
+  field matching one of their CN or alt subjects. Wildcards are supported,
where
+  a wildcard character '*' is used instead of the first hostname component
+  (eg: *.example.org matches www.example.org but not www.sub.example.org).
+
+  If no SNI is provided by the client or if the SSL library does not
support TLS
+  extensions, or if the client provides an SNI hostname which does not
match any
+  certificate, then the first loaded certificate will be presented. This
means
+  that when loading certificates from a directory, it is highly recommended
+  to load the default one first as a file.
+
   Note that the same cert may be loaded multiple times without side
effects.

+  Some CAs (such as Godaddy) offer a drop down list of server types that do
+  not include HAProxy when obtaining a certificafte. If this happens be
sure
+  to choose a webserver that the CA believes requires a intermediate CA
+  (for Godaddy, selection Apache Tomcat will get the correct bundle, but
many
+  others, e.g. nginx, result in a wrong bundle that will not work for some
clients).
+
 crt-ignore-err errors
   This setting is only available when support for OpenSSL was built in.
   Sets a comma separated list of errorIDs to ignore during verify at depth
== 0.
--
1.7.1

Re: Experience with HAProxy SSL intermediate certificates - Godaddy and others

[Alex Davies]

Hi Willy,

I've just sent a simple patch for this (sorry for the delay). While
preparing this, I was not sure what the difference is between the block in
section 5.2 (Server and default-server options) and the main documentation;
it seems in the case of crt the former has the first few sentances and
misses off the last few. Since i'm making teh crt block significantly
longer, it seems silly to duplicate this - at the same time it would be a
shame for someone to read the top version. I wonder if its worth mentioning
that many of the settings are further explained, in more detail, lower down
the file in section 4.2, but i've not included this in my patch because its
possible i'm missing something!

I considered adding something mentioning the very useful tools in my email,
but it seems wrong to signpost the ones I selected; I am far from an expert
in this area. If there was a intro to SSL section in teh docs, i'd write
something explaining that its important to check that non-SNI compliant
browsers and ones with older root certificate databases (Java being the one
that bit me - as PayPal use it for their API requests) rather than just
testing in modern IE/Chrome/FF. But i'm not quite sure where to put this,
so i've left it out; if you think this would be helpful i'd be very happy
to add it.

Thanks,

Alex




On Sun, Feb 10, 2013 at 10:23 PM, Willy Tarreau w...@1wt.eu wrote:

 Hi Alex,

 On Sun, Feb 10, 2013 at 08:46:46PM +, Alex Davies wrote:
  Hi All,
 
  I saw some traffic on the list from Guillaume and others about this, but
 I
  thought i'd confirm a real world production use of a GoDaddy[1] SSL
  certificate (which requires an intermediate certificates for some use
  cases, specifically Java applications).
 
  I used the following line in haproxy.cfg:
 
  bind :443 ssl crt /path/to/domain.com.crt ca-file /path/to/ca.pem
  ciphers ECDHE-RSA-AES256-SHA:RC4-SHA:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM
 
  In /path/to/domain.com.crt, I have
  My private key (the one used to generate the CSR)
  The final certificate for my domain (downloaded from the provider)
  The intermediate certificate (from the .zip file with the CA bundle
  provided by godaddy)
 
  In /path/to/ca.pem I just took the gd_bundle file provided by GoDaddy.
 
  I took the ciphers restriction from an exceliance blog post[2]; it seems
 to
  effectively shut up the BEAST attack alerts on teh SSL checker websites.
 
  I found the utility at https://www.digicert.com/util/ and the online
 check
  at https://www.ssllabs.com/ssltest/ super useful.
 
  I wonder if it might be helpful to ammend the crt cert documentation
 to
  say that Intermediate certificates should be loaded with this (either
 with
  multiple crt lines, or concatenation as it already says). This was not
  obvious to me, and I spent some time failing to get it working using the
  intermediate certificate within the SSL.
 
  I have said it before, but i'd like to say it again - HAProxy is awesome,
  and the removal of one more chain in the link is fantastic. Thanks guys!

 Thank you for the informative feedback. You're right, if it took you some
 time to figure how to do this, then the doc needs to be updated. Could you
 propose some prose that you think would have helped you ? That would be
 really great. Do you think it would also be worth adding some scripts in
 the contrib/ directory to better handle the certificates concatenation
 into a single file ? Or maybe to check some certs, I don't know. You surely
 have ideas after all the tests you've performed !

  Thanks,
 
  -Alex
 
  [1] Godaddy ask you to choose your type of SSL provider. I selected
 nginx
  as I have used that before, but for some reason it does not provide the
  intermediate certificate in this case. Select Apache Tomcat to get a zip
  file with both gd_bundle and the intermediate Cert.

 This could go into the doc as well, until they add haproxy :-)

 Cheers,
 Willy




-- 
Alex Davies

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the sender immediately by e-mail and delete this e-mail permanently.

Re: Consistent hashing based on cookie - across multiple HAProxy boxes

[Alex Davies]

I was trying to troubleshoot this with a packet dump on the peer traffic.
The raw tcpdump does not mean anything to me, and Wireshark is decoding it
as Java RMI traffic which isnt much use, and it looks like a binary
protocol.

So I can report that 'peer' traffic is certainly working because when I
restart one of the haproxy processes there are a bunch of different packets
sent (Wireshark doesent even attempt to decode these). I was hoping to be
able to grep for one of the offending session IDs to see if that gave me a
hint.

Does anybody have a working configuration for multi peer persistence based
on a cookie that I could test, by chance?

Thanks,

-Alex

Re: Consistent hashing based on cookie - across multiple HAProxy boxes

[Alex Davies]

H Baptiste and others,

I've not been able to get this to work. I have two different haproxy
servers with identical config (relevant sections below). When I hit each
one with the same PHPSESSID (domain set to .domain.tld) I get put on a
(different) consistent server. I've setup cookie capturing in the log to
show that PHPSESSID is good; I can telnet across from one peer to the
other. I've started both process with -L peer_name_and_also_hostname, but
it still wont work. I'm running 1.5-dev17.

Relevant log - server 1
[root@frontend1 ~]# tail -f /var/log/haproxy.log | grep /info.php
Feb 28 22:55:24 localhost haproxy[18069]:
213.146.182.195:47073[28/Feb/2013:22:55:11.237] main webservers/tc_ww6
13063/0/1/0/13064 200 318
PHPSESSID=d40c28be77d7c9e364ae9fd35f87f62b   11670/11670/17/1/0 0/0
GET /info.php HTTP/1.1

Relevant log - server 2
[root@frontend2 ~]# tail -f /var/log/haproxy.log | grep info.php
Feb 28 22:55:37 localhost haproxy[30244]:
213.146.182.195:47120[28/Feb/2013:22:55:36.991] main webservers/tc_ww1
7/0/0/3/10 200 318
PHPSESSID=d40c28be77d7c9e364ae9fd35f87f62b   5/5/0/0/0 0/0 GET
/info.php HTTP/1.1


Connection from 1 to 2:
[root@frontend1 ~]# telnet 10.99.99.162 1099
Trying 10.99.99.162...
Connected to 10.99.99.162.
Escape character is '^]'.
SOMETHING
501
Connection closed by foreign host.

Relevant config:

peers frontends
peer frontend1 10.99.99.161:1099
peer frontend2 10.99.99.162:1099

frontend main
bind :80
capture cookie PHPSESSID len 50
..

backend webservers
balance roundrobin
stick on cookie(PHPSESSID)
stick-table type string size 32k peers frontends expire 24h
stick store-response set-cookie(PHPSESSID)

server   ww1 10.99.99.201:80 weight 5  check observe layer4 inter 1
server   ww2 10.99.99.202:80 weight 6 check observe layer4 inter 1
server   ww3 10.99.99.203:80 weight 10 check observe layer4 inter 1
...


Thanks,

Alex


On Sun, Feb 10, 2013 at 9:47 PM, Baptiste bed...@gmail.com wrote:

 Hi,

 It's weird you don't need the store-response.
 Test it as best as you can to avoid any issues later.

 There is no way to monitor the peering, as far as I know.
 If a peer is unavailable, it means both haproxy can't reach each
 others. so you may have a bigger issue to fix before :)
 This is a multi-master way of working: each haproxy notifies its peers
 about the changes it made into the table.

 cheers



 On Sun, Feb 10, 2013 at 9:36 PM, Alex Davies a...@davz.net wrote:
  Hi Baptiste (apologies!)
 
  I did some testing with this, it actually seemed that stick
 store-response
  was not needed.
 
  I think this is working, at least it works in my test environment. Will
 test
  it for real shortly.
 
  One question I had - what is the best way to monitor this peering? I
 didnt
  see anything in the logs about the peers, nor anything in the haproxy
 status
  URL. Obviously if the peering were to silently break down, this would be
 bad
  for me!
 
  Thanks,
 
  -Alex
 
 
  On Fri, Feb 8, 2013 at 4:22 AM, Baptiste bed...@gmail.com wrote:
 
  ahah, you can call me Baptiste :)
 
  You miss a stick on cookie(PHPSESSID).
  Also consider using the same expire delay you have on your application
  server.
 
  And last but not least, add a peers section (and a peer directive on
  the stick-table definition) where you provide all your HAProxy server
  IPs in order to get the table of each HAProxy synchronized.
 
  then you're done.
 
  Baptiste
 
  On 2/8/13, Alex Davies a...@davz.net wrote:
   Hi Willy,
  
   Thanks for your suggestion. I'm guessing you mean something like this
   backend:
  
   backend x
   balance roundrobin
   stick-table type string size 32k peers other_haproxy_server expire
   24h
   stick store-response set-cookie(PHPSESSID)
  
   If I understand you correctly, you are saying that this will only mean
   that
   sessions become persistant once PHPSESSID is set. So, to translate
   into
   practicality, as long as the login page creates the relevant cooke
 (and
   it
   does not subsequently change once logged in), this should work nicely.
  
   Thanks,
  
   -Alex
  
  
  
   On Sun, Feb 3, 2013 at 7:59 AM, Baptiste bed...@gmail.com wrote:
  
   Hi,
  
   the only way you could do what you want for now is using stick tables
   (and haproxy 1.5-dev17).
   You can learn the Set-Cookie from the server and match the Cookie in
   the table from the client.
   That way, all the request from a user will be sent to the same
 server,
   from the first to the last one.
  
   Today, haproxy is able to hash a HTTP header for load-balancing, so a
   configuration like:
balance hdr(Cookie)
   could do the trick, but it means that ALL clients cookie to
   load-balance. And worste, since there is no phpsessionid cookie on
 the
   first request, there are chances that the first and the second
   requests won't be routed to the same server.
  
   I guess it would be possible soon to have a:
balance cook

Re: Consistent hashing based on cookie - across multiple HAProxy boxes

[Alex Davies]

Hi Baptiste (apologies!)

I did some testing with this, it actually seemed that stick store-response
was not needed.

I think this is working, at least it works in my test environment. Will
test it for real shortly.

One question I had - what is the best way to monitor this peering? I
didnt see anything in the logs about the peers, nor anything in the haproxy
status URL. Obviously if the peering were to silently break down, this
would be bad for me!

Thanks,

-Alex


On Fri, Feb 8, 2013 at 4:22 AM, Baptiste bed...@gmail.com wrote:

 ahah, you can call me Baptiste :)

 You miss a stick on cookie(PHPSESSID).
 Also consider using the same expire delay you have on your application
 server.

 And last but not least, add a peers section (and a peer directive on
 the stick-table definition) where you provide all your HAProxy server
 IPs in order to get the table of each HAProxy synchronized.

 then you're done.

 Baptiste

 On 2/8/13, Alex Davies a...@davz.net wrote:
  Hi Willy,
 
  Thanks for your suggestion. I'm guessing you mean something like this
  backend:
 
  backend x
  balance roundrobin
  stick-table type string size 32k peers other_haproxy_server expire
 24h
  stick store-response set-cookie(PHPSESSID)
 
  If I understand you correctly, you are saying that this will only mean
 that
  sessions become persistant once PHPSESSID is set. So, to translate into
  practicality, as long as the login page creates the relevant cooke (and
 it
  does not subsequently change once logged in), this should work nicely.
 
  Thanks,
 
  -Alex
 
 
 
  On Sun, Feb 3, 2013 at 7:59 AM, Baptiste bed...@gmail.com wrote:
 
  Hi,
 
  the only way you could do what you want for now is using stick tables
  (and haproxy 1.5-dev17).
  You can learn the Set-Cookie from the server and match the Cookie in
  the table from the client.
  That way, all the request from a user will be sent to the same server,
  from the first to the last one.
 
  Today, haproxy is able to hash a HTTP header for load-balancing, so a
  configuration like:
   balance hdr(Cookie)
  could do the trick, but it means that ALL clients cookie to
  load-balance. And worste, since there is no phpsessionid cookie on the
  first request, there are chances that the first and the second
  requests won't be routed to the same server.
 
  I guess it would be possible soon to have a:
   balance cook(PHPSessionID)
  but it won't fix the sticking issue between first and second request
  since the cookie is not present in the first request.
 
  So if you really want using the algorithm method, you must be able to
  share cookies between your backend servers, only lucky people will be
  able to get authenticated.
  Well, maybe there are some dirty tricks like managing a farm for
  cookie-less clients and configuring PHP to learn an unknown session on
  the fly.
 
  Baptiste
 
 
  On Sun, Feb 3, 2013 at 2:03 AM, Alex Davies a...@davz.net wrote:
   Hi All,
  
   What is the best way to configure haproxy to hash based on an
   application
   cookie (such as PHPSESSID), in a way that is consistent (meaning
   multiple
   haproxy servers will route to the same backend), ideally including the
   ability to weight backends (the configuration would clearly have to be
  the
   same on these different boxes).
  
   appsession obviously allows this for a single HAProxy server, but it
  seems
   from the documentation that it generates a server based on the hash at
  the
   start of each session, so if the same session hit a different but
   identically configured haproxy server it would end up with a
  
   Thanks,
  
   -Alex
 
 




-- 
Alex Davies

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the sender immediately by e-mail and delete this e-mail permanently.

Experience with HAProxy SSL intermediate certificates - Godaddy and others

[Alex Davies]

Hi All,

I saw some traffic on the list from Guillaume and others about this, but I
thought i'd confirm a real world production use of a GoDaddy[1] SSL
certificate (which requires an intermediate certificates for some use
cases, specifically Java applications).

I used the following line in haproxy.cfg:

bind :443 ssl crt /path/to/domain.com.crt ca-file /path/to/ca.pem
ciphers ECDHE-RSA-AES256-SHA:RC4-SHA:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM

In /path/to/domain.com.crt, I have
My private key (the one used to generate the CSR)
The final certificate for my domain (downloaded from the provider)
The intermediate certificate (from the .zip file with the CA bundle
provided by godaddy)

In /path/to/ca.pem I just took the gd_bundle file provided by GoDaddy.

I took the ciphers restriction from an exceliance blog post[2]; it seems to
effectively shut up the BEAST attack alerts on teh SSL checker websites.

I found the utility at https://www.digicert.com/util/ and the online check
at https://www.ssllabs.com/ssltest/ super useful.

I wonder if it might be helpful to ammend the crt cert documentation to
say that Intermediate certificates should be loaded with this (either with
multiple crt lines, or concatenation as it already says). This was not
obvious to me, and I spent some time failing to get it working using the
intermediate certificate within the SSL.

I have said it before, but i'd like to say it again - HAProxy is awesome,
and the removal of one more chain in the link is fantastic. Thanks guys!

Thanks,

-Alex

[1] Godaddy ask you to choose your type of SSL provider. I selected nginx
as I have used that before, but for some reason it does not provide the
intermediate certificate in this case. Select Apache Tomcat to get a zip
file with both gd_bundle and the intermediate Cert.
[2]
http://blog.exceliance.fr/2013/01/21/mitigating-the-ssl-beast-attack-using-the-aloha-load-balancer-haproxy/

Re: Consistent hashing based on cookie - across multiple HAProxy boxes

[Alex Davies]

Hi Willy,

Thanks for your suggestion. I'm guessing you mean something like this
backend:

backend x
balance roundrobin
stick-table type string size 32k peers other_haproxy_server expire 24h
stick store-response set-cookie(PHPSESSID)

If I understand you correctly, you are saying that this will only mean that
sessions become persistant once PHPSESSID is set. So, to translate into
practicality, as long as the login page creates the relevant cooke (and it
does not subsequently change once logged in), this should work nicely.

Thanks,

-Alex



On Sun, Feb 3, 2013 at 7:59 AM, Baptiste bed...@gmail.com wrote:

 Hi,

 the only way you could do what you want for now is using stick tables
 (and haproxy 1.5-dev17).
 You can learn the Set-Cookie from the server and match the Cookie in
 the table from the client.
 That way, all the request from a user will be sent to the same server,
 from the first to the last one.

 Today, haproxy is able to hash a HTTP header for load-balancing, so a
 configuration like:
  balance hdr(Cookie)
 could do the trick, but it means that ALL clients cookie to
 load-balance. And worste, since there is no phpsessionid cookie on the
 first request, there are chances that the first and the second
 requests won't be routed to the same server.

 I guess it would be possible soon to have a:
  balance cook(PHPSessionID)
 but it won't fix the sticking issue between first and second request
 since the cookie is not present in the first request.

 So if you really want using the algorithm method, you must be able to
 share cookies between your backend servers, only lucky people will be
 able to get authenticated.
 Well, maybe there are some dirty tricks like managing a farm for
 cookie-less clients and configuring PHP to learn an unknown session on
 the fly.

 Baptiste


 On Sun, Feb 3, 2013 at 2:03 AM, Alex Davies a...@davz.net wrote:
  Hi All,
 
  What is the best way to configure haproxy to hash based on an application
  cookie (such as PHPSESSID), in a way that is consistent (meaning multiple
  haproxy servers will route to the same backend), ideally including the
  ability to weight backends (the configuration would clearly have to be
 the
  same on these different boxes).
 
  appsession obviously allows this for a single HAProxy server, but it
 seems
  from the documentation that it generates a server based on the hash at
 the
  start of each session, so if the same session hit a different but
  identically configured haproxy server it would end up with a
 
  Thanks,
 
  -Alex

Consistent hashing based on cookie - across multiple HAProxy boxes

[Alex Davies]

Hi All,

What is the best way to configure haproxy to hash based on an application
cookie (such as PHPSESSID), in a way that is consistent (meaning multiple
haproxy servers will route to the same backend), ideally including the
ability to weight backends (the configuration would clearly have to be the
same on these different boxes).

appsession obviously allows this for a single HAProxy server, but it seems
from the documentation that it generates a server based on the hash at the
start of each session, so if the same session hit a different but
identically configured haproxy server it would end up with a

Thanks,

-Alex

Re: Rate limiting- queueing requests

[Alex Davies]

Hi Willy,

I never replied to this mail, my apologies!

Thank you for your suggestions. Sadly, in our case, neither approach works
- but it was worth asking you first. I solved our problem  with something
that executes right at the start of our application,

Thank you!

Alex


On Wed, Nov 28, 2012 at 7:20 AM, Willy Tarreau w...@1wt.eu wrote:

 Hi Alex,

 On Tue, Nov 27, 2012 at 11:41:08PM +, Alex Davies wrote:
  Hey All,
 
  I have an application that can only handle. to some URLs, one request
 every
  x seconds from each session (identified by a specific cookie).
 
  Rather than adding logic to handle this to the application itself (which
 I
  fear I will have to do), I would like to know if it is possible to use
 the
  rate limiting functionality in HAProxy to delay the sending of a request
 to
  the backend to the time the last request was sent + x fraction of a
 second?
 
  I see a bunch of example configs on the internet to reject connections
 over
  x per second but nothing that queues requests in a order.

 If you have just a few such URLs, one thing you could do which will
 approach
 your need is to have a backend per URL (or per group of URL), in which a
 tcp-request content rule causes artificially long pauses when the session
 rate is too high. This can work well as long as the number of concurrent
 connections on that backend remains low and known (so that you can adjust
 the timer). If you need only one connection at a time, you could chain
 a server with maxconn 1 to such an installation.

 A simple example would consist in this :

 frontend front
 use_backend limited if { path_beg /foo /bar }

 # limited to 2 requests per second
 backend limited
 tcp-request inspect-delay 500
 tcp-request content accept if { be_sess_rate le 2 } || WAIT_END
 server ...

 This can work well for printing devices or PDF generators for example.
 It won't work well at all if you don't know the number of concurrent
 users, in which case it could be done like this (even uglier) :

 frontend front
 use_backend serialize if { path_beg /foo /bar }

 # serialize requests
 backend serialize
 server limited 127.0.0.1:1 maxconn 1 send-proxy

 # limited to 2 requests per second
 listen limited
 bind 127.0.0.1:1 accept-proxy
 tcp-request inspect-delay 500
 tcp-request content accept if { be_sess_rate le 2 } || WAIT_END
 server ...

 This time it will do the job whatever the number of concurrent clients.
 But it's not very pretty...

 Willy




-- 
Alex Davies

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the sender immediately by e-mail and delete this e-mail permanently.

Rate limiting- queueing requests

[Alex Davies]

Hey All,

I have an application that can only handle. to some URLs, one request every
x seconds from each session (identified by a specific cookie).

Rather than adding logic to handle this to the application itself (which I
fear I will have to do), I would like to know if it is possible to use the
rate limiting functionality in HAProxy to delay the sending of a request to
the backend to the time the last request was sent + x fraction of a second?

I see a bunch of example configs on the internet to reject connections over
x per second but nothing that queues requests in a order.

Thanks!

-Alex

Source IP rate limiting

[Alex Davies]

Hi,

I am interested in rate limiting connections from users to stop small DOS
'attacks' from individual users.

I see the excellent post at http://blog.serverfault.com/post/1016491873/ and
have followed this in a test enviroment.

I have the following questions:
* What is the best way to monitor the # of connections that are being
rejected as a result of this from the log? The socat example in that post
seems - to me - to show the number of IPs in the relevant tables as opposed
to the number of connections that are being rejected. Is it possible also
to know which 'reject' the request is blocked by (from the example post
there are 2)
* Is it possible to 'hash' on a specific cookie value (i'm thinking
PHPSESSID) as well as IP, i.e. if connections for any given PHPSESSID value
reaches x per minute block?

Many thanks,

Alex

-- 
Alex Davies

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the sender immediately by e-mail and delete this e-mail permanently.

Re: Source IP rate limiting

[Alex Davies]

Hi,

Thanks for your reply.

What is the 'signature' that is unique in the logs for this kind of block?
I have HTTP logs enabled; for the Session state at disconnection part of
the log would I be looking for a first letter of P or R for clients blocked
by these filters? (and it it possible to distinguish clients blocked
because of these ACLs compared to other reasons for a P or R?

I am using a command like this:

[root@frontend2 ~]# tail -f /var/log/haproxy.log | awk -F' ' '{ if ($15 ~
/^P/) print $0 }'
Nov 10 16:17:26 localhost haproxy[11102]:
64.41.0.39:4192[10/Nov/2011:16:17:26.788] main webservers/NOSRV
-1/-1/-1/-1/15 -1 0 - -
PRNN 7694/7694/119/0/30 0/0 GET /images/face_05.gif HTTP/1.1

But in this particular case the problem is (from my reading) that the
client failed to send a complete request, not a ACL.

As for a sticky table to store the PHPSESSID and block, am I along the
right track with a config like this?:

backend x
stick-table type string size 500k
stick store-request cookie(PHPSESSID) table cookie_sc2_conn_rate
acl cookie_conn_rate_abuse  cookie_sc2_conn_rate gt 300
acl mark_as_abuser   sc1_inc_gpc0 gt 0
tcp-request content  reject if cookie_conn_rate_abuse mark_as_abuser

Many thanks,

Alex



On Thu, Nov 10, 2011 at 12:56 PM, Baptiste bed...@gmail.com wrote:

 On Thu, Nov 10, 2011 at 12:48 PM, Alex Davies a...@davz.net wrote:
  Hi,
  I am interested in rate limiting connections from users to stop small DOS
  'attacks' from individual users.
  I see the excellent post at http://blog.serverfault.com/post/1016491873/
  and
  have followed this in a test enviroment.
  I have the following questions:
  * What is the best way to monitor the # of connections that are being
  rejected as a result of this from the log? The socat example in that post
  seems - to me - to show the number of IPs in the relevant tables as
 opposed
  to the number of connections that are being rejected. Is it possible
 also to
  know which 'reject' the request is blocked by (from the example post
 there
  are 2)
  * Is it possible to 'hash' on a specific cookie value (i'm thinking
  PHPSESSID) as well as IP, i.e. if connections for any given PHPSESSID
 value
  reaches x per minute block?
  Many thanks,
  Alex
  --
  Alex Davies
 

 Hin,

 You can know the numbre of rejected request through the logs.

 You can use a str stick table and store the PHPSESSID in it.

 And you can capture the cookie value in the logs as well to know how
 many request have been rejected.

 cheers




-- 
Alex Davies

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the sender immediately by e-mail and delete this e-mail permanently.

Re: haproxy - kernel socket and # files limits

[Alex Davies]

Hi Willy,

Thank you for your detailed explanation; you are entirely right and the
limit that was screwing me was the limit of fds/process which I have now
fixed.

Can you explain which parameter should be set to 2 to prevent malloc() fails
when there is no more memory left?

Many thanks,

Alex


On Tue, Oct 4, 2011 at 6:49 AM, Willy Tarreau w...@1wt.eu wrote:

 Hi Alex,

 On Tue, Oct 04, 2011 at 01:38:40AM +0100, Alex Davies wrote:
  Hi,
 
  I am running haproxy 1.4.18 on RHEL6 (64 bit). I am trying to run a
  large number of connections (I have long pooling backends), ideally
  several hundred k. My servers have 24G RAM and are doing nothing else.
  I have two problems.
 
  Firstly, if I increase teh nofile hard limit in
  /etc/security/limits.conf to anything over 1048576 (1 Megabyte,
  although I think this is defined as a # of files) I can't SSH to the
  box or start haproxy - new SSH connections are immediately closed
  without entries in the logs. The kernel limit (/proc/sys/fs/file-max)
  is set to 200k without problem. Haproxy balks as follows:
 
  Starting haproxy: [WARNING] 276/012759 (2924) :
  [/usr/sbin/haproxy.main()] Cannot raise FD limit to 246.
  [WARNING] 276/012759 (2924) : [/usr/sbin/haproxy.main()] FD limit
  (1048576) too low for maxconn=100/maxsock=246. Please raise
  'ulimit-n' to 246 or more to avoid any trouble.

 There is a default limit of 1048576 fds per process, which you can
 change with nr_open. But be careful when increasing all those values!
 Some programs might not like it at all and some scripts may become
 slow as hell when they close every possible fd before a fork(). So
 you don't want to have everything running unlimited. Ideally, keep
 the default ulimit at 1024 and let haproxy change it without impacting
 other processes.

 Note: the highest test that was done on haproxy was with 1M fds, nothing
 above (about 480k conns).

  Secondly, I notice that as my number of connections rises I get Out
  of socket memory errors in the kernel log. Google led me to believe
  that this would be a problem caused by excessive orphan sockets, but
  this seems not to be the case:

 I don't believe in orphans too for this case, because normally you see
 a message too many of orphans long before running out of memory, since
 orphans are often limited to a small number.

  [root@frontend2 log]# cat /proc/net/sockstat | grep orphan
  TCP: inuse 134376 orphan 0 tw 87 alloc 150249 mem 100217
  (output taken during a time of problem, where haproxy was mostly
  inaccessible and most connections were refused)
 
  I also seem to have plenty of spare sockets:
 
  [root@frontend2 log]# cat /proc/sys/net/ipv4/tcp_mem
  2303712 3071616 4607424
  [root@frontend2 log]# cat /proc/net/sockstat
  sockets: used 150458
  TCP: inuse 134376 orphan 1 tw 69 alloc 150250 mem 100217
  UDP: inuse 12 mem 3
  UDPLITE: inuse 0
  RAW: inuse 0
  FRAG: inuse 0 memory 0
 
  (100217  4607424 by some margin).

 The usual thing is when some socket buffers are too large. For instance,
 the default 87k read and 64k write buffers imply that you consume 150k
 by default for each socket, until the kernel starts to reclaim socket
 memory. All the people I know who use very large amounts of sockets use
 small buffers, both in haproxy and in the kernel, because most often
 it's for chat/websocket or such things which transport few data. You
 could very well set your default kernel TCP buffers to 8k and let them
 grow larger if needed :

   tcp_rmem  4096 8192 65536
   tcp_wmem  4096 8192 65536

 You should never forget that a full connection through haproxy uses :
  - 2 TCP sockets, so 2 tcp_rmem and 2 tcp_wmem
  - 2 haproxy buffers (one per direction), each of global.bufsize.

 So if you set haproxy to 8kB and your kernel to 8kB, you will need
 16kB for haproxy and 32kB for the system = 48kB per connection. That's
 already too much for 1M connections at 24 GB. The kernel will never
 go below 4kB, and at this point you'll have 32kB total = 32 GB of RAM
 per million connection.

  Ideally i'd like to set the number of open files and sockets to as
  close to infinite as possible - and allow them to use all the system
  RAM.

 You will use all the system's RAM if socket buffers are properly tuned.
 I'm used to say that you need 1/3 of the RAM for TCP buffers, another 1/3
 for haproxy and the last 1/3 for the rest of the system (caches, ...).
 With 24 GB, you can even assume close to 12+12 and almost nothing for
 the system. Your TCP settings already allow you to use up to 18 GB RAM
 for sockets. Haproxy will allocate what it needs for its maxconn, so in
 practice you can max out memory. It is even a bit dangerous, you should
 ensure that the system will never overcommit memory (set the param to 2)
 so that it fails on malloc() when the limit is reached instead of
 pretending the memory exists and killing the process when it tries to
 use it. But that's for a second point.

 Regards,
 Willy




-- 
Alex

haproxy - kernel socket and # files limits

[Alex Davies]

Hi,

I am running haproxy 1.4.18 on RHEL6 (64 bit). I am trying to run a
large number of connections (I have long pooling backends), ideally
several hundred k. My servers have 24G RAM and are doing nothing else.
I have two problems.

Firstly, if I increase teh nofile hard limit in
/etc/security/limits.conf to anything over 1048576 (1 Megabyte,
although I think this is defined as a # of files) I can't SSH to the
box or start haproxy - new SSH connections are immediately closed
without entries in the logs. The kernel limit (/proc/sys/fs/file-max)
is set to 200k without problem. Haproxy balks as follows:

Starting haproxy: [WARNING] 276/012759 (2924) :
[/usr/sbin/haproxy.main()] Cannot raise FD limit to 246.
[WARNING] 276/012759 (2924) : [/usr/sbin/haproxy.main()] FD limit
(1048576) too low for maxconn=100/maxsock=246. Please raise
'ulimit-n' to 246 or more to avoid any trouble.

Secondly, I notice that as my number of connections rises I get Out
of socket memory errors in the kernel log. Google led me to believe
that this would be a problem caused by excessive orphan sockets, but
this seems not to be the case:

[root@frontend2 log]# cat /proc/net/sockstat | grep orphan
TCP: inuse 134376 orphan 0 tw 87 alloc 150249 mem 100217
(output taken during a time of problem, where haproxy was mostly
inaccessible and most connections were refused)

I also seem to have plenty of spare sockets:

[root@frontend2 log]# cat /proc/sys/net/ipv4/tcp_mem
2303712 3071616 4607424
[root@frontend2 log]# cat /proc/net/sockstat
sockets: used 150458
TCP: inuse 134376 orphan 1 tw 69 alloc 150250 mem 100217
UDP: inuse 12 mem 3
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

(100217  4607424 by some margin).

Ideally i'd like to set the number of open files and sockets to as
close to infinite as possible - and allow them to use all the system
RAM.

Would anybody with more knowledge in this area be able to shed any light?

- Alex

Re: haproxy / Python 'tornado' framework - digging into 502/504 errors

[Alex Davies]

Hi Willy, Cyril,

Thank you for your detailed analysis. I still notice 504 errors almost
immediately on a HAproxy start, and the PID matches the new process:

[root@frontend2 log]# ps aux | grep haproxy
haproxy  21242  6.6  0.1 133176 47984 ?Rs   07:17   0:00
/usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
[root@frontend2 log]# service haproxy stop; service rsyslog stop; rm -f
/var/log/haproxy.log;  service rsyslog start; service haproxy start;
Stopping haproxy:  [  OK  ]
Shutting down system logger:   [  OK  ]
Starting system logger:[  OK  ]
Starting haproxy:  [  OK  ]
[root@frontend2 log]# tail -f haproxy.log | grep 504
Sep 14 07:16:14 localhost haproxy[21178]:
94.197.40.185:3504[14/Sep/2011:07:16:08.216] main
python_8001/python_8001_fe1 80/0/0/-1/6449
502 204 - - SH-- 3375/3375/950/950/0 0/0 POST /xxx/chat/status/updates
HTTP/1.1
Sep 14 07:16:15 localhost haproxy[21178]:
118.101.95.88:49504[14/Sep/2011:07:16:10.298] main
python_9003/python_9003_fe1 22/0/0/-1/5088
502 204 - - SH-- 3312/3312/386/386/0 0/0 POST /xxx/chat/message/3/updates
HTTP/1.1
^C
[root@frontend2 log]# ps aux | grep haproxy
haproxy  21178  5.4  0.2 137268 51480 ?Ss   07:16   0:01
/usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid

I don't understand how this is - can you shed any light? (the configuration
is the same as in my email - i.e. server timeout 2hrs).

I will investigate the 502 errors with more benchmarking directly against
the backends. Thank you for confirming that this is the source of the
problem.

Many thanks,

Alex


On Tue, Sep 13, 2011 at 11:41 PM, Willy Tarreau w...@1wt.eu wrote:

 Hi Alex,

 On Tue, Sep 13, 2011 at 03:18:54PM +0100, Alex Davies wrote:
  Hi,
 
  Thank you for your observation - indeed I did notice some of those as I
 was
  writing my email - I have updated my globals to increase the server
 timeout
  (as we are doing long polling) and reduce the others, and remove the
  duplicates:
 
  defaults
  mode  http
  optionhttplog
  #option   tcplog
  optiondontlognull
  optiondontlog-normal
 
  log   global
  retries   10
  maxconn   5
  option forwardfor except 127.0.0.1/32 # Apache on https://127.0.0.1
   option httpclose  # Required for REMOTE HEADER
  option redispatch
 
  timeout connect 1
  timeout client  1
  timeout server  720
 
  I still notice the same errors in the logs! (slightly less 504, as I
 would
  expect through the increase in timeout server - but I still don't
  understand why I get any at all in the first minute of a new process).

 To complete Cyril's detailed analysis, I'd like to add that you'll only
 see 502s when you restart, and it will take some time before you see 504s
 again (eg: 2 hours with the config above).

 The 502s mean that the server has suddenly aborted the connection (flags
 SH),
 while the 504s indicate that haproxy was fed up with waiting and closed
 after
 timeout server was elapsed.

 So yes it's very possible that your server has its own timeout, but it
 should
 be in the 30s from what I saw in your logs. It sill does not explain why
 some
 requests never time out on the server, maybe they don't wake the same
 components
 up ?

 Regards,
 Willy




-- 
Alex Davies

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the sender immediately by e-mail and delete this e-mail permanently.

haproxy / Python 'tornado' framework - digging into 502/504 errors

[Alex Davies]

Hi,

I am not a haproxy expert, but have been using it in production for some
time with excellent results and I wonder if I can seek some expert advice on
running the fairly fast application server http://www.tornadoweb.org/ behind
HAproxy (haproxy-1.3.23 using the EPEL RPM (-1) on RHEL6 x86_64). Haproxy is
working very well for me, but i'm looking for help understanding how I can
diagnose problems with the Tornado application I have running behind it.

I have ~8 tornado processes running on two servers. Its important that one
is active and the other is failover (some state is stored in memory). The
parts of my haproxy configuration relevant to my question are below. I
notice a large number of entries in the logs like this:

502 errors:
Sep 13 12:42:45 localhost haproxy[15128]:
188.222.50.208:61001[13/Sep/2011:12:42:43.881] main
python_8001/python_8001_fe1 10/0/0/-1/1527
502 204 - - SH-- 6676/6676/2082/2082/0 0/0 POST /xxx/chat/status/updates
HTTP/1.1
Sep 13 12:42:45 localhost haproxy[15128]:
81.246.46.162:29456[13/Sep/2011:12:42:14.289] main
python_8001/python_8001_fe1 28/0/0/-1/31118
502 204 - - SH-- 6675/6675/2081/2081/0 0/0 POST /xxx/chat/status/updates
HTTP/1.1

504 errors:
Sep 13 12:43:08 localhost haproxy[15128]:
180.234.122.248:52888[13/Sep/2011:12:38:08.822] main
python_9004/python_9004_fe1 45/0/0/-1/300045
504 194 - - sH-- 6607/6607/697/697/0 0/0 POST /xxx/chat/message/4/updates
HTTP/1.1
Sep 13 12:43:09 localhost haproxy[15128]:
82.26.136.198:61758[13/Sep/2011:12:38:09.071] main
python_8001/python_8001_fe1 19/0/0/-1/300020
504 194 - - sH-- 6569/6569/2085/2085/0 0/0 POST /xxx/chat/status/updates
HTTP/1.1

It seems to me that all of these involve 0 seconds waiting in a queue, 0
seconds to make a connection to the final app server and then a aborted
connection to the app server before a complete response could be received
The total time in milliseconds between accept and last close seems to be
~300 seconds for most of the requests (although far from all of them, as the
first entry shows). If I *restart* (not reload) haproxy, I still get lines
with the fifth of these numbers (Tt in the docs) as ~300,000 (the timeout
server value in the config at the moment I copied the logs above), a few
seconds after the haproxy process starts. I also get lots that seem to end
on almost exactly 300k even when I change both timeout client and timeout
server to very different numbers. It is possible that the application
(jQuery) has a 300s timeout hardcoded, but in any case I do not understand
why the haproxy logs show connections with a connection of 300k failing when
I stop haproxy, increase timeout server by a order of magnitude and start
it again.

Looking at the next part of the log entries it seems that bytes_read is
always 204 for 504 errors and 194 for 504 errors. This does seem to be a
fairly regular pattern:
[root@frontend2 log]# cat /var/log/haproxy.log | grep  504 194 | wc -l
1975
[root@frontend2 log]# cat /var/log/haproxy.log | grep  502 204 | wc -l
12401
[root@frontend2 log]# cat /var/log/haproxy.log | wc -l
18721

 My second question is how do I find out exactly what is being returned
(easily), i.e. what are those 194/204 bytes? This might give me a hint as to
what is going wrong or timing out on the application server. I guess I could
try to tcpdump but I might struggle to actually filter down the correct data
(there are large numbers of successful connections going on)

The next part of the logs are most interesting; ignoring the two cookie
fields we see that the the server-side timeout expired for the 504 errors
and the TCP session was unexpectedly aborted by the server, or the server
explicitly refused it in the case of the 502. Subject to my questions above
I have a theory that the 504s are caused by the long-pooling application,
but I do not understand why in the case of the 502 haproxy is not retrying
the TCP connection before returning a 502 - I thought that the option
redispatch and retries 10 would ensure another go.

If anybody is able to shed some thoughts on my two questions I would be very
grateful!

Many thanks,

Alex

# haproxy.conf
global
log 127.0.0.1 local4 debug

chroot  /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 5
userhaproxy
group   haproxy
daemon

defaults
mode  http
optionhttplog
#option tcplog
optiondontlognull
optiondontlog-normal

log   global
retries   10
maxconn   5
timeout connect   20
contimeout20
clitimeout9
option forwardfor except 127.0.0.1/32 # Apache running https on
localhost
option httpclose # Required for REMOTE HEADER
option redispatch
timeout connect 1
timeout client  30
timeout server  30

frontend  main *:80
acl url_py_8001   path_beg   -i /url1
acl url_py_8002   path_beg   -i 

Re: haproxy / Python 'tornado' framework - digging into 502/504 errors

[Alex Davies]

Hi,

Thank you for your observation - indeed I did notice some of those as I was
writing my email - I have updated my globals to increase the server timeout
(as we are doing long polling) and reduce the others, and remove the
duplicates:

defaults
mode  http
optionhttplog
#option   tcplog
optiondontlognull
optiondontlog-normal

log   global
retries   10
maxconn   5
option forwardfor except 127.0.0.1/32 # Apache on https://127.0.0.1
option httpclose  # Required for REMOTE HEADER
option redispatch

timeout connect 1
timeout client  1
timeout server  720

I still notice the same errors in the logs! (slightly less 504, as I would
expect through the increase in timeout server - but I still don't
understand why I get any at all in the first minute of a new process).

Cheers,

Alex

On Tue, Sep 13, 2011 at 1:46 PM, Cyril Bonté cyril.bo...@free.fr wrote:

 clitimeout

haproxy / Python 'tornado' framework - digging into 502/504 errors

[Alex Davies]

Hi,

Thank you for your observation - indeed I did notice some of those as I was
writing my email - I have updated my globals to increase the server timeout
(as we are doing long polling) and reduce the others, and remove the
duplicates:

defaults
mode  http
optionhttplog
#option   tcplog
optiondontlognull
optiondontlog-normal

log   global
retries   10
maxconn   5
option forwardfor except 127.0.0.1/32 # Apache on https://127.0.0.1
 option httpclose  # Required for REMOTE HEADER
option redispatch

timeout connect 1
timeout client  1
timeout server  720

I still notice the same errors in the logs! (slightly less 504, as I would
expect through the increase in timeout server - but I still don't
understand why I get any at all in the first minute of a new process).

Cheers,

Alex

On Tue, Sep 13, 2011 at 1:46 PM, Cyril Bonté cyril.bo...@free.fr wrote:

 clitimeout