HAProxy SSL performance issue
Hi all, I am getting a big performance hit with SSL termination for small I/O, and errors when testing with bigger I/O sizes (ab version is 2.3): 1. Non-SSL vs SSL for small I/O (128 bytes): ab -k -n 100 -c 500 http://HAPROXY/128 RPS: 181763.65 vs 133611.69- 27% drop BW: 63546.28 vs 46711.90 - 27% drop 2. Non-SSL vs SSL for medium I/O (16 KB): ab -k -n 100 -c 500 http://HAPROXY/16K RPS: 62646.13vs 21876.33 (fails mostly with 70007 error as below) - 65% drop BW: 1016531.41 vs 354977.59 (fails mostly with 70007 error) - 65% drop 3. Non-SSL vs SSL for large I/O (128 KB): ab -k -n 10 -c 500 http://HAPROXY/128K RPS: 8476.99 vs apr_poll: The timeout specified has expired (70007) BW: 1086983.11 vs same error, this happens after 9 requests (always reproducible). --- HAProxy Build info - HA-Proxy version 1.5.12 2015/05/02 Copyright 2000-2015 Willy Tarreau w...@1wt.eu Build options : TARGET = linux2628 CPU = native CC = gcc CFLAGS = -O3 -march=native -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 USE_TFO=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.8 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1k 8 Jan 2015 Running on OpenSSL version : OpenSSL 1.0.1k 8 Jan 2015 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.35 2014-04-04 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. --- Config file - even cpu cores are on 1st socket on the mb, odd cpus are on 2nd global daemon maxconn 5 quiet nbproc 6 cpu-map 1 0 cpu-map 2 2 cpu-map 3 4 cpu-map 4 6 cpu-map 5 8 cpu-map 6 10 user haproxy group haproxy stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m tune.bufsize 32768 userlist stats-auth group adminusers admin user admininsecure-password admin defaults mode http maxconn 5 retries 3 option forwardfor option redispatch option prefer-last-server option splice-auto frontend www-http bind-process 1 2 3 bind *:80 stats uri /stats stats enable acl AUTH http_auth(stats-auth) acl AUTH_ADMIN http_auth(stats-auth) admin stats http-request auth unless AUTH default_backend www-backend frontend www-https bind-process 4 5 6 bind *:443 ssl crt /etc/ssl/private/haproxy.pem reqadd X-Forwarded-Proto:\ https default_backend www-backend-ssl backend www-backend bind-process 1 2 3 mode http balance roundrobin cookie FKSID prefix indirect nocache server nginx-1 172.20.232.122:80 maxconn 25000 check server nginx-2 172.20.232.125:80 maxconn 25000 check backend www-backend-ssl bind-process 4 5 6 mode http balance roundrobin cookie FKSID prefix indirect nocache server nginx-1 172.20.232.122:80 maxconn 25000 check server nginx-2 172.20.232.125:80 maxconn 25000 check --- CPU is E5-2670, 48 core system, nic interrupts are pinned to correct cpu's, etc. Can someone suggest what change is required to get better results as well as fix the 70007 error, or share their config settings? The stats are also captured. For 128 byte, all 3 haproxy's are running, but for 16K, and for 128K, only the last haproxy is being used (and seen consistently): -- MPSTAT and PIDSTAT - 128 byte, port 80 Average: CPU%usr %nice%sys %iowait%irq %soft %steal %guest %gnice %idle Average: 0 22.330.00 39.430.000.009.980.00 0.000.00 28.27 Average: 2 22.000.00 33.560.000.00 15.110.00 0.000.00 29.33 Average: 4 23.390.00 36.990.000.00 10.500.00 0.000.00 29.12 (First 3 haproxy's are used, last 3 are zero and not shown): Average: UID PID%usr %system %guest%CPU CPU Command Average: 110 5728 22.80 50.000.00 72.80 - haproxy Average: 110 5729 22.20 48.600.00 70.80 - haproxy Average: 110 5730 24.20 48.000.00 72.20 - haproxy 128 byte, port 443 Average: CPU%usr %nice%sys
[no subject]
DearSir/Madam, BelowLEDlightsaretop4sellingin=Europe.1,LEDHighBayLights30Wto200W.2,LEDstreetlight40W-400W3,LEDFloodlight10W-500W nb=sp; =nbsp; nbsp=; nb=sp; 4,LEDtubeandled=panellight9W-72W,dimmableisavailable. OurPartners:CREE,Sa=msungLED,Meanwelldriver,LiFuddriver. OurStrengths:Highqualityledlight-Yes,wedo!C=REEorBridgeluxLED,Meanwelldriver,bestheatsink!!5yearswarranty--Yes,weprovide5=yearswarrantyforourLEDfloodlight,ledstreetlight,ledhighbayl=ight.LowCosts-Yes!Wearethefactoryto=providetheledlampdirectly,thepriceisverynice!BestService-Yes!Replyyouwithin1=2hoursforanyquestion!Oursalespersonisveryprofessional,wellt=rained!FastDelievry-Yes!Forsampelsorder=within3days.Forthemassproduction,itwillneed3-7days,dependon=thequantity!Shippingwayandcheapcost-ByUPS,DH=L,Fedex,byairorbysea(forlargeorder).Fastandgooddiscountof=thefreight! Formoreinfo,pleasecontactsales@taipl=ed.com KindRegards,Anna-=-ShenzhenTaipOptronicsCO.,LTDhttp://ta=ipled.en.made-in-china.com/Ema=il:sa...@taipled.com
Re: 1.4 - 1.5 migration resulted in degraded performance
Hi Pawel, On Tue, May 19, 2015 at 02:47:41PM -0700, Pawel Veselov wrote: This settings should theoretically make haproxy behave exactly the same. So think that somehow, 1.5 was creating or keeping a lot more open connections at a time, and depriving the kernel, or its own limits of available connections? There's no reason, you should definitely have the same number of connections with both versions. Also, even on small machines the number of connections a single process will sustain when reaching servers will be much lower than the kernel's default FD limits due to the limited number of source ports. For example, here on a 2GB-equipped atom, the limit is already at 204k fd, which is much larger than the 2*64k you'd have from a single source address, and I'm expecting that your production LB is larger than this box : $ cat /proc/sys/fs/file-max 204282 In its absence, the behavior would indeed have changed, from the old default: option http-tunnel [1] to: option http-keep-alive [2] One thing I can suggest is to replace option httpclose in the default section with the following configuration: option http-keep-alive option prefer-last-server timeout http-keep-alive 5s (or whatever keep alive timeout works for you) I personally don't see any reason for having any connections left kept alive. We want to minimize the amount of open files, we've had problems with keep-alive choking us quite quickly. OK that's fair enough. Then you should use http-server-close. httpclose is a hack from version 1.1 which consists in simply adding connection: close to the headers and to let each side decide to close. In practice it used to result in a number of servers to keep the connection alive despite this, forcing us to implement extra hacks like forceclose. Since 1.4 this connection management is much cleaner as haproxy can actively close a connection, while it was only watching it previously. I see the doc says that httpclose is ...is deprecated since what it does is very cheap but not reliable, which I guess means that it won't work with all the servers, but our servers honor the Connection:close header quite fine, so I don't see a reason to change that. Perfect, then just use http-server-close, even on 1.4 since you have it. I guess what I should do - is try 1.5 during quiet time, and compare the environment (open fds, etc) with 1.4, and see what is different... Or maybe first test http-server-close on 1.4 to see if there's anything related to this. It could very well be that something on the network doesn't always see the closed connections. Regards, Willy
Re: HAProxy SSL performance issue
Hi, On Thu, May 21, 2015 at 11:31:52AM +0530, Krishna Kumar (Engineering) wrote: Hi all, I am getting a big performance hit with SSL termination for small I/O, and errors when testing with bigger I/O sizes (ab version is 2.3): 1. Non-SSL vs SSL for small I/O (128 bytes): ab -k -n 100 -c 500 http://HAPROXY/128 RPS: 181763.65 vs 133611.69- 27% drop BW: 63546.28 vs 46711.90 - 27% drop That's expected and it matches my measurements. I've found up to about 30% drop on keep-alive request rate between clear and SSL, which is much better than I'd have expected. 2. Non-SSL vs SSL for medium I/O (16 KB): ab -k -n 100 -c 500 http://HAPROXY/16K RPS: 62646.13vs 21876.33 (fails mostly with 70007 error as below) - 65% drop BW: 1016531.41 vs 354977.59 (fails mostly with 70007 error) - 65% drop I suspect the BW unit is bytes per second above though I could be wrong. It matches what you can expect from bidirectionnal TCP streams over a single port. Thus you're at about 8 Gbps of payload bitrate in each direction but in practice the link is close to 10 due to packet headers and acks flying back. But that means about 2.8 Gbps of SSL traffic. Here it will depend on the algorithm and CPU frequengy of course. I remember seeing about 4.6 Gbps on a single core at 3.5 GHz using AES-NI. 3. Non-SSL vs SSL for large I/O (128 KB): ab -k -n 10 -c 500 http://HAPROXY/128K RPS: 8476.99 vs apr_poll: The timeout specified has expired (70007) BW: 1086983.11 vs same error, this happens after 9 requests (always reproducible). Hmmm, would you be running from multiple load generators connected via a switch with small buffers ? I've already experienced a similar situation caused by the inability of the switch to buffer all the data sent by haproxy and distribute it fairly to all clients. You can have a similar issue if injecting over a same port on a saturated link and/or switch, and return ACKs are getting lost because of the full link. I'm thinking about something else, could you retry with less or more total objects in the /128 case (or the 16k case) ? The thing is that ab starts all the connections at the same time, leading to parallel key generation, and will then keep them all during the rest of the test. On a reasonable machine, this will not cost more than one second, but it would be interesting to know the part caused by the negociation and the rest. --- HAProxy Build info - HA-Proxy version 1.5.12 2015/05/02 Copyright 2000-2015 Willy Tarreau w...@1wt.eu Build options : TARGET = linux2628 CPU = native CC = gcc CFLAGS = -O3 -march=native -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 USE_TFO=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.8 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1k 8 Jan 2015 Running on OpenSSL version : OpenSSL 1.0.1k 8 Jan 2015 You may want to try openssl-1.0.2a which significantly improved performance in several use cases. OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.35 2014-04-04 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. --- Config file - even cpu cores are on 1st socket on the mb, odd cpus are on 2nd global daemon maxconn 5 quiet nbproc 6 cpu-map 1 0 cpu-map 2 2 cpu-map 3 4 cpu-map 4 6 cpu-map 5 8 cpu-map 6 10 user haproxy group haproxy stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m tune.bufsize 32768 userlist stats-auth group adminusers admin user admininsecure-password admin defaults mode http maxconn 5 retries 3 option forwardfor option redispatch option prefer-last-server option splice-auto frontend www-http bind-process 1 2 3 bind *:80 Here you have an imbalance at the kernel level : the same socket is bound to several processes, so any process who picks a connection will process it, and the initial rush of key generations can make this imbalance quite visible. You should do this instead to have 3 distinct sockets each in its own process (but warning, this requires a kernel = 3.9) : bind *:80 process 1 bind *:80 process 2 bind *:80 process 3 stats
[SPAM] Inscription gratuite pour bénéficier jusqu'à 80 % de remise
Title: BricoPriv Pour visualiser correctement ce message, accédez à la version en ligne Ventes Prives Bricolage & Jardinage : remises jusqu -80% ! Bricoprive.com est le site de ventes prives ddies au bricolage et au jardinage. Tous les jours de nouvelles ventes. Profitez de remises exceptionnelles allant jusqu -80%. Pour accder aux ventes prives inscrivez-vous... cest simple et cest gratuit ! Pour un trs grand nombre de franaises et de franais, le bricolage et le jardinage sont des passions, parfois une ncessit et toujours un plaisir ! Mais il ny a pas de bons artisans sans bons outils En crant BRICOPRIVE, nous avons souhait permettre aux bricoleurs confirms comme occasionnels, aux amateurs de jardinage et damnagement extrieur, davoir accs du matriel de grande marque et de qualit professionnelle moindre cot. Que vous soyez passionn, amateur clair, en plein chantier ou que vous souhaitiez simplement vous quiper pour amliorer votre intrieur et votre extrieur, rejoignez BRICOPRIVE et invitez vos amis ! Pour cesser de recevoir nos informations sur l'adresse haproxy@formilux.org , désabonnez-vous.
Re: 1.4 - 1.5 migration resulted in degraded performance
Wiilly, Lucas, thank you so much for analyzing my configs and your help. We did find out what was wrong. Some long time ago we added 'option nolinger' to the defaults section. This was figured by trial and error, and that option, on 1.4, served us well to the point of us forgetting about it. When were trying out 1.5, we started experiencing phantom RSTs that, of course, were happening because of nolinger. At which point we said nolinger is evil and should not be used, and took it out all together. The problems that we were experiencing after version update didn't remind us of the time where we had to put that nolinger in to combat exactly the same symptoms. We rolled the haproxy version back, and of course it didn't make things any better until we put the nolinger back in. I think that in 1.4, nolinger in defaults section only affected backend side, and in 1.5 it started applying to both. I honestly don't quite understand what's wrong with our set up except that: - it's not the application per se, because the application processes the same volume of the requests just fine (when nolinger is turned on) - it's either the network framework (J2EE) settings or kernel settings on the application side that are to fault The failure comes when the session rate reaches some critical volume. At that point we see sharp rise of concurrent sessions (sockets in ESTAB state rise the same), then sharp rise of sockets in SYN-SENT state, and then the whole TCP stack stalls (no impact on existing TCP connections though, like any open SSH shells are not affected to the slightest). There aren't really any problems with unfinished sockets (high number of sockets are only in ESTAB, SYN-SENT and TIME-WAIT states), there is a moderate (few thousand) amount of sockets in client-side TIME_WAIT, but nowhere close to numbers that would cause a problem. So, the stall really occurs on the network side of the application, in turn causing haproxy to stall. I'll see if I can eventually create steps to reproduce that stall in a simple environment, at least to figure out if its haproxy queue that gets stalled, or the whole TCP stack of the machine where haproxy is running, but I don't understand how that scenario would cause the haproxy machine to stop promptly responding to incoming SYNs. I don't think the problem is maxing out maxconn, because our connect timeout is 3s (no explicit queue timeout), and there isn't even a 503 response, and attempting to connect through haproxy during the stall times out way longer than 3s. Thank you, Pawel. On Thu, May 21, 2015 at 5:04 AM, Willy Tarreau w...@1wt.eu wrote: Hi Pawel, On Tue, May 19, 2015 at 02:47:41PM -0700, Pawel Veselov wrote: This settings should theoretically make haproxy behave exactly the same. So think that somehow, 1.5 was creating or keeping a lot more open connections at a time, and depriving the kernel, or its own limits of available connections? There's no reason, you should definitely have the same number of connections with both versions. Also, even on small machines the number of connections a single process will sustain when reaching servers will be much lower than the kernel's default FD limits due to the limited number of source ports. For example, here on a 2GB-equipped atom, the limit is already at 204k fd, which is much larger than the 2*64k you'd have from a single source address, and I'm expecting that your production LB is larger than this box : $ cat /proc/sys/fs/file-max 204282 In its absence, the behavior would indeed have changed, from the old default: option http-tunnel [1] to: option http-keep-alive [2] One thing I can suggest is to replace option httpclose in the default section with the following configuration: option http-keep-alive option prefer-last-server timeout http-keep-alive 5s (or whatever keep alive timeout works for you) I personally don't see any reason for having any connections left kept alive. We want to minimize the amount of open files, we've had problems with keep-alive choking us quite quickly. OK that's fair enough. Then you should use http-server-close. httpclose is a hack from version 1.1 which consists in simply adding connection: close to the headers and to let each side decide to close. In practice it used to result in a number of servers to keep the connection alive despite this, forcing us to implement extra hacks like forceclose. Since 1.4 this connection management is much cleaner as haproxy can actively close a connection, while it was only watching it previously. I see the doc says that httpclose is ...is deprecated since what it does is very cheap but not reliable, which I guess means that it won't work with all the servers, but our servers honor the Connection:close header quite fine, so I don't see a reason to change that. Perfect, then just use http-server-close, even on 1.4 since you have it. I guess what I
SSL custom dhparam problem
Hello, I encounter a problem with dhparam configuration, if i have 2 bind lines, a tune.ssl.default-dh-param 2048, and a custom group dhparam in one of the pem file, ALL bind lines will use 1024, the one with the custom group will work as expected, and the one without will use the default Oakley group 2 instead of the 2048-bit MODP group 14 (thx Remi for the wording, i'm not sure to well understand all of that :)) here is a test config which will fail for 1.1.1.2:443 : global ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA ssl-default-bind-options no-sslv3 tune.ssl.default-dh-param 2048 tune.ssl.maxrecord 1419 tune.ssl.cachesize 5 tune.ssl.lifetime 600 frontend foo bind 1.1.1.1:443 ssl crt certs_with_static_1024_dhparam.pem bind 1.1.1.2:443 ssl crt cert_without_static_dhparam.pem this is clearly a bug amha, thx anyone who can help (Remi ? :) ) Hervé C.
Re: SSL custom dhparam problem
Hi Hervé,
On 05/21/2015 10:11 PM, Hervé Commowick wrote:
I encounter a problem with dhparam configuration, if i have 2 bind lines, a
tune.ssl.default-dh-param 2048, and a custom group dhparam in one of the
pem file, ALL bind lines will use 1024, the one with the custom group will
work as expected, and the one without will use the default Oakley group 2
instead of the 2048-bit MODP group 14 (thx Remi for the wording, i'm not
sure to well understand all of that :))
this is clearly a bug amha, thx anyone who can help (Remi ? :) )
Oh, this is a bug indeed, and it's my fault. In order to prevent the
display of warning messages about default-dh-param not being set when a
static DH group value is used, the value of default-dh-param is
overridden when a static DH group value is found. It does work when you
have only one bind, but it's clearly wrong when more than one is used,
like in your configuration.
Could you try with the attached patch? It's a patch against the 1.6
trunk but it does apply cleanly against 1.5.12.
It will result in false positive messages about default-dh-param not
being set when it's not needed, but to prevent that we will need to
check if each bind has a static DH group value, which I'm not very fond of.
--
Rémi
diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 9302869..5317a28 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -1347,10 +1347,6 @@ int ssl_sock_load_dh_params(SSL_CTX *ctx, const char *file)
if (dh) {
ret = 1;
SSL_CTX_set_tmp_dh(ctx, dh);
- /* Setting ssl default dh param to the size of the static DH params
- found in the file. This way we know that there is no use
- complaining later about ssl-default-dh-param not being set. */
- global.tune.ssl_default_dh_param = DH_size(dh) * 8;
}
else {
/* Clear openssl global errors stack */
signature.asc
Description: OpenPGP digital signature
Custom SSL DHparams prime
Hi, from what I've seen in the sources and documentation a default and pre-generated prime will be used as default (unless appended to the certificate). HAProxy uses the related functions provided by OpenSSL itself (get_rfc3526_prime_2048, ...). What I miss here is an option to specify my own dhparams file to avoid using those pre-generated ones and/ore appending some to all certificates. Wouldn't it make sense to allow it to be read from a file, globally? -- Regards, Christian Ruppert
Re: Need help with HAProxy configuration mixed content http and https
Thanks Baptise, Let me give this a try. On May 21, 2015, at 5:26 AM, Baptiste bed...@gmail.com wrote: it seems your client get connected using HTTPs on the HTTP port of haproxy. you must make your application aware that SSL offloading is being performed by a device in front of it. Some hints: http://blog.haproxy.com/2013/02/26/ssl-offloading-impact-on-web-applications/ http://blog.haproxy.com/2013/02/26/ssl-offloading-impact-on-web-applications/ Baptiste On Wed, May 20, 2015 at 9:41 AM, Tu Nguyen nguyenhongtu...@gmail.com mailto:nguyenhongtu...@gmail.com wrote: Hi Baptiste, This is all my configuration. be_game5_http is be_host01_http, I changed it because of sensitive reason. I’m sorry if it made you confused! Here is haproxy log: # this log generated when I visit host01.vn http://host01.vn/ May 20 14:21:36 localhost haproxy[22380]: 123.30.173.99:41879 http://123.30.173.99:41879/ [20/May/2015:14:21:36.235] fe_http be_host01_http/HOST_9 0/0/0/6/7 200 12764 - - 1/1/0/1/0 0/0 GET / HTTP/1.1 May 20 14:21:36 localhost haproxy[22380]: 123.30.173.99:41879 http://123.30.173.99:41879/ [20/May/2015:14:21:36.242] fe_http be_host01_http/HOST_10 290/0/0/1/291 304 348 - - 1/1/0/1/0 0/0 GET /static/css/fancybox/jquery.fancybox-1.3.4.css HTTP/1.1 May 20 14:21:36 localhost haproxy[22380]: 123.30.173.99:41879 http://123.30.173.99:41879/ [20/May/2015:14:21:36.534] fe_http be_host01_http/HOST_9 39/0/0/1/40 304 349 - - 2/2/0/1/0 0/0 GET /static/js/default/jquery-1.8.3.min.js HTTP/1.1 May 20 14:21:36 localhost haproxy[22380]: 123.30.173.99:41879 http://123.30.173.99:41879/ [20/May/2015:14:21:36.574] fe_http be_host01_http/HOST_10 82/0/1/0/83 304 348 - - 2/2/0/1/0 0/0 GET /static/js/fancybox/jquery.fancybox-1.3.4.min.js HTTP/1.1 # this log generated when I visit https://host01.vn https://host01.vn/ May 20 14:21:56 localhost haproxy[22380]: 123.30.173.99:32217 http://123.30.173.99:32217/ [20/May/2015:14:21:36.568] fe_http fe_http/NOSRV -1/-1/-1/-1/19768 400 0 - - CR-- 1/1/0/0/0 0/0 BADREQ There are some images when I visit host01.vn http://host01.vn/, with tcp mode, it works: qq2015052...@2x.png With mode http, it doesn’t work: qq2015052...@2x.png Thanks and Best regards, Tu Nguyen, On May 20, 2015, at 1:56 PM, Baptiste bed...@gmail.com mailto:bed...@gmail.com wrote: On Wed, May 20, 2015 at 6:49 AM, Tu Nguyen nguyenhongtu...@gmail.com mailto:nguyenhongtu...@gmail.com wrote: Hello, I’m new to HAProxy. I’m trying to set up HA to handle mixed content site (http and https). My site runs on http except login box which needs https. When I apply the configure with mode tcp, everything is fine. But I cannot modify header, which I need to identify user’s real IP. So I want to use mode http and when I do that, the content of login box does not appear ( all css, static files cannot be loaded) Could you please tell me if anyway to fix this problem or HAProxy does not support this yet? This is my HAProxy config, below: ## # HAPROXY 1.5.12 ## global daemon user haproxy group haproxy chroot /home/haproxy maxconn 1 stats socket /tmp/haproxy pidfile /var/run/haproxy.pid log 127.0.0.1 local1 tune.ssl.default-dh-param 2048 defaults #mode http retries 3 option redispatch timeout connect 5ms timeout client 5ms timeout server 5ms errorfile 400 /prefix/haproxy/errors/400.http errorfile 403 /prefix/haproxy/errors/403.http errorfile 408 /dev/null errorfile 500 /prefix/haproxy/errors/500.http errorfile 502 /prefix/haproxy/errors/502.http errorfile 503 /prefix/haproxy/errors/503.http errorfile 504 /prefix/haproxy/errors/504.http FRONTEND frontend fe_https bind *:443 ssl crt /prefix/haproxy/ca/domain01.pem mode http option httpclose option forwardfor acl ssl_host01.vn http://ssl_host01.vn/ hdr(host) -i host01.vn http://host01.vn/ www.host01.vn http://www.host01.vn/ use_backend be_host01_https if ssl_host01.vn http://ssl_host01.vn/ frontend fe_http bind *:80 log global mode http option httplog option forwardfor acl host01.vn http://host01.vn/ hdr(host) -i host01.vn http://host01.vn/ www.host01.vn http://www.host01.vn/ use_backend be_host01_http if host01.vn http://host01.vn/ BACKEND backend be_host01_https mode http option httplog option forwardfor log global balance roundrobin server SSL_HOST_1 10.0.0.1:80 http://10.0.0.1/ check server SSL_HOST_2 10.0.0.2:80 http://10.0.0.2/ check backend be_host01_http mode http log global balance roundrobin server HOST_9 10.0.0.1:80 http://10.0.0.1/ check server HOST_10 10.0.0.2:80 http://10.0.0.2/ check ## END Thanks in advanced, Tu Nguyen, Hi Tu, Could you share also logs generated by HAProxy when trying to acess login page? Also, your HTTP frontend points to a backend whose name is be_game5_http. Could you confirm this is a
Re: SSL custom dhparam problem
Hi Rémi, On Thu, May 21, 2015 at 11:19:15PM +0200, Remi Gacogne wrote: Hi Hervé, On 05/21/2015 10:11 PM, Hervé Commowick wrote: I encounter a problem with dhparam configuration, if i have 2 bind lines, a tune.ssl.default-dh-param 2048, and a custom group dhparam in one of the pem file, ALL bind lines will use 1024, the one with the custom group will work as expected, and the one without will use the default Oakley group 2 instead of the 2048-bit MODP group 14 (thx Remi for the wording, i'm not sure to well understand all of that :)) this is clearly a bug amha, thx anyone who can help (Remi ? :) ) Oh, this is a bug indeed, and it's my fault. In order to prevent the display of warning messages about default-dh-param not being set when a static DH group value is used, the value of default-dh-param is overridden when a static DH group value is found. It does work when you have only one bind, but it's clearly wrong when more than one is used, like in your configuration. Could you try with the attached patch? It's a patch against the 1.6 trunk but it does apply cleanly against 1.5.12. It will result in false positive messages about default-dh-param not being set when it's not needed, but to prevent that we will need to check if each bind has a static DH group value, which I'm not very fond of. I think we could proceed differently then : if no global dhparam is set when parsing the bind line *and* no dhparam is set in the file, then we should set it and emit the warning at this moment. Or alternatively, we could even just set the global param from the bind line when it was not set. Willy
Re: Reducing HAProxy System Time
Hi Robert, On Tue, May 19, 2015 at 04:10:54PM -0700, Robert Brooks wrote: On Mon, May 18, 2015 at 7:58 PM, Willy Tarreau w...@1wt.eu wrote: It's useless at such sizes. A rule of thumb is that splicing will not be used at all for anything that completely fits in a buffer since haproxy tries to read a whole response at once and needs to parse the HTTP headers anyway. In general, the cost of the splice() system calls compared to copying data sees a break-even around 4-16kB depending on arch, memory speed, setup and many factors. I find splice very useful on moderately large objects (64kB) at high bit rates (10-60 Gbps). On most gigabit NICs, it's inefficient as most such NICs have limited capabilities which make them less efficient at performing multiple DMA accesses for a single packet (eg: no hardware scatter-gather to enable GSO/TSO for example). I had considered doing tcp proxying at the top level haproxy such that this workload is passed down to individual hosts and then maybe splice would be effective, but I am not sure it is enough of a win. Managing http as such seems to be more correct and gettting http stats is useful too. Yes and haproxy is faster in HTTP than in TCP. While it can sound surprizing, it's because there are states in HTTP which allow haproxy to know what events to care about (eg: poll for reads only at certain moments, etc) and when small packets may be aggregated. Regardless of this, splicing is really useless for small packets. We are running Intel I350 Gigabit NICs which appear support most tcp offloading except LRO. OK, I have one such as well. They're quite good indeed, but don't expect to gain anything by using splicing. Running at gig speed is very cheap even with recv/send. With 1500-byte packets, that's only 8 packets per second. The atom in my NFS server can saturate its link with about half a core. When it comes to doing it with small HTTP responses, it will clearly not saturate the gig link however. We previously had Broadcom interfaces, which again had good TCP offload, but appear to have interrupt coalescing issues. Are you still using Myricom cards? I still have them in my home lab and use them from time to time, yes. These cards are very interesting to test your software because they combine very low latency with very little CPU usage in the driver. So you can reach 10Gbps of forwarding rate with only 25% of one core on an old core-2 duo, and you're never concerned with ksoftirqd triggering during your tests. However I found that I was facing some packet rate limitations with them, meaning it's not possible to reach 10G in TCP with packets smaller than 1500 bytes and their respective ACKs, which is about 800kpps in each direction, so in our appliances we have switched to intel 82599 NICs whose driver is much heavier, but which can saturate 10G at any packet size. Any thoughts if these would be a win with our workload? Our data rates are relatively small, it's all about request rates. I know at least one site who managed to significantly increase their request rate by switching from gig NICs to myricom for small requests. If you look there : http://www.haproxy.org/10g.html You'll see in the old benchmark (2009) that at 2kB objects we were at about 33k req/s on a single process on the core-2 using haproxy 1.3. On recent hardware, intel NICs can go higher than this because you easily have more CPU power to dedicate to the driver. Reaching 60-100k is not uncommon with fine tuning on such small request sizes. Nothing apparently wrong here, though the load could look a bit high for that traffic (a single-core Atom N270 can do that at 100% CPU). You should run a benchmark to find your system's limits. The CPU load in general is not linear since most operations will be aggregated when the connection rate increases. Still you're running at approx 50% CPU and a lot of softirq, so I suspect that conntrack is enabled on the system with default settings and not tuned for performance, which may explain why the numbers look a bit high. But nothing to worry about. Conntrack is not used or loaded, however, what I believe is contributing to higher than expected system load is non keep-alive traffic. We also serve a lot of http 302s, about 1000 reqs/sec, keep-alive is disabled as these are single requests from individual clients. 1000 http 302/s is almost idle. You should barely notice haproxy in top at such a rate. Here's what I'm seeing on my core2quad @3 GHz at exactly 1000 connections per second : Tasks: 178 total, 1 running, 175 sleeping, 2 stopped, 0 zombie Cpu(s): 0.2%us, 1.0%sy, 0.2%ni, 98.0%id, 0.0%wa, 0.0%hi, 0.5%si, 0.0%st Mem: 8168416k total, 7927908k used, 240508k free, 479316k buffers Swap: 8393956k total,18148k used, 8375808k free, 4886752k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 9253 willy 20 0 4236 512
Re: [PATCH] MEDIUM: backend: Allow redispatch on retry intervals
Hi Joseph, On Thu, May 21, 2015 at 10:50:17AM -0700, Joseph Lynch wrote: Hello Willy, On Sat, May 16, 2015 at 2:05 AM, Willy Tarreau w...@1wt.eu wrote: I moved the order of the comparisons around a little bit to ensure that the redispatch_after variable would be defined (namely if PR_O_REDISP is set then redispatch_after must have been set to -1). I think that we'll be able to get rid of the option after that, as a cleanup resulting from your work. I believe in this iteration of the patch I have initialized the redispatch_after variable so that this cleanup is easier in the future. Perfect, I've applied it, thank you. It should just be an inversion of the check but against redispatch_after. If you like I can follow up this patch with that cleanup. As you like! Thanks! Willy
Re: 1.4 - 1.5 migration resulted in degraded performance
Hi Pawel, On Thu, May 21, 2015 at 01:04:42PM -0700, Pawel Veselov wrote: Wiilly, Lucas, thank you so much for analyzing my configs and your help. We did find out what was wrong. Some long time ago we added 'option nolinger' to the defaults section. This was figured by trial and error, and that option, on 1.4, served us well to the point of us forgetting about it. It should not be used in defaults since it will affect the frontend and will result in truncated responses for far remote clients due to resets being sent for unacked data. When were trying out 1.5, we started experiencing phantom RSTs that, of course, were happening because of nolinger. At which point we said nolinger is evil and should not be used, and took it out all together. exactly! The problems that we were experiencing after version update didn't remind us of the time where we had to put that nolinger in to combat exactly the same symptoms. We rolled the haproxy version back, and of course it didn't make things any better until we put the nolinger back in. I think that in 1.4, nolinger in defaults section only affected backend side, and in 1.5 it started applying to both. I don't remember the possibility that it didn't affect the frontend in 1.4 since it was initially introduced for the frontend to work around one client bug. However, it would be possible that there was a bug in 1.4 causing it to be ineffective. Or maybe just that it was used *after* the close response from the client, making it useless. I honestly don't quite understand what's wrong with our set up except that: - it's not the application per se, because the application processes the same volume of the requests just fine (when nolinger is turned on) - it's either the network framework (J2EE) settings or kernel settings on the application side that are to fault I think you're in a situation where the application doesn't close and lets the client close first. When using TCP, a client must never ever close first because it leaves TIME_WAIT on it and it cannot reuse source ports for 1-2 minutes depending on the OS. That would explain why you needed nolinger since it forces a TCP RST to the server as well. And given you were running with option httpclose which is the passive mode, haproxy didn't interact in the connection close. Thus if you used option http-server-close or even option forceclose, haproxy would actively close the connection itself and use RST to the server but a clean FIN to the client. The failure comes when the session rate reaches some critical volume. Yes, and I can even give you the session rate : #source ports / TW timeout. If you have the default 28000 ports and 60s timeouts, that's a bit less than 500 sessions per second before all ports are in TIME_WAIT and cannot be reused. At that point we see sharp rise of concurrent sessions (sockets in ESTAB state rise the same), then sharp rise of sockets in SYN-SENT state, If you see SYN-SENT on the client, it can mean you have a firewall between haproxy and the server which prevents connections from establishing using the same source ports. Normally we don't find such bogus firewalls anymore. However, a few types of firewalls do wrongly randomize source ports and/or sequence numbers and break TCP because of this, because when the transformed SYN reaches the server, it doesn't match the previous session and cannot recycle it. One firewall was known for this, the Cisco FWSM. However, having time-stamps enabled on both haproxy and the server worked around it. Another solution consisted in fixing the firewall using no normalization to disable the bogus behaviour. and then the whole TCP stack stalls (no impact on existing TCP connections though, like any open SSH shells are not affected to the slightest). There aren't really any problems with unfinished sockets (high number of sockets are only in ESTAB, SYN-SENT and TIME-WAIT states), there is a moderate (few thousand) amount of sockets in client-side TIME_WAIT, but nowhere close to numbers that would cause a problem. Except if they consume all source ports :-) Where do you see them, between client and haproxy (normal and harmless) or between haproxy and server ? So, the stall really occurs on the network side of the application, in turn causing haproxy to stall. I'll see if I can eventually create steps to reproduce that stall in a simple environment, at least to figure out if its haproxy queue that gets stalled, or the whole TCP stack of the machine where haproxy is running, but I don't understand how that scenario would cause the haproxy machine to stop promptly responding to incoming SYNs. I don't think the problem is maxing out maxconn, because our connect timeout is 3s (no explicit queue timeout), and there isn't even a 503 response, and attempting to connect through haproxy during the stall times out way longer than 3s. The fact that you see the SYN leave the machine makes me think that the
Re: A few thoughts on Haproxy and weakdh/logjam
Hi Remi, On Thu, May 21, 2015 at 06:07:34PM +0200, Remi Gacogne wrote: In the default configuration, Haproxy uses a 1024-bit DH key generated from the second Oakley group [2] for Diffie-Hellman Ephemeral (DHE) key exchange. This group is widely used, and is likely to be the first target for pre-computation by an adversary with large enough computing capabilities. I would advise using instead a 2048-bit key generated from the MODP group 14, by setting the tune.ssl.default-dh-param parameter to 2048, or even disabling DHE altogether if you are expecting every client to support ECDHE key exchange. Note that increasing the tune.ssl.default-dh-param will increase the CPU load on your server, and may therefore increase the connection establishment latency. That makes me think about something, as you advocated a long time ago for increasing the dh-param default size. Do you think we should take the opportunity of 1.6 to increase the default size ? It will use more CPUs for people who migrate from 1.5 only, and such people are expected to run tests during the migration anyway so they should not be surprized. If you cannot increase the DH key size above 1024-bit, please at least generate a custom DH group with the openssl dhparam 2048 command, and add the result of this command to your certificate file. Does that improve the situation regarding the CPU usage ? I must confess this is still very cryptic to me (no pun intended). Thanks! Willy
A few thoughts on Haproxy and weakdh/logjam
Haproxy and weakdh/logjam Hi, Everyone has probably heard about the recently disclosed weakdh/logjam attack [0] already. Here are a few personal thoughts on the impact on Haproxy. The weakdh issue is twofold: - if the HTTPS server is willing to accept a cipher suite using a very weak Diffie-Hellman (DH) group, like for example TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, then it is possible for an adversary in position of man-in-the-middle (MitM) to downgrade the security level of a given connection to this cipher suite, thus reducing the security to 512-bit DH ; - a nation-state adversary may be able to pre-compute all the possible values of a commonly-shared 1024-bit DH group, thus being able to decrypt all exchanges using that group. The first point depends on the cipher suite specified by the administrator with the ssl-default-bind-ciphers or ciphers parameters. I strongly encourage everyone to use the modern cipher suite described on the Mozilla wiki [1]. Please at least consider using the Intermediate one. In the default configuration, Haproxy uses a 1024-bit DH key generated from the second Oakley group [2] for Diffie-Hellman Ephemeral (DHE) key exchange. This group is widely used, and is likely to be the first target for pre-computation by an adversary with large enough computing capabilities. I would advise using instead a 2048-bit key generated from the MODP group 14, by setting the tune.ssl.default-dh-param parameter to 2048, or even disabling DHE altogether if you are expecting every client to support ECDHE key exchange. Note that increasing the tune.ssl.default-dh-param will increase the CPU load on your server, and may therefore increase the connection establishment latency. If you cannot increase the DH key size above 1024-bit, please at least generate a custom DH group with the openssl dhparam 2048 command, and add the result of this command to your certificate file. Best regards, [0]: https://weakdh.org/ [1]: https://wiki.mozilla.org/Security/Server_Side_TLS [2]: https://tools.ietf.org/html/rfc2409#section-6.2 -- Remi signature.asc Description: OpenPGP digital signature
Re: Custom SSL DHparams prime
On 2015-05-21 18:20, Remi Gacogne wrote: Hi, from what I've seen in the sources and documentation a default and pre-generated prime will be used as default (unless appended to the certificate). HAProxy uses the related functions provided by OpenSSL itself (get_rfc3526_prime_2048, ...). What I miss here is an option to specify my own dhparams file to avoid using those pre-generated ones and/ore appending some to all certificates. Wouldn't it make sense to allow it to be read from a file, globally? I don't think the 2048-bit MODP group 14 used by Haproxy is at risk right now, still it can't hurt to use a large number of different groups. You can use your own dhparam by appending it to the file specified with the crt command, after your certificate chain and key. Well, I meant globally, as default. global tune.ssl.default-dh-param /path/to/custom/dhparams.pem 2048 was just an example. There is 1024 and IIRC 768 as well. One might be forced to use 1024. Also, according to the documentation HAProxy wouldn't allow/use anything greater than tune.ssl.default-dh-param which is 1024 by default - unless I misunderstood something. -- Regards, Christian Ruppert
Re: Custom SSL DHparams prime
Hi, from what I've seen in the sources and documentation a default and pre-generated prime will be used as default (unless appended to the certificate). HAProxy uses the related functions provided by OpenSSL itself (get_rfc3526_prime_2048, ...). What I miss here is an option to specify my own dhparams file to avoid using those pre-generated ones and/ore appending some to all certificates. Wouldn't it make sense to allow it to be read from a file, globally? I don't think the 2048-bit MODP group 14 used by Haproxy is at risk right now, still it can't hurt to use a large number of different groups. You can use your own dhparam by appending it to the file specified with the crt command, after your certificate chain and key. -- Rémi signature.asc Description: OpenPGP digital signature
Re: Custom SSL DHparams prime
You can use your own dhparam by appending it to the file specified with the crt command, after your certificate chain and key. Well, I meant globally, as default. global tune.ssl.default-dh-param /path/to/custom/dhparams.pem I don't think it's possible right now, but it should not be too hard to add this feature. 2048 was just an example. There is 1024 and IIRC 768 as well. One might be forced to use 1024. Also, according to the documentation HAProxy wouldn't allow/use anything greater than tune.ssl.default-dh-param which is 1024 by default - unless I misunderstood something. If you add a custom group to your certificate file, it will override the default-dh-param configuration. The default-dh-param has been added to provide a default group when no custom one has been provided. Maybe the documentation is not very clear on this point (sorry about that). - No custom group in the certificate file, tune.ssl.default-dh-param not specified or set to 1024 = The default Oakley group 2 (1024) is used - No custom group in the certificate file, tune.ssl.default-dh-param is set to 2048 = the 2048-bit MODP group 14 will be used, except if the certificate has a RSA key smaller than 2048-bit, then Oakley group 2 (1024) is used - No custom group in the certificate file, tune.ssl.default-dh-param is set to 4096 = the 4096-bit MODP group 16 will be used, except if the certificate has a RSA key smaller than 4096-bit but larger or equal to 2048, then 2048-bit MODP group 14 is used. If the key is smaller than 2048, then Oakley group 2 (1024) is used. - No custom group in the certificate file, tune.ssl.default-dh-param is set to 8192 = the 8192-bit MODP group 18 will be used, with the same exceptions than previously. - A custom group is set in the certificate file, no matter what the tune.ssl.default-dh-param is set to, no matter the size of the RSA key in your certificate, no matter the size of the custom group, this custom group is used. -- Rémi signature.asc Description: OpenPGP digital signature
