Re: Tuning HAProxy for Production
Hi Jordan, If you don't provide us any information on the type of services you're going to deliver with HAProxy, then we can't help you! Baptiste On Thu, Jan 2, 2014 at 4:25 AM, Jordan Arentsen jor...@bliss.io wrote: I'm trying to prepare HAProxy for a production, and I'm trying to figure out some good default configuration settings that will at least give me a good place to start. My main question revolves around the maxconn option and the various timeouts. I was thinking about setting the maxconn to 15k or so, is this a bad place to start? Any other advice on baseline performance tuning? My main HAProxy instance will be running on an EC2 c3.large. Thanks!
Bye, bye
We are sorry that you decided to opt-out. We confirm that this email account haproxy@formilux.org has un-subscribed.
Re: haproxy return 502 if loadbalance a fortiweb WAF protected website
Delta, Please let us know Fortinet's answer here. Baptiste On Tue, Dec 31, 2013 at 7:41 AM, Delta Yeh delta@gmail.com wrote: Hi Willy, Yeah, I agree with you. I report it only to make haproxy team know this side case. I have contacted fortinet's tech to check if this is the feature of fortiweb product or product configuration mistake. BR, DeltaY 2013/12/31 Willy Tarreau w...@1wt.eu On Tue, Dec 31, 2013 at 02:04:02PM +0800, Delta Yeh wrote: Hi Lukas, I know the response is crappy like Baptiste said. But as a reverse proxy, nginx works OK for this website, it would be better if haproxy also works for such website. The debug output of wget is: Could you please provide a PCAP output instead ? Your copy-paste is clearly missing some parts. The fact that some headers are left in the body should not block anything, they will just be delivered as a body to the client. So there's something else. Also, the fact that proxy X or browser Y accepts to deliver non-compliant contents isn't a good sign in general, it often means that it's vulnerable to security issues. Just like haproxy when you enable option accept-invalid-http-responses. If someone told me that haproxy works with this option and squid does not, I would not consider it squid's fault. And as Lukas said, please check with Fortinet's support, this bug seems so huge that it there's obviously a fix already. Best regards, Willy
Re: Feature request: TOS based ACL.
Hi, all! What I wanna to do is using acl to capture the TOS field on http-request traffic. On Thu, Jan 2, 2014 at 10:29 AM, Ge Jin altman87...@gmail.com wrote: Hi, Lukas! Thats great, but is there can be anything like this? acl bad_guys tos-acl 0x20 block if bad_guys On Tue, Dec 31, 2013 at 7:14 PM, Lukas Tribus luky...@hotmail.com wrote: Hi, Could haproxy add a tos based acl? http://en.wikipedia.org/wiki/Type_of_service We want to do some action on the traffic based on the tos field. Should work already with something like this: acl local_net src 192.168.0.0/16 http-response set-tos 46 if local_net http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4.2-http-response Regards, Lukas
HA-Proxy version 1.5-dev21-51437d2 2013/12/29 sticky ssl sessons are not working in my environment
Hello ! Problem description - then i access my two web servers through HA-Proxy version 1.5-dev21-51437d2 2013/12/29 it acts as round robin load balancing with out any ssl sticky sessions effect. I would be very pleased if some could help to make sticky ssl sessions work with out ssl offload. Additional information: My scticky tabel output produced by following command: echo show table etlive_https | socat unix-connect:/var/run/haproxy.stat stdio # table: etlive_https, type: binary, size:30720, used:4 0x11b7974: key=0F242856F62F68D2E7C50F7B809D577B00CE7758F74992B4F104A50724153CC6 use=0 exp=1777208 server_id=2 0x11b7ad4: key=11B93E6CEC80076086F73CAFCDA6CEC90E55E12BCBCDD6278181201DA01E505A use=0 exp=1778917 server_id=2 0x11b7a24: key=7A4D134D9E7E02F35E68D69A516EA3DD965C75CA424E1E9BF08014232F7D3A3A use=0 exp=1777300 server_id=1 0x11b7774: key=D2564D3480E88117FD3864376E17BA6C5BA27E18D5000CEB2C888F18ADAAB550 use=0 exp=1773268 server_id=1 I compiled and linked haproxy Under Debian linux using following make options: make TARGET=custom CPU=native USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_LINUX_SPLICE=1 TARGET_CFLAGS=-O2 -mmmx -msse -mfpmath=sse -ffast-math -funsafe-loop-optimizations -funsafe-math-optimizations -fweb -frename-registers -fforce-addr -maccumulate-outgoing-args -momit-leaf-frame-pointer -funswitch-loops -fstack-protector and installed it: make PREFIX=/usr/local/haproxy install My linux operatsystem is Linux lb1 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux My haproxy information (haproxy -vv): HA-Proxy version 1.5-dev21-51437d2 2013/12/29 Copyright 2000-2013 Willy Tarreau w...@1wt.eu Build options : TARGET = custom CPU = native CC = gcc CFLAGS = -O2 -march=native -g -fno-strict-aliasing -O2 -mmmx -msse -mfpmath=sse -ffast-math -funsafe-loop-optimizations -funsafe-math-optimizations -fweb -frename-registers -fforce-addr -maccumulate-outgoing-args -momit-leaf-frame-pointer -funswitch-loops -fstack-protector OPTIONS = USE_LINUX_SPLICE=1 USE_ZLIB=1 USE_POLL=default USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): no Built with zlib version : 1.2.7 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.30 2012-02-04 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND Available polling systems : poll : pref=200, test result OK select : pref=150, test result OK Total: 2 (2 usable), will use poll. My haproxy configuration file haproxy.cfg content : global #stats socket /var/run/haproxy.sock mode 666 stats socket /var/run/haproxy.stat mode 666 log /dev/loglocal0 info log /dev/loglocal0 notice # log 127.0.0.1 local0 chroot /var/lib/haproxy maxconn 10 maxpipes 3 ulimit-n 50 user root group haproxy daemon defaults log global option tcplog option dontlognull retries 3 option redispatch option splice-auto timeout connect 5000ms timeout client 5ms timeout server 5ms option tcp-smart-accept # option tcp-smart-connect frontend etlive_https bind 192.168.35.254:4431,192.168.35.253:4431 option tcplog maxconn 1 log global default_backend etlive_https backend etlive_https mode tcp option ssl-hello-chk # option httpchk GET /test.html option tcplog balance roundrobin stick-table type binary len 32 size 30k expire 30m acl clienthello req_ssl_hello_type 1 acl serverhello rep_ssl_hello_type 2 # use tcp content accepts to detects ssl client and server hello. tcp-request inspect-delay 5s tcp-request content accept if clienthello # no timeout on response inspect delay by default. tcp-response content accept if serverhello # SSL session ID (SSLID) may be present on a client or server hello. # Its length is coded on 1 byte at offset 43 and its value starts # at offset 44. # Match and learn on request if client hello. stick on payload_lv(43,1) if clienthello # Learn on response if server hello. stick store-response payload_lv(43,1) if serverhello server etlive1 192.168.35.232:443 check maxconn 5000 server etlive2 192.168.35.233:443 check maxconn 5000 Lauri-Alo Adamson
RE: HA-Proxy version 1.5-dev21-51437d2 2013/12/29 sticky ssl sessons are not working in my environment
Hi, Problem description – then i access my two web servers through HA-Proxy version 1.5-dev21-51437d2 2013/12/29 it acts as round robin load balancing with out any ssl sticky sessions effect. I would be very pleased if some could help to make sticky ssl sessions work with out ssl offload. Was this previously working and a upgrade to dev-21 introduced this problem or is this a new configuration which never worked? If the former is the case, please indicate what release you used previously and if possible, try dev-20 and dev-19. Also, you said you are using dev21-51437d2, which is actually post dev-21, so I suspect your are using git to download the source code. Are you able to git bisect this behavior? Regards, Lukas
RE: Feature request: TOS based ACL.
Hi, Thats great, but is there can be anything like this? acl bad_guys tos-acl 0x20 block if bad_guys Ah ok, you want to match incoming TOS. That is indeed not supported currently. Also, not all *nixes provide an API for this. Linux has IP_RECVTOS/IPV6_RECVTCLASS to do it, but BSD hasn't, also see: http://stackoverflow.com/questions/1029849/what-is-the-bsd-or-portable-way-to-get-tos-byte-like-ip-recvtos-from-linux Not sure what effort it would be to implement this. Regards, Lukas
Advanced redis health check
Hi Guys, Some of you may have notified that there is a new send/expect style health check in HAProxy. I've just written a short blog post to explain how to use it to make a redis master failover transparent: http://blog.exceliance.fr/2014/01/02/haproxy-advanced-redis-health-check/ More example to come soon. And any feedback on this new feature is welcome :) Baptiste
Re: Immediate health check on startup / after reload
Hi, Le 02/01/2014 04:56, Sok Ann Yap a écrit : I am using haproxy 1.5-dev21, and it seems like health check only happens immediately on startup / after reload for the first backend defined. (...) With `siege http://localhost:8080/` running on a separate console, we now get a bunch of 502 responses with each restart / reload of haproxy. It seems like health check for the rails backend doesn't happen immediately after startup / reload anymore. The gap is much smaller than the default health check interval of 2 seconds, but there is still a gap. You're right, this is the case since 2006. Initial checks are spread over time depending on the minimal inter value and the number of servers to check. This is documented in the inter section : http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#inter In order to reduce resonance effects when multiple servers are hosted on the same hardware, the agent and health checks of all servers are started with a small time offset between them. It is also possible to add some random noise in the agent and health checks interval using the global spread-checks keyword. This makes sense for instance when a lot of backends use the same servers. Imagine 2 servers with inter 10s, the first one will be checked immediately, the second one nearly 5 seconds later. First commit to introduce this spread : http://haproxy.1wt.eu/git?p=haproxy.git;a=commit;h=3759f98d441cc457edf6637c4ba123ca4f42217f A second one that included a minimal threshold : http://haproxy.1wt.eu/git?p=haproxy.git;a=commit;h=2c43a1e2f05161cac4f88c9e9c01bd16f1b2cb5b -- Cyril Bonté
Re: Tuning HAProxy for Production
Hi Baptiste, Sorry about that, it makes sense that you would want that information. :) Mostly this will be routing to various front-end web servers based on the incoming url. There is a main PHP application running on a couple servers, a Tomcat server running authentication, and a few node.js servers. Mostly the PHP servers will be handling the bulk of the load for now. Is that the information you were looking for, or is there something I can dig into more in-depth? On Thu, Jan 2, 2014 at 2:45 AM, Baptiste bed...@gmail.com wrote: Hi Jordan, If you don't provide us any information on the type of services you're going to deliver with HAProxy, then we can't help you! Baptiste On Thu, Jan 2, 2014 at 4:25 AM, Jordan Arentsen jor...@bliss.io wrote: I'm trying to prepare HAProxy for a production, and I'm trying to figure out some good default configuration settings that will at least give me a good place to start. My main question revolves around the maxconn option and the various timeouts. I was thinking about setting the maxconn to 15k or so, is this a bad place to start? Any other advice on baseline performance tuning? My main HAProxy instance will be running on an EC2 c3.large. Thanks!
RE: http-keep-alive broken?
On 31.12.2013 00:50, Lukas Tribus wrote: Hi, Subject: http-keep-alive broken? Hi, I'm using haproxy ss-20131229 to reverse proxy some windows iis server with ntlm-auth enabled (one of them being exchange 2012). While I understood that using 'option http-keep-alive' would make ntlm-auth work, it doesn't work for me. Are there still some issue with http-keep-alive and ntlm-auth? Honestly I would just use the default tunnel mode for this, so I don't have to think about the NTLM crap when choosing keep-alive/load-balancing parameters. If you would like to combine NTLM-auth plus keep-alive, I'd propose enabling: option prefer-last-server http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4.2-option%20prefer-last-server Wile I do agree that using tcp-mode would make stuff easier, I also need to do some redirecting on the host-header. Which is AFAIK not possible while in tcp-mode. (I might be wrong) I tried moving 'option http-keep-alive' to the frontend section but that didn't help. I also used 'option prefer-last-server' but that didn't help as well and I think it wouldn't make any difference since it only redirects to one server. The docs say that http-keep-alive should be useful if (quote): - when the server is non-HTTP compliant and authenticates the connection instead of requests (eg: NTLM authentication) http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#option%20http-keep-alive But as far as I have tested it only breaks NTML auth badly. So, either I'm doing something wrong, or haproxy is doing something wrong, or the docs are wrong about the NTLM part :-) Greets, Sander
Re: http-keep-alive broken?
Sander, I successfully use mode http-keep-alive in my lab for outlook clients getting connected to an exchange 2010 cluster with RPC over HTTP and NTLM auth. So please share here your configuration, so we can have a look at it. Baptiste On Thu, Jan 2, 2014 at 9:16 PM, Sander Klein roe...@roedie.nl wrote: On 31.12.2013 00:50, Lukas Tribus wrote: Hi, Subject: http-keep-alive broken? Hi, I'm using haproxy ss-20131229 to reverse proxy some windows iis server with ntlm-auth enabled (one of them being exchange 2012). While I understood that using 'option http-keep-alive' would make ntlm-auth work, it doesn't work for me. Are there still some issue with http-keep-alive and ntlm-auth? Honestly I would just use the default tunnel mode for this, so I don't have to think about the NTLM crap when choosing keep-alive/load-balancing parameters. If you would like to combine NTLM-auth plus keep-alive, I'd propose enabling: option prefer-last-server http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4.2-option%20prefer-last-server Wile I do agree that using tcp-mode would make stuff easier, I also need to do some redirecting on the host-header. Which is AFAIK not possible while in tcp-mode. (I might be wrong) I tried moving 'option http-keep-alive' to the frontend section but that didn't help. I also used 'option prefer-last-server' but that didn't help as well and I think it wouldn't make any difference since it only redirects to one server. The docs say that http-keep-alive should be useful if (quote): - when the server is non-HTTP compliant and authenticates the connection instead of requests (eg: NTLM authentication) http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#option%20http-keep-alive But as far as I have tested it only breaks NTML auth badly. So, either I'm doing something wrong, or haproxy is doing something wrong, or the docs are wrong about the NTLM part :-) Greets, Sander
Re: Tuning HAProxy for Production
On 2 January 2014 20:09, Jordan Arentsen jor...@bliss.io wrote: I'm trying to prepare HAProxy for a production, and I'm trying to figure out some good default configuration settings that will at least give me a good place to start. My main question revolves around the maxconn option and the various timeouts. I was thinking about setting the maxconn to 15k or so, is this a bad place to start? Any other advice on baseline performance tuning? Mostly this will be routing to various front-end web servers based on the incoming url. There is a main PHP application running on a couple servers, a Tomcat server running authentication, and a few node.js servers. Mostly the PHP servers will be handling the bulk of the load for now. Is that the information you were looking for, or is there something I can dig into more in-depth? That sounds pretty vanilla, so my suggestion would be to start with the defaults and see where that gets your specific application and workload. HAProxy's defaults are sane (I /think/ the default queue timeout and queue size might need increasing, but it's been a while since I've set up a greenfield app from scratch). Remember the sine qua nons of performance tuning are to change one thing at a time, measure things precisely and accurately, and make sure you're comparing apples with apples. You should have an idea of what you'll need maxconn to be, based on either existing logs or your business' traffic predictions. If you have neither of these, set it high and drop it down as you observe you're able to over time. Others may well have more specific recommendations, but that's where I'd start. Jonathan
RE: http-keep-alive broken?
Hi, Wile I do agree that using tcp-mode would make stuff easier, I also need to do some redirecting on the host-header. Which is AFAIK not possible while in tcp-mode. (I might be wrong) No, I really meant http mode, but in the (default) tunneling mode, which can only analyze the first request and then creates a tunnel, meaning that it effectively transforms the session into a tcp mode session. This still gives you the possibility to content-switch based on the Host header in the frontend (at least, in the first requests, but this should be enough, as I doubt Internet Explorer will mix NTLM with non-NTLM requests in a single TCP session - that would bring the brokeness of NTLM to a new level ...). Also, since you can influence the keep-alive settings from the backend, you could still enable keep-alive on the non-NTLM backends, to have full content switching abilities, etc for the regular HTTP traffic. But as far as I have tested it only breaks NTML auth badly. So, either I'm doing something wrong, or haproxy is doing something wrong, or the docs are wrong about the NTLM part :-) I was not implying we shouldn't troubleshoot this problem, I was rather speaking out loudly how I would use NTML backends with current code in general and pre dev-20 code in particular. As for NTLM troubleshooting itself, I'm sure Baptiste can be of better help, as I'm not using NTLM myself. Regards, Lukas
RE: proxy protocol for varnish 3.0.5
Hi, Dumb question: what is the advantage of the proxy protocol for http (as would be the case with varnish)? I assumed the proxy protocol was used to enable load balancing of non-http protocols. It can be useful even when you are load-balancing http. Perhaps you don't want to touch the HTTP headers, because you would like to see the headers exactly as they come from the client. Perhaps you don't run HAProxy in a mode where it can insert the header in every request (http tunneling mode), or at all (in tcp mode). Perhaps your frontend proxy load-balances HTTPS in TCP mode, because you don't allow SSL termination on your first loadbalancing layer for scalability, security or policy reasons (recent question from the varnish mailing list [1]). I'm sure there are other use cases with HTTP(S) I forgot. Regards, Lukas [1] http://www.gossamer-threads.com/lists/varnish/misc/29128
Does haproxy could be a forward proxy?
Hi, this question is silly, but I use haproxy even on my laptop to split traffic, for example, there's a ACL to let some special domains go via remote proxy, and the default goes local proxy, I wonder is it possible to replace local proxy with haproxy, so I could have: server default local:1080 directly without creating a proxy by another tool. Thanks. Bests, -Igor
RE: HA-Proxy version 1.5-dev21-51437d2 2013/12/29 sticky ssl sessons are not working in my environment
Hello , Many thanks for your replay. This thing is more stranger i downloaded and compiled serverl versions of HAproxy 1.5.x.x and the result was alwase the same I experimented with following versions At first i testing with http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev21.tar.gz After i tested with these http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev20.tar.gz http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev18.tar.gz http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev17.tar.gz latest downloded was haproxy-ss-LATEST.tar.gz from http://haproxy.1wt.eu/download/1.5/src/snapshot/ All the time the result was same Lauri-Alo Adamson -Original Message- From: Lukas Tribus [mailto:luky...@hotmail.com] Sent: Thursday, January 02, 2014 5:35 PM To: Lauri-Alo Adamson; haproxy@formilux.org Subject: RE: HA-Proxy version 1.5-dev21-51437d2 2013/12/29 sticky ssl sessons are not working in my environment Hi, Problem description - then i access my two web servers through HA-Proxy version 1.5-dev21-51437d2 2013/12/29 it acts as round robin load balancing with out any ssl sticky sessions effect. I would be very pleased if some could help to make sticky ssl sessions work with out ssl offload. Was this previously working and a upgrade to dev-21 introduced this problem or is this a new configuration which never worked? If the former is the case, please indicate what release you used previously and if possible, try dev-20 and dev-19. Also, you said you are using dev21-51437d2, which is actually post dev-21, so I suspect your are using git to download the source code. Are you able to git bisect this behavior? Regards, Lukas
Re: Feature request: TOS based ACL.
man ip on the freebsd box: If the IP_RECVTTL option is enabled on a SOCK_DGRAM socket, the recvmsg(2) call will return the IP TTL (time to live) field for a UDP datagram. The msg_control field in the msghdr structure points to a buffer that contains a cmsghdr structure followed by the TTL. The cms- ghdr fields have the following values: cmsg_len = CMSG_LEN(sizeof(u_char)) cmsg_level = IPPROTO_IP cmsg_type = IP_RECVTTL If the IP_RECVTOS option is enabled on a SOCK_DGRAM socket, the recvmsg(2) call will return the IP TOS (type of service) field for a UDP datagram. The msg_control field in the msghdr structure points to a buffer that contains a cmsghdr structure followed by the TOS. The cms- ghdr fields have the following values: cmsg_len = CMSG_LEN(sizeof(u_char)) cmsg_level = IPPROTO_IP cmsg_type = IP_RECVTOS FreeBSD only support recv tos or ttl for udp packets. If you want split some tcp request traffic for special purpose, may be you can set ttl or tos on the front router/firewall ,then capture it with ipfw tool and redirect it to the customed frontend. But that leads complex configurations. Simon 于 2/1/14 下午11:56, Lukas Tribus 写道: Hi, Thats great, but is there can be anything like this? acl bad_guys tos-acl 0x20 block if bad_guys Ah ok, you want to match incoming TOS. That is indeed not supported currently. Also, not all *nixes provide an API for this. Linux has IP_RECVTOS/IPV6_RECVTCLASS to do it, but BSD hasn't, also see: http://stackoverflow.com/questions/1029849/what-is-the-bsd-or-portable-way-to-get-tos-byte-like-ip-recvtos-from-linux Not sure what effort it would be to implement this. Regards, Lukas
