tproxy bug in haproxy-1.5.10
Hi, I wanted to report a nasty bug I discovered today: Portforwarding to a different IP on the same haproxy-box causes haproxy behave buggy. This error happens when one uses the TPROXY target for portforwarding, ie. like this: Let's say the IP of the main interface is 192.168.100.100, and traffic from outside to port 1234 shall be forwarded to a virtual IP 192.168.100.101 and port 5678 on the same host, ie: auto eth0 allow-hotplug eth0 iface eth0 inet static address 192.168.100.100 netmask 255.255.255.0 broadcast 192.168.100.255 gateway 192.168.100.254 metric1 up ip addr add 192.168.100.101/32 dev eth0 label eth0:101 down ip addr del 192.168.100.101/32 dev eth0 label eth0:101 ... iptables -t mangle -A PREROUTING -p tcp --dport 1234 -j TPROXY --tproxy-mark 666 --on-ip 192.168.100.101 --on-port 5678 When REDIRECT instead of TPROXY is used then it works, but with that one can forward only to another port of the main IP (192.168.100.100) of the interface, ie. here eth0. I can confirm this bug because putting a different transparent proxy program (actually my own test program) at the same IP:Port works fine with such portforwardings. My environment: Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) x86_64 GNU/Linux, ie. Debian 8 I compiled haproxy myself w/o any modifications: # ./haproxy -vv HA-Proxy version 1.5.10 2014/12/31 Copyright 2000-2014 Willy Tarreau w...@1wt.eu Build options : TARGET = linux26 CPU = x86_64 CC = gcc CFLAGS = -g -fno-strict-aliasing OPTIONS = USE_LINUX_TPROXY=1 USE_STATIC_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built without OpenSSL support (USE_OPENSSL not set) Built with PCRE version : 8.35 2014-04-04 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll.
Cookie persistence - what I am I doing wrong?
I'm trying to ensure that multiple connections from the same browser end up on the same back end server, and having lots of trouble. All my work with haproxy up to now has been with connections that don't need persistence - everything relevant happens in one http request. This is probably PEBCAK or ID10T ... but I am not seeing my mistake. Here's the frontend and backend I've got: frontend nc-80 description Front end that accepts requests for production. bind X.X.X.72:80 acl blockit path_beg-i /v2.0 http-request deny if blockit default_backend nc-80-backend backend nc-80-backend description Back end for main site cookie JSESSIONID prefix server frontier 10.100.2.25:80 weight 100 track apache80/frontier server fremont 10.100.2.26:80 weight 100 track apache80/fremont server fiesta 10.100.2.29:80 weight 150 track apache80/fiesta The log lines all have --NN in them with these settings, and requests from a single browser page load are hitting all three webservers. I also tried 'cookie SRV insert indirect nocache' with no better results. With this, the log lines all have --NI. Here's the -vv output: HA-Proxy version 1.5.8 2014/10/31 Copyright 2000-2014 Willy Tarreau w...@1wt.eu Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.3 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008 Running on OpenSSL version : OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : no (version might be too old, 0.9.8f min needed) OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 6.6 06-Feb-2006 PCRE library supports JIT : no (USE_PCRE_JIT not set) Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Thanks, Shawn
Re: Cookie persistence - what I am I doing wrong?
Hi Shawn, Le 15/01/2015 01:59, Shawn Heisey a écrit : I'm trying to ensure that multiple connections from the same browser end up on the same back end server, and having lots of trouble. All my work with haproxy up to now has been with connections that don't need persistence - everything relevant happens in one http request. This is probably PEBCAK or ID10T ... but I am not seeing my mistake. Here's the frontend and backend I've got: frontend nc-80 description Front end that accepts requests for production. bind X.X.X.72:80 acl blockit path_beg-i /v2.0 http-request deny if blockit default_backend nc-80-backend backend nc-80-backend description Back end for main site cookie JSESSIONID prefix server frontier 10.100.2.25:80 weight 100 track apache80/frontier server fremont 10.100.2.26:80 weight 100 track apache80/fremont server fiesta 10.100.2.29:80 weight 150 track apache80/fiesta You need to specify a cookie value for each server, with the cookie keyword : http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#cookie%20%28Server%20and%20default-server%20options%29 For example : server frontier 10.100.2.25:80 weight 100 cookie frontier track apache80/frontier server fremont 10.100.2.26:80 weight 100 cookie fremont track apache80/fremont server fiesta 10.100.2.29:80 weight 150 cookie fiesta track apache80/fiesta Use any value you want but keep them unique to have stickiness requests on a specific server. For an application session JSESSIONID 12345678901234567890123456789012, from the client side it will result in cookies like : JSESSIONID=frontier~12345678901234567890123456789012 JSESSIONID=fremont~12345678901234567890123456789012 JSESSIONID=fiesta~12345678901234567890123456789012 The log lines all have --NN in them with these settings, and requests from a single browser page load are hitting all three webservers. I also tried 'cookie SRV insert indirect nocache' with no better results. With this, the log lines all have --NI. It will also work once cookie values are set. Here's the -vv output: HA-Proxy version 1.5.8 2014/10/31 Copyright 2000-2014 Willy Tarreau w...@1wt.eu Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.3 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008 Running on OpenSSL version : OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : no (version might be too old, 0.9.8f min needed) OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 6.6 06-Feb-2006 PCRE library supports JIT : no (USE_PCRE_JIT not set) Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Thanks, Shawn -- Cyril Bonté
Re: Haproxy SSL Redirection issue
Hi Cyrill, Thanks for the update. In Application side we are not using any SSL offloading. I have tested with pound (for ssl termination) haproxy (for load balancing) combination and issue got resolved. But if i tried with haproxy only , then getting the issue. I think there is some configuration changes need to made with haproxy to support application side URL redirect. Please find the status codes details. Please help me. 1. haproxy (only) https://www.example.com/search/quotes? : 302 http://www.example.com/search/quotes? : 301 https://www.example.com/search/quotes? : 200 2. Pound (termination ) haproxy ( loadbalancing) https://www.example.com/search/quotes? : 302 https://www.example.com/search/quotes? : 200 3. On Tue, Jan 13, 2015 at 5:49 PM, RAKESH P B pb.rakes...@gmail.com wrote: Hi Cyrill, Thanks for the update. I'll check with my developer and let you know how the way SSL offloading support in Application side. On Tue, Jan 13, 2015 at 3:14 AM, Cyril Bonté cyril.bo...@free.fr wrote: Hi, Le 12/01/2015 22:27, RAKESH P B a écrit : Hi All, Can I have an update on this. From what you describe and your configuration, this is not haproxy related but you should take a look on the application or on the server itself. But we can't tell you more, you didn't provide any information on them. By adding a X-Forwarded-Proto header, your server/application must take it into account. And this is where there is no standard. For example : - some applications require X-Forwarded-Proto: https - some others require that the server set an environment variables, and depending on the component, it can have different names/values, some are case sensitive, some others not (HTTPS=on, HTTPS=On, ...) - apache redirects will use a special syntax on ServerName with a https:// prefix, ... As you see, it depends on what the developers thought at the time they included SSL Offloading support. On Sun, Jan 11, 2015 at 6:56 PM, RAKESH P B pb.rakes...@gmail.com mailto:pb.rakes...@gmail.com wrote: Please find updated configuration file. On Sun, Jan 11, 2015 at 6:53 PM, RAKESH P B pb.rakes...@gmail.com mailto:pb.rakes...@gmail.com wrote: Hi Lukas, Thanks you for the quick response. Please find the attached Haproxy configuration. On Sun, Jan 11, 2015 at 5:21 PM, Lukas Tribus luky...@hotmail.com mailto:luky...@hotmail.com wrote: Hi Team, I have an issue Haproxy SSL redirection. Whenever any request is redirected from HAproxy , then two redirected request is send, one is with http and other with https while URL for both request is same. For example, when the URL is redirected tohttps://www.example.com/to/ path? , then these two request are send http://www.example.com/path1/path2/path3? https://www.example.com/path1/path2/path3? Can you share the config? Lukas -- Cyril Bonté
Round Robin not very random
Hi All, We got hit with a bit of traffic and we saw haproxy dump most of the traffic to 3-4 app servers, sometimes even just one and driving load on there to 90. We were running 1.5.9, I upgraded to 1.5.10 and the same problem remained. Currently traffic is low so everything is load balanced evenly, but we expect another spike in a few hours and I expect the issue to return. Here's what haproxy-status looked like: Do I need to switch to maybe add a weight and tracking? We have 12 frontend appservers load balancing to 28. All run haproxy and the app server software. Thanks! Alex
Send specific string on TCP
Hello folks, I'd like to use the proxy / webirc feature of ngIRCd. For that, it is required that the proxy sends the following command at the beginning of the communication : WEBIRC password username hostname ip-address (see http://ngircd.barton.de/doc/Protocol.txt, section II.4) How can I do this in my backend ? Shall I use something like tcp-check send ? I'd like, in this example, to replace the ip-address variable by something computed by HAProxy, and which reflects the actual IP address of the client. Thanks ! signature.asc Description: OpenPGP digital signature
Re: Haproxy SSL Redirection issue
Hi, Le 15/01/2015 03:59, RAKESH P B a écrit : Hi Cyrill, Thanks for the update. In Application side we are not using any SSL offloading. I have tested with pound (for ssl termination) haproxy (for load balancing) combination and issue got resolved. I guess you have RewriteLocation 0 in your pound configuration (the default is 1), this will explain why you see a difference. But if i tried with haproxy only , then getting the issue. I think there is some configuration changes need to made with haproxy to support application side URL redirect. Please find the status codes details. Please help me. This is really not the job of the reverse proxy to decide by itself to change the protocol in the redirects (it can add very bad side effects) but in your case, if you're sure that there won't be any valid http redirect, you can rewrite the headers sent in the response. For example with : http-response replace-header Location ^http://(.*) https://\1 1. haproxy (only) https://www.example.com/search/quotes? : 302 Here you'll have Location: http://www.example.com/search/quotes? catched by the rule given previously, rewriting the header to : Location: https://www.example.com/search/quotes? http://www.example.com/search/quotes? : 301 This step won't happen anymore. https://www.example.com/search/quotes? : 200 2. Pound (termination ) haproxy ( loadbalancing) https://www.example.com/search/quotes? : 302 https://www.example.com/search/quotes? : 200 3. On Tue, Jan 13, 2015 at 5:49 PM, RAKESH P B pb.rakes...@gmail.com mailto:pb.rakes...@gmail.com wrote: Hi Cyrill, Thanks for the update. I'll check with my developer and let you know how the way SSL offloading support in Application side. On Tue, Jan 13, 2015 at 3:14 AM, Cyril Bonté cyril.bo...@free.fr mailto:cyril.bo...@free.fr wrote: Hi, Le 12/01/2015 22:27, RAKESH P B a écrit : Hi All, Can I have an update on this. From what you describe and your configuration, this is not haproxy related but you should take a look on the application or on the server itself. But we can't tell you more, you didn't provide any information on them. By adding a X-Forwarded-Proto header, your server/application must take it into account. And this is where there is no standard. For example : - some applications require X-Forwarded-Proto: https - some others require that the server set an environment variables, and depending on the component, it can have different names/values, some are case sensitive, some others not (HTTPS=on, HTTPS=On, ...) - apache redirects will use a special syntax on ServerName with a https:// prefix, ... As you see, it depends on what the developers thought at the time they included SSL Offloading support. On Sun, Jan 11, 2015 at 6:56 PM, RAKESH P B pb.rakes...@gmail.com mailto:pb.rakes...@gmail.com mailto:pb.rakes...@gmail.com mailto:pb.rakes...@gmail.com__ wrote: Please find updated configuration file. On Sun, Jan 11, 2015 at 6:53 PM, RAKESH P B pb.rakes...@gmail.com mailto:pb.rakes...@gmail.com mailto:pb.rakes...@gmail.com mailto:pb.rakes...@gmail.com__ wrote: Hi Lukas, Thanks you for the quick response. Please find the attached Haproxy configuration. On Sun, Jan 11, 2015 at 5:21 PM, Lukas Tribus luky...@hotmail.com mailto:luky...@hotmail.com mailto:luky...@hotmail.com mailto:luky...@hotmail.com wrote: Hi Team, I have an issue Haproxy SSL redirection. Whenever any request is redirected from HAproxy , then two redirected request is send, one is with http and other with https while URL for both request is same. For example, when the URL is redirected tohttps://www.example.com/to/__path http://www.example.com/to/path? , then these two request are send http://www.example.com/path1/__path2/path3 http://www.example.com/path1/path2/path3? https://www.example.com/__path1/path2/path3 https://www.example.com/path1/path2/path3? Can you share the config? Lukas -- Cyril Bonté -- Cyril Bonté
No TCP RST on tcp-request connection reject
Hey guys, just a thought... wouldn't it make sense to add an option to tcp-request connection reject to disable the actual TCP RST? So, an attacker tries to (keep) open a lot of ports: a) HAProxy (configured with rate limiting etc.) does a tcp-request connection reject which ends up as a TCP RST. The attacker gets the RST and immediately again b) the same as a) but the socket will be closed on the server side but no RST, nothing will be sent back to the remote side. The connections on the remote side will be kept open until timeout. Wouldn't it make sense to implement an option for b) so it can be used during major attacks or so?
Re: No TCP RST on tcp-request connection reject
On Wed, Jan 14, 2015 at 5:00 PM, Christian Ruppert c.rupp...@babiel.com wrote: Hey guys, just a thought... wouldn't it make sense to add an option to tcp-request connection reject to disable the actual TCP RST? So, an attacker tries to (keep) open a lot of ports: a) HAProxy (configured with rate limiting etc.) does a tcp-request connection reject which ends up as a TCP RST. The attacker gets the RST and immediately again b) the same as a) but the socket will be closed on the server side but no RST, nothing will be sent back to the remote side. The connections on the remote side will be kept open until timeout. Wouldn't it make sense to implement an option for b) so it can be used during major attacks or so? Hi Christian, Have you had a look at tarpit related options from HAProxy? You can slowdown the attack thanks to it. Baptiste
Re: No TCP RST on tcp-request connection reject
Hi Baptiste, tarpit is pretty handy but as far as I understood it will keep the connection open, on both sides. So at some point (pretty quickly actually) we cannot handle any more connections on that host. The host will become slow and/or unresponsive. When we close the connection on our local side but don't notify the remote side it will probably exhaust the attacker and we could handle more connections and/or free and re-use such connections that has been classified too much. On 01/14/2015 05:28 PM, Baptiste wrote: On Wed, Jan 14, 2015 at 5:00 PM, Christian Ruppert c.rupp...@babiel.com wrote: Hey guys, just a thought... wouldn't it make sense to add an option to tcp-request connection reject to disable the actual TCP RST? So, an attacker tries to (keep) open a lot of ports: a) HAProxy (configured with rate limiting etc.) does a tcp-request connection reject which ends up as a TCP RST. The attacker gets the RST and immediately again b) the same as a) but the socket will be closed on the server side but no RST, nothing will be sent back to the remote side. The connections on the remote side will be kept open until timeout. Wouldn't it make sense to implement an option for b) so it can be used during major attacks or so? Hi Christian, Have you had a look at tarpit related options from HAProxy? You can slowdown the attack thanks to it. Baptiste -- Mit freundlichen Grüßen, Christian Ruppert Systemadministrator .. Babiel GmbH Erkrather Str. 224 a D-40233 Düsseldorf Tel: 0211-179349 0 Fax: 0211-179349 29 c.rupp...@babiel.com http://www.babiel.com GESCHÄFTSFÜHRER Georg Babiel, Dr. Rainer Babiel, Harald Babiel Amtsgericht Düsseldorf HRB 38633 DISCLAIMER The information transmitted in this electronic mail message may contain confidential and or privileged materials. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you receive such e-mails in error, please contact the sender and delete the material from any computer.
