Re: Haproxy CPU 100%, after running about two weeks
Thanks! I follow your advise, and upgrade my haproxy. And I will observe if there is any problem. Regards Jinge On 2013-5-2, at 下午3:49, Lukas Tribus luky...@hotmail.com wrote: Hi Jinge! I believe you are facing 2 different issues here. Today, our haproxy CPU grow to 100%. And the machine become terribly slow. Please upgrade to recent 1.4 code, you are missing a a few fixes, including one a security fix. I suggest the snapshot 20130427 which also includes a loop fix (causing 100% load from haproxy). Download at [1]. [1297314.773541] cleanup rbuf bug: copied DBE7B6DA seq DBE7B3C8 rcvnxt DBE7B6DA [...] [1297314.773625] [81046a75] ? warn_slowpath_common+0x78/0x8c This is a kernel issue with tcp splicing and has probably been fixed. Please see [2]. Not sure if Debian is backporting this fix though. You could just disable tcp splicing as a intermediate workaround. Cheers, Lukas [1] http://haproxy.1wt.eu/download/1.4/src/snapshot/ [2] http://comments.gmane.org/gmane.linux.network/231555
haproxy crashes with ddos mitigation config
Hello,
I currently have some troubles enabling ddos as described there :
http://blog.exceliance.fr/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/
When i enable the following lines :
stick-table type ip size 100k expire 30s store conn_cur
# Shut the new connection as long as the client has already 10 opened
tcp-request connection reject if { src_conn_cur ge 10 }
tcp-request connection track-sc1 src
...
haproxy crashes with the following error :
kernel: [334012.858141] haproxy[6914] general protection ip:46832d
sp:7fffe5e219e8 error:0 in haproxy[40+89000]
Regards,
Smana
Re: haproxy crashes with ddos mitigation config
More information
OS : debian 6
version : 1.5-dev18
2013/5/3 Smain Kahlouch smain...@gmail.com
Hello,
I currently have some troubles enabling ddos as described there :
http://blog.exceliance.fr/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/
When i enable the following lines :
stick-table type ip size 100k expire 30s store conn_cur
# Shut the new connection as long as the client has already 10 opened
tcp-request connection reject if { src_conn_cur ge 10 }
tcp-request connection track-sc1 src
...
haproxy crashes with the following error :
kernel: [334012.858141] haproxy[6914] general protection ip:46832d
sp:7fffe5e219e8 error:0 in haproxy[40+89000]
Regards,
Smana
Re: haproxy crashes with ddos mitigation config
Another information,
This behavior appears only with ssl frontends.
I'm trying with non ssl frontend
2013/5/3 Smain Kahlouch smain...@gmail.com
More information
OS : debian 6
version : 1.5-dev18
2013/5/3 Smain Kahlouch smain...@gmail.com
Hello,
I currently have some troubles enabling ddos as described there :
http://blog.exceliance.fr/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/
When i enable the following lines :
stick-table type ip size 100k expire 30s store conn_cur
# Shut the new connection as long as the client has already 10 opened
tcp-request connection reject if { src_conn_cur ge 10 }
tcp-request connection track-sc1 src
...
haproxy crashes with the following error :
kernel: [334012.858141] haproxy[6914] general protection ip:46832d
sp:7fffe5e219e8 error:0 in haproxy[40+89000]
Regards,
Smana
RE: haproxy crashes with ddos mitigation config
Hi Smana! haproxy crashes with the following error : kernel: [334012.858141] haproxy[6914] general protection ip:46832d sp:7fffe5e219e8 error:0 in haproxy[40+89000] Please share the output of haproxy -vv. This behavior appears only with ssl frontends. Upgrade to latest snapshot from [1] and reprovide haproxy -vv. The snapshot contains ssl related fixes (missing in dev18). If this still happens with the snapshot, please provide a coredump? You can follow this instructions (quoting myself, I didn't find that email in the archives): Make sure: - haproxy is started as root - it doesn't use chrooting - don't use the init script, but start haproxy manually right after you used the ulimit command - otherwise follow the description here: http://www.mail-archive.com/haproxy@formilux.org/msg09472.html The ´cat /proc/sys/kernel/core_pattern´ setting specifies in what directory the coredump is saved to. By following the above suggestion you make your setup less secure, so use it with care and revert those things after you obtained the coredump. The coredump itself and the gdb output as well will contain things like ip addresses, etc. Please be aware of that when you post it to the mailing list. If you cannot publish such data you may just sent the gdb output to Willy Tarreau w...@1wt.eu instead of the mailing list. [1] http://haproxy.1wt.eu/download/1.5/src/snapshot/ Cheers, Lukas
haproxy 1.4.23 bug on TCP content inspection rules ?
hi, all Go straight to the point 1. base information haproxy version: 1.4.23 ip: 192.168.1.1 snippet of haproxy.cfg *..* *frontendtcp_frontend bind:3128 modetcp* ** *tcp-request inspect-delay 3s tcp-request content accept if HTTP* ** *use_backend http_backend if HTTP !HTTP_URL_ABS default_backend tcp_backend* ** *backend tcp_backend modetcp option transparent* ** *backend http_backend modehttp server s1 127.0.0.1:8081* *...* 2. when curl -v -x 192.168.1.1:3128 http://www.sohu.com/;, it is horrible... * PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 17260 nobody18 0 151m 45m 456 R 97.7 0.6 25:14.91 haproxy * = VIRT up to 151m, RES up to 45m, and CPU up to 97.7% 3. config line *use_backend http_backend if HTTP !HTTP_URL_ABS* change to *use_backend http_backend if HTTP* it is ok now, bug the PROXY requests can not be filtered, any suggest? huaqiuyu
RE: haproxy crashes with ddos mitigation config
Yes, please reproduce with latest snapshot, and provide the output
of haproxy -vv. Also, setup haproxy so it can generate a core.
Make sure you CC the list haproxy@formilux.org when responding.
Thanks,
Lukas
Date: Fri, 3 May 2013 11:14:14 +0200
Subject: Re: haproxy crashes with ddos mitigation config
From: smain...@gmail.com
To: luky...@hotmail.com
Below is my haproxy -vv :
A-Proxy version 1.5-dev18 2013/04/03
Copyright 2000-2013 Willy Tarreau w...@1wt.eumailto:w...@1wt.eu
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing
OPTIONS = USE_OPENSSL=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built without zlib support (USE_ZLIB not set)
Compression algorithms supported : identity
Built with OpenSSL version : OpenSSL 0.9.8o 01 Jun 2010
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
I'll try the snapshot
---
Regarding the ddos mitigation config on non ssl frontend, it doesn't
seem to work properly.
My frontend config
...
frontend public
bind 0.0.0.0:80http://0.0.0.0:80
# # Table definition
stick-table type ip size 5m expire 30s store conn_cur
# # Allow clean known IPs to bypass the filter
# tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
# # Shut the new connection as long as the client has already 10 opened
tcp-request connection reject if { src_conn_cur ge 10 }
tcp-request connection track-sc1 src
mode http
log global
option httplog
timeout client 25s
maxconn 1000
timeout http-request 80s
capture request header Host len 64
capture request header Referer len 256
capture request header User-Agent len 64
...
Then i open 10 connexions :
ab -n 50 -c 10 http://127.0.0.1:80/
I open another connexion with a telnet and i'm able to open it.
watch 'echo show table services | socat unix:./haproxy.stats -'
0xb5c6a4: key=127.0.0.1 use=11 exp=3 conn_cur=11
Maybe i should open a new thread ?
Regards,
Smana
Re: Monitor always returns HTTP 200
Hi guys, Thanks for the responses, my replies are below! On 2 May 2013 17:48, Lukas Tribus luky...@hotmail.com wrote: I always receive a HTTP 200 response to my browser How do you know that? tcpdump In what condition does this happen (when you have less than 2 backends alive or even with 2 or more backends alive?) With 2, 1, or 0 back ends alive (I have been adding iptables rules to the Apache servers [there are two] one at a time so that haproxy backend server checks fail, and I can see this reflected in the haproxy log) . default_backend http--servers [...] backend http-servers The config doesn't seem to match That was just a typo from me copying the config into my original email, sorry about that! Please post the output of haproxy -vv. sudo haproxy -vv HA-Proxy version 1.4.8 2010/06/16 Copyright 2000-2010 Willy Tarreau w...@1wt.eu Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -O2 -g OPTIONS = Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Available polling systems : sepoll : pref=400, test result OK epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 4 (4 usable), will use sepoll. On 2 May 2013 22:39, Bryan Talbot btal...@aeriagames.com wrote: On Thu, May 2, 2013 at 8:55 AM, James Bensley jwbens...@gmail.com wrote: acl backend_down nbsrv(http--servers) lt 2 # HAProxy can see lee than 2 backend servers monitor-uri /checkuri monitor-net 172.22.0.0/24 What's the address of the computer making the requests? If it's in the 172.22.0.0/24 network, all responses for any URI will be 200 as long as monitor fail is false. Ah! This was the information I was missing. I changed that to a /32 and tried from another machine and the behaviour is now more like what I expected is seen. The other machine passes through haproxy to the backends as it should, and requests to the monitor URL return 404, which is what I wanted. However I do still have another problem, I always get HTTP 200 from the machine in monitor-net. I can add an iptables rules on either of my two Apache servers or both, to stop haproxy from contacting them (which I can verify via tcpdump on Apache and tcpump on the haproxy server. I can see it can't get past TCP syn because there is no syn-ack back from the apache servers). This first paste bin entry shows the haproxy server detecting the first Apache server as down, I browse to my monitor uri and as you can see from the tcpdump output, I get HTTP 200 back: http://pastebin.com/raw.php?i=va57gf0K In this second paste bin entry I have added the log line from haproxy after adding a drop rule to the second Apache server iptables config, you can see here that haproxy can now see neither Apache server. yet I still get HTTP 200; http://pastebin.com/raw.php?i=bPcNP8kH If anyone can shed any light on this I would be very grateful. Cheers, James.
Re: haproxy 1.4.23 bug on TCP content inspection rules ?
Hi, On Fri, May 03, 2013 at 05:45:45PM +0800, Jianhua Qin wrote: hi, all Go straight to the point 1. base information haproxy version: 1.4.23 ip: 192.168.1.1 snippet of haproxy.cfg *..* *frontendtcp_frontend bind:3128 modetcp* ** *tcp-request inspect-delay 3s tcp-request content accept if HTTP* ** *use_backend http_backend if HTTP !HTTP_URL_ABS default_backend tcp_backend* ** *backend tcp_backend modetcp option transparent* ** *backend http_backend modehttp server s1 127.0.0.1:8081* *...* 2. when curl -v -x 192.168.1.1:3128 http://www.sohu.com/;, it is horrible... * PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 17260 nobody18 0 151m 45m 456 R 97.7 0.6 25:14.91 haproxy * = VIRT up to 151m, RES up to 45m, and CPU up to 97.7% 3. config line *use_backend http_backend if HTTP !HTTP_URL_ABS* change to *use_backend http_backend if HTTP* it is ok now, bug the PROXY requests can not be filtered, any suggest? I suspect from your configuration that you're making haproxy reconnect to itself : the transparent option in the backend makes it connect to the original destination ip:port. So unless you used some iptables rules to force the traffic into haproxy when it as not destinated to it, I'm quite certain that the backend reconnects to the frontend. Willy
Re: Monitor always returns HTTP 200
Hi James, On Thu, May 02, 2013 at 04:55:16PM +0100, James Bensley wrote: Hi all, I have configured haproxy using the below configuration. No matter what URL I browser to I always receive a HTTP 200 response to my browser. If I comment out the ACL and three monitor lines from the frontend configuration, normal behaviour is resumed. I have that gut feeling that I have done something obviously wrong, but I can't spot it :) Where am I going wrong with this? I assume that haproxy captures requests to /checkuri and sends back HTTP 200 when at least 2 back end servers are up (which they are, I can see the requests coming into them) and 503 when one or more is down. Otherwise, all other requests are passed directly to the back end servers. It seems to be intercepting whatever URI I request via GET and returns HTTP 200 OK, nothing reaches the back end servers when the monitor URI is configured. frontend monitor-http-servers bind 1.2.3.4:80 acl backend_down nbsrv(http--servers) lt 2 # HAProxy can see lee than 2 backend servers monitor-uri /checkuri monitor-net 172.22.0.0/24 monitor fail if backend_down As explained in the doc, monitor-net unconditionally returns 200 to all connections coming from the specified network. If your request comes from another network, then monitor fail will apply to requests matching monitor-uri. I must confess it's the first time I see the two mechanisms mixed and that's a bit confusing. Still, as you can see in the doc, monitor-net doesn't even give you a chance to reach monitor-uri : In HTTP mode, a connection coming from a source matching source will be accepted, the following response will be sent without waiting for a request, then the connection will be closed : HTTP/1.0 200 OK. Simply remove this monitor-net statement and it should work as expected. Willy
Set cookie with external service
Hi guys. Is it possible to do load balancing based on a cookie, and if the cookie is not set, make HAProxy connect to a external service that returns a cookie value to HAProxy and make HAProxy assing this cookie before continue to the correct backend? Example first time user: 1. Client connects, and no cookie is found. 2. HAProxy connects to IP/Port and gets a cookie value in return. 3. HAProxy sets this cookie. 4. HAProxy assigns backend based on cookie. Example on a returning user: 1. Cookie is found. 2. HAProxy assigns backend based on cookie. Thanks, Daniel Storjordet
Re: Set cookie with external service
Hi Daniel, On Fri, May 03, 2013 at 01:57:35PM +0200, Daniel Storjordet wrote: Hi guys. Is it possible to do load balancing based on a cookie, and if the cookie is not set, make HAProxy connect to a external service that returns a cookie value to HAProxy and make HAProxy assing this cookie before continue to the correct backend? This looks somewhat related to what haproxy used to do, in its very first implementation something like 12 years ago, except that it will not be able to pass the reqeust to the backend server since the request is already lost once sent to the cookie server. Example first time user: 1. Client connects, and no cookie is found. 2. HAProxy connects to IP/Port and gets a cookie value in return. 3. HAProxy sets this cookie. 4. HAProxy assigns backend based on cookie. Use the dispatch statement for this, and have your cookie server use a redirect. First-time visitors will be sent to the ip:port specified in dispatch, and this server will handle the request. If it performs a redirect in addition to setting a cookie, the browser will send a new request with the cookie and haproxy will pass it to the appropriate server. However it's still unclear to me why you'd rely on a specific server to do this when haproxy does it for free by default. Willy
Re: Monitor always returns HTTP 200
Hi Willy, Thanks for clearing that up; On 3 May 2013 12:28, Willy Tarreau w...@1wt.eu wrote: As explained in the doc, monitor-net unconditionally returns 200 to all connections coming from the specified network. If your request comes from another network, then monitor fail will apply to requests matching monitor-uri. I must confess it's the first time I see the two mechanisms mixed and that's a bit confusing. I did read the docs but bad sadly, understanding fail! OK so now everything is working as expected; If I make a request from a host in my monitor subnet I always receive HTTP 200 so that tells my that HAProxy is running. Then from a host outside the monitor subnet to the monitor URI I can see how many backends HAProxy sees (any less than my configured 2 and it returns 503, which I have tested and is working perfectly now thanks!). One last point! :D I have noticed that if I point a browse to my monitor URI (not in the monitor subnet) when both my back ends are up I receive HTTP 200 OK HAProxy: service ready. However if I refresh this page any quicker than about once a minute, it doesn't load? Check out this paste bin; http://pastebin.com/raw.php?i=1xyNtcYq I am packet capturing on a client (172.22.0.220, not in the monitor subnet), browsing to the monitor uri (GET /oowahboh6eibooca) you can see at 14:08:24 I get a response 200 OK. Then I refresh the page 2 seconds later at 14:08:26.215969 and at 14:08:26.217989 I get a 404 response. This doesn't really matter as I will only be checking every 5 minutes or so, but I thought I should mentioned it in case it's a bug or I'm being silly again. Cheers, James.
Re: Set cookie with external service
Hi Willy. Thanks for the answer. Making the client have to do a redirect is a drawback as it will result in making the first request quite slower. The reason for this setup is because each portal can have multiple channels, each with its own domain and we need to maintain session and portal state inbetween the channels. I am also considering using FiddlerCore as a inbetween proxy that can handle the logic of picking the correct application pool. Mvh, Daniel Destino AS 2013/5/3 Willy Tarreau w...@1wt.eu Hi Daniel, On Fri, May 03, 2013 at 01:57:35PM +0200, Daniel Storjordet wrote: Hi guys. Is it possible to do load balancing based on a cookie, and if the cookie is not set, make HAProxy connect to a external service that returns a cookie value to HAProxy and make HAProxy assing this cookie before continue to the correct backend? This looks somewhat related to what haproxy used to do, in its very first implementation something like 12 years ago, except that it will not be able to pass the reqeust to the backend server since the request is already lost once sent to the cookie server. Example first time user: 1. Client connects, and no cookie is found. 2. HAProxy connects to IP/Port and gets a cookie value in return. 3. HAProxy sets this cookie. 4. HAProxy assigns backend based on cookie. Use the dispatch statement for this, and have your cookie server use a redirect. First-time visitors will be sent to the ip:port specified in dispatch, and this server will handle the request. If it performs a redirect in addition to setting a cookie, the browser will send a new request with the cookie and haproxy will pass it to the appropriate server. However it's still unclear to me why you'd rely on a specific server to do this when haproxy does it for free by default. Willy
RE: Monitor always returns HTTP 200
Hi James, I am packet capturing on a client (172.22.0.220, not in the monitor subnet), browsing to the monitor uri (GET /oowahboh6eibooca) you can see at 14:08:24 I get a response 200 OK. Then I refresh the page 2 seconds later at 14:08:26.215969 and at 14:08:26.217989 I get a 404 response. You are running HAProxy 1.4.8, which is ancient. Please upgrade to 1.4.23 which contains a ton of bugfixes. Lukas
Re: Monitor always returns HTTP 200
On 3 May 2013 14:49, Lukas Tribus luky...@hotmail.com wrote: Hi James, I am packet capturing on a client (172.22.0.220, not in the monitor subnet), browsing to the monitor uri (GET /oowahboh6eibooca) you can see at 14:08:24 I get a response 200 OK. Then I refresh the page 2 seconds later at 14:08:26.215969 and at 14:08:26.217989 I get a 404 response. I just tested this with Telnet and I always get 200 back so it must be something odd with my browser or test machine etc. Ignore this :) You are running HAProxy 1.4.8, which is ancient. Please upgrade to 1.4.23 which contains a ton of bugfixes. Received, roger! Cheers, James.
Re: haproxy 1.4.23 bug on TCP content inspection rules ?
Thanks for the quick reply. I fix it by using some iptables rules just as you said. Can it be avoided naturely? huaqiuyu 2013/5/3 Willy Tarreau w...@1wt.eu Hi, On Fri, May 03, 2013 at 05:45:45PM +0800, Jianhua Qin wrote: hi, all Go straight to the point 1. base information haproxy version: 1.4.23 ip: 192.168.1.1 snippet of haproxy.cfg *..* *frontendtcp_frontend bind:3128 modetcp* ** *tcp-request inspect-delay 3s tcp-request content accept if HTTP* ** *use_backend http_backend if HTTP !HTTP_URL_ABS default_backend tcp_backend* ** *backend tcp_backend modetcp option transparent* ** *backend http_backend modehttp server s1 127.0.0.1:8081* *...* 2. when curl -v -x 192.168.1.1:3128 http://www.sohu.com/;, it is horrible... * PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 17260 nobody18 0 151m 45m 456 R 97.7 0.6 25:14.91 haproxy * = VIRT up to 151m, RES up to 45m, and CPU up to 97.7% 3. config line *use_backend http_backend if HTTP !HTTP_URL_ABS* change to *use_backend http_backend if HTTP* it is ok now, bug the PROXY requests can not be filtered, any suggest? I suspect from your configuration that you're making haproxy reconnect to itself : the transparent option in the backend makes it connect to the original destination ip:port. So unless you used some iptables rules to force the traffic into haproxy when it as not destinated to it, I'm quite certain that the backend reconnects to the frontend. Willy
Re: haproxy 1.4.23 bug on TCP content inspection rules ?
On Fri, May 03, 2013 at 10:01:50PM +0800, Jianhua Qin wrote: Thanks for the quick reply. I fix it by using some iptables rules just as you said. Can it be avoided naturely? No, because transparent is really meant to be used that way. And haproxy has no way of knowing that the destination address will point to it, as it is supposed to be dealt with by the system. Willy
RE: TCP Keepalives
Hi James! Are the docs refering to these timers? http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html Correct. Cheers, Lukas
Re: TCP Keepalives
On 3 May 2013 17:28, Lukas Tribus luky...@hotmail.com wrote: Hi James! Are the docs refering to these timers? http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html Correct. Thanks Lukas, just wanted to check before I start trashing my test servers :) Cheers, James.
