Re: Weird stick-tables / peers behaviour
On 03/24/2016 04:07 PM, Christian Ruppert wrote: Hi all, I've just upgraded some hosts to 1.6.4 (from 1.5) and immediately got a [...] and two for doing some "curl -Lvs http://127.0.0.1:8080"; by hand. If you do some on the first and some on the second host you'll notice different values on one side. Also the counter may e.g. double while the other side has the correct/actual value. This results into several thousands of requests on our prod. systems but according to the logs it can't be correct. Does anybody else have similar weirdness or can you guys confirm false values? The *_cnt values seem to be ok but the *_rate ones seem to be false in some cases. Hi, This is a know bug in the 1.6 series. Willy spoke about it in a few posts on this list. It is not fixed yet. Regards. Sylvain
Re: Protecting against slow HTTP POST queries
On 02/04/2016 12:06 PM, Baptiste wrote: On Thu, Feb 4, 2016 at 9:44 AM, Sylvain Faivre wrote: Hi, Is there a timeout setting in HAproxy that can help protect against slow HTTP POST queries ? >> [...] please run the same test against HAProxy 1.6 and enable "option buffer-http-request": http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#4-option%20http-buffer-request Then your timeout http-request will also match the POSTed data. Hi Baptiste, This did the trick, thanks. Sylvain
Protecting against slow HTTP POST queries
Hi, Is there a timeout setting in HAproxy that can help protect against slow HTTP POST queries ? I'm not talking about "slow loris" type attacks (where the client sleeps between request headers) but "slow HTTP POST" (where the client sleeps between POST data lines). Here is an example : - Test 1 : root@proxy1>: telnet localhost 85 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. POST /test HTTP/1.1 Host: host.domain.com User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) Content-Type: application/json; charset=utf-8 Content-Length: 1234 test <> HTTP/1.1 408 Request Time-out Date: Wed, 03 Feb 2016 13:03:30 GMT Content-Length: 223 Content-Type: text/html; charset=iso-8859-1 408 Request Time-out Request Time-out Server timeout waiting for the HTTP request from the client. Connection closed by foreign host. - Test 2 : Here we send the POST body very slowly (line by line, wait 10 seconds between each line). root@proxy1>: telnet localhost 85 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. POST /test HTTP/1.1 Host: host.domain.com User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) Content-Type: application/json; charset=utf-8 Content-Length: 1234 test <> test <> test <> test <> test <> test <> test <> Connection closed by foreign host. In each case, HAproxy log shows termination flags "SD--" which means the application server closed the connection. So, the app server timeouts after 30 seconds, but this duration is reset each time the client sends data in the POST body. Is there an option to set a timeout on this part of the request ? It should be similar to "timeout http-request" but work against the request body, instead of the request headers. We already have these settings, but none of them seems to act against HTTP POST content (I was able to stay connected while sending HTTP POST content for 5+ minutes) : timeout connect 5s timeout http-request12s timeout queue 180s timeout client 180s timeout server 180s timeout http-keep-alive 10s timeout tarpit 30s Best regards, Sylvain
Re: stick table replication problem
Hi Tom, According to Emeric Brun, this bug was fixed by two commits, a few days ago. Unfortunately, I didn't have time to check this yet. Quoting Emeric : It outputs two major bugs in peersv1 and v2 protocol implementation (present from 1.4 haproxy version) This bug have been fixed in the latest dev git commits: 1c6235dbba0a67bad1d5e57ada88f28e1270a5cb 234fc3c31e751f8191b9b78fa5fd16663c2627fe They are also backported in 1.5 and 1.6 branches. On 12/24/2015 11:00 AM, Tom Atkinson wrote: I had the same thing happen. Here is my peers config and 'show sess all' output for the peer sessions. In this case, www1 had a new stick table entry which was not propagated. peers mypeers peer www1.example.com 192.168.140.3:1024 peer www2.example.com 192.168.132.3:1024 peer www3.example.com 192.168.149.3:1024 www1.example.com: 0x55c97d047120: [08/Dec/2015:17:12:51.249700] id=0 proto=tcpv4 flags=0x23006, conn_retries=0, srv_conn=(nil), pend_pos=(nil) frontend=www1.example.com (id=4294967295 mode=tcp), listener=? (id=0) backend= (id=-1 mode=-) addr=192.168.140.3:43847 server= (id=-1) addr=192.168.149.3:1024 task=0x55c97d0470a0 (state=0x08 nice=0 calls=2 exp= age=15d16h) si[0]=0x55c97d047318 (state=CLO flags=0x4040 endp0=APPCTX:0x55c97d047420 exp=, et=0x000) si[1]=0x55c97d047338 (state=CON flags=0x50 endp1=CONN:0x55c97d04beb0 exp=, et=0x020) app0=0x55c97d047420 st0=11 st1=0 st2=0 applet= co1=0x55c97d04beb0 ctrl=tcpv4 xprt=RAW data=STRM target=PROXY:0x55c97cf83100 flags=0x0020b310 fd=3 fd_spec_e=22 fd_spec_p=0 updt=0 req=0x55c97d047130 (f=0x80a020 an=0x0 pipe=0 tofwd=0 total=0) an_exp= rex= wex= buf=0x55c97b7964e0 data=0x55c97b7964f4 o=0 p=0 req.next=0 i=0 size=0 res=0x55c97d047170 (f=0x80402020 an=0x0 pipe=0 tofwd=0 total=0) an_exp= rex= wex= buf=0x55c97b7964e0 data=0x55c97b7964f4 o=0 p=0 rsp.next=0 i=0 size=0 0x55c97d05bb40: [10/Dec/2015:22:44:23.248632] id=0 proto=tcpv4 flags=0x23006, conn_retries=0, srv_conn=(nil), pend_pos=(nil) frontend=www1.example.com (id=4294967295 mode=tcp), listener=? (id=0) backend= (id=-1 mode=-) addr=192.168.140.3:58534 server= (id=-1) addr=192.168.149.3:1024 task=0x55c97d05bac0 (state=0x08 nice=0 calls=2 exp= age=13d10h) si[0]=0x55c97d05bd38 (state=CLO flags=0x4040 endp0=APPCTX:0x55c97d064a50 exp=, et=0x000) si[1]=0x55c97d05bd58 (state=CON flags=0x50 endp1=CONN:0x55c97d04d370 exp=, et=0x020) app0=0x55c97d064a50 st0=11 st1=0 st2=0 applet= co1=0x55c97d04d370 ctrl=tcpv4 xprt=RAW data=STRM target=PROXY:0x55c97cf83100 flags=0x0020b310 fd=13 fd_spec_e=22 fd_spec_p=0 updt=0 req=0x55c97d05bb50 (f=0x80a020 an=0x0 pipe=0 tofwd=0 total=0) an_exp= rex= wex= buf=0x55c97b7964e0 data=0x55c97b7964f4 o=0 p=0 req.next=0 i=0 size=0 res=0x55c97d05bb90 (f=0x80402020 an=0x0 pipe=0 tofwd=0 total=0) an_exp= rex= wex= buf=0x55c97b7964e0 data=0x55c97b7964f4 o=0 p=0 rsp.next=0 i=0 size=0 0x55c97d04e1c0: [11/Dec/2015:18:52:25.323298] id=0 proto=tcpv4 flags=0x23006, conn_retries=0, srv_conn=(nil), pend_pos=(nil) frontend=www1.example.com (id=4294967295 mode=tcp), listener=? (id=0) backend= (id=-1 mode=-) addr=192.168.140.3:38958 server= (id=-1) addr=192.168.132.3:1024 task=0x55c97d04e140 (state=0x08 nice=0 calls=2 exp= age=12d14h) si[0]=0x55c97d04e3b8 (state=CLO flags=0x4040 endp0=APPCTX:0x55c97d04fdc0 exp=, et=0x000) si[1]=0x55c97d04e3d8 (state=CON flags=0x50 endp1=CONN:0x55c97d064e60 exp=, et=0x020) app0=0x55c97d04fdc0 st0=11 st1=0 st2=0 applet= co1=0x55c97d064e60 ctrl=tcpv4 xprt=RAW data=STRM target=PROXY:0x55c97cf83100 flags=0x0020b310 fd=23 fd_spec_e=22 fd_spec_p=0 updt=0 req=0x55c97d04e1d0 (f=0x80a020 an=0x0 pipe=0 tofwd=0 total=0) an_exp= rex= wex= buf=0x55c97b7964e0 data=0x55c97b7964f4 o=0 p=0 req.next=0 i=0 size=0 res=0x55c97d04e210 (f=0x80402020 an=0x0 pipe=0 tofwd=0 total=0) an_exp= rex= wex= buf=0x55c97b7964e0 data=0x55c97b7964f4 o=0 p=0 rsp.next=0 i=0 size=0 0x55c97d0a45b0: [16/Dec/2015:19:52:57.872505] id=0 proto=tcpv4 flags=0x23006, conn_retries=0, srv_conn=(nil), pend_pos=(nil) frontend=www1.example.com (id=4294967295 mode=tcp), listener=? (id=0) backend= (id=-1 mode=-) addr=192.168.140.3:38595 server= (id=-1) addr=192.168.132.3:1024 task=0x55c97d0a4530 (state=0x08 nice=0 calls=2 exp= age=7d13h) si[0]=0x55c97d0a47a8 (state=CLO flags=0x4040 endp0=APPCTX:0x55c97d054800 exp=, et=0x000) si[1]=0x55c97d0a47c8 (state=CON flags=0x50 endp1=CONN:0x55c97d0bd640 exp=, et=0x020) app0=0x55c97d054800 st0=11 st1=0 st2=0 applet= co1=0x55c97d0bd640 ctrl=tcpv4 xprt=RAW data=STRM target=PROXY:0x55c97cf83100 flags=0x0
Re: stick table replication problem
Hi Willy, Thanks for your help. On 12/03/2015 03:25 PM, Willy Tarreau wrote: > > [...] Thanks for this precision. All I can say for now is that you clearly encountered a bug but that we don't know what this bug is. We'll have to check in the code for something which could cause this. It would be interesting to know for how long the process has been running before the issue appeared, eventhough it will not tell us for how long the connection remained alive. According to our logs, both HAproxy processes were started at Nov 24 11:25:xx and application errors caused by lack of session replication started happenning at Dec 1 17:05:35 So that's a bit more than 1 week later. We'll keep looking for replication errors in the next days, and specially in 1 week, and report back. Sylvain
stick table replication problem
Hi, We just had a strange replication problem on our staging environment. We have 2 HAproxy servers. They were running for 2 weeks now. At the beginning, I checked that the stick tables were properly synced. Today, stick tables were not synced, for example : root@proxy1>: echo "show table front_jsessionid" | socat stdio /usr/share/haproxy/mysocket # table: front_jsessionid, type: string, size:10485760, used:3 0xd1d944: key=MkJm-rbcE3V5EJg1RmTRAw__ use=0 exp=3474723 server_id=1 0xd762c4: key=Z2rxYufqv7B0C2gIgztSfA__ use=0 exp=3571578 server_id=2 0xea9484: key=aUuiOrlUDb7RnvcFMLK9oA__ use=0 exp=2879968 server_id=2 root@proxy2>: echo "show table front_jsessionid" | socat stdio /usr/share/haproxy/mysocket # table: front_jsessionid, type: string, size:10485760, used:5 0x1bc9104: key=D8EWzwrGr2UK3btCwfpweQ__ use=0 exp=3552238 server_id=1 0x1c3ba54: key=Z2rxYufqv7B0C2gIgztSfA__ use=0 exp=3315450 server_id=1 0x1cbecb4: key=aUuiOrlUDb7RnvcFMLK9oA__ use=0 exp=2624851 server_id=2 0x1c49664: key=lmqFlaNhdHXYwH4M4QQjqQ__ use=0 exp=3510898 server_id=2 0x1cbf074: key=zpE6NHQJr~aStxzmJTVEgA__ use=0 exp=2587567 server_id=2 -- I tried : echo "set table front_jsessionid key toto data.server_id 1" | socat stdio /usr/share/haproxy/mysocket The new entry was not replicated on proxy2, and tcpdump showed no outgoing traffic from proxy1 to proxy2. Then, I reloaded the haproxy service on proxy2, and tables were synced, including old data set in proxy1 before the reload. New data replicates all right too. Should this situation happen again, is there anything I can do to debug this further ? -- Here is netstat output from proxy1 while replication was broken : root@proxy1>: netstat -anp |grep 9421 tcp0 0 [proxy1]:94210.0.0.0:*LISTEN 1163/haproxy tcp 35 0 [proxy1]:9421[proxy2]:41205 CLOSE_WAIT - tcp1 0 [proxy1]:20512 [proxy2]:9421CLOSE_WAIT 1163/haproxy tcp1 0 [proxy1]:35662 [proxy2]:9421CLOSE_WAIT 1163/haproxy tcp 35 0 [proxy1]:9421[proxy2]:41257 CLOSE_WAIT - tcp0 0 [proxy1]:54443 [proxy2]:9421FIN_WAIT2 1163/haproxy tcp1 0 [proxy1]:52254 [proxy2]:9421CLOSE_WAIT 1163/haproxy tcp1 0 [proxy1]:54249 [proxy2]:9421CLOSE_WAIT 1163/haproxy tcp 35 0 [proxy1]:9421[proxy2]:41160 CLOSE_WAIT - tcp 35 0 [proxy1]:9421[proxy2]:41110 CLOSE_WAIT - tcp1 0 [proxy1]:40469 [proxy2]:9421CLOSE_WAIT 1163/haproxy -- Here are relevant parts from our setup : peers prod peer proxy1[proxy1]:9421 peer proxy2[proxy2]:9421 backend front stick on urlp(jsessionid),url_dec table front_jsessionid stick on urlp(jsessionid,;),url_dec table front_jsessionid stick on cookie(JSESSIONID) table front_jsessionid stick store-response cookie(JSESSIONID) table front_jsessionid backend front_jsessionid stick-table type string len 24 size 10m expire 1h peers prod -- We are running HAproxy 1.6.2 from the vbernat PPA on Ubuntu 12.04, with nbproc = 1. Thanks in advance. Sylvain
Re: Email checks in defaults section
On 11/01/2015 06:34 PM, Tommy Atkinson wrote: I want to enable email alerts for all my backends so I added the "email-alert" options to the defaults section and a mailers section at the top level. The documentation indicates this is supported but it doesn't seem to work. HAProxy connects to the mail server but doesn't actually send anything. Copy/pasting the options to a backend works. 1.6.1 on Linux Hi, This seems to be a bug, I had the same problem. Copying & pasting the "email-alert" settings in all the backend sections did the trick. When email-alert options are set only in the defaults section, just like Tommy said, HAproxy connects to the mail server but doesn't send anything. Here are the postfix logs from the problem : Nov 30 09:31:01 host1 postfix/smtpd[2502]: connect from localhost[127.0.0.1] Nov 30 09:31:03 host1 postfix/smtpd[2502]: lost connection after CONNECT from localhost[127.0.0.1] Nov 30 09:31:03 host1 postfix/smtpd[2502]: disconnect from localhost[127.0.0.1] With fixed config : Nov 30 10:03:58 host1 postfix/smtpd[8335]: connect from localhost[127.0.0.1] Nov 30 10:03:58 host1 postfix/smtpd[8335]: 79A28402E7: client=localhost[127.0.0.1] Nov 30 10:03:58 host1 postfix/cleanup[8338]: 79A28402E7: message-id=<20151130090358.79A28402E7@host1> Nov 30 10:03:58 host1 postfix/smtpd[8335]: disconnect from localhost[127.0.0.1] Nov 30 10:03:58 host1 postfix/qmgr[1346]: 79A28402E7: from=, size=600, nrcpt=1 (queue active) Nov 30 10:03:58 host1 postfix/smtp[8339]: 79A28402E7: to=, relay=x:25, delay=0.14, delays=0.06/0.03/0.04/0, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 962E420D9C) Nov 30 10:03:58 host1 postfix/qmgr[1346]: 79A28402E7: removed (using 1.6.2 Ubuntu package) Best regards, Sylvain
Re: appsession replacement in 1.6
Hi Willy, On 11/18/2015 09:31 PM, Willy Tarreau wrote: Hi Sylvain, On Tue, Nov 17, 2015 at 03:09:49PM +0100, Sylvain Faivre wrote: option http-buffer-request maybe you should stick on the header ;-) OK I added "option http-buffer-request", it will help for sure ! What do you mean "stick on the header" ? Oh well, it seems like I forgot to stick on the request URL parameter, when passed with a question mark. What Aleks meant is that you don't need http-buffer-request as it's only used to process POST data which isn't your case. Oh OK. Well, I did not talk about http-buffer-request in the beginning, so I thought Aleks recommended to use it. I can see that it is not useful in my case, so I removed it again. So I just added this to my config : stick on urlp(JSESSIONID) table back_jsessionid stick store-request urlp(JSESSIONID) table back_jsessionid It should cover all cases now. These two are partially redundant, since "stick on" is already a shortcut for "stick match" + "stick store-request". So you end up having this : stick match urlp(JSESSIONID) table back_jsessionid stick store-request urlp(JSESSIONID) table back_jsessionid stick store-request urlp(JSESSIONID) table back_jsessionid Thus you can safely remove the "stick store-request" rule, you won't notice the difference. OK, I did that, and also changed urlp(jsessionid) to lower case, since our application server passes it in lower case when in URL parameters. For future reference, here is my latest config. It has been running for a few hours on our staging servers without a problem : backend front [stick-table for another purpose] [other stuff] # sticky session on JSESSIONID stick on urlp(jsessionid) table front_jsessionid stick on urlp(jsessionid,;) table front_jsessionid stick on cookie(JSESSIONID) table front_jsessionid stick store-response cookie(JSESSIONID) table front_jsessionid backend front_jsessionid stick-table type string len 24 size 10m expire 1h peers prod Best regards, Sylvain
Re: appsession replacement in 1.6
Replying to myself : On 11/17/2015 10:58 AM, Sylvain Faivre wrote: Hi Aleks, and thanks again for your help. Concerning this point : On 11/16/2015 11:16 PM, Aleksandar Lazic wrote: As described here http://git.haproxy.org/?p=haproxy-1.6.git;a=blob;f=doc/configuration.txt;h=45d1aacfbe0d2d53193f7956a0dd03e5f8151ea6;hb=HEAD#l5043 option http-buffer-request maybe you should stick on the header ;-) OK I added "option http-buffer-request", it will help for sure ! What do you mean "stick on the header" ? Oh well, it seems like I forgot to stick on the request URL parameter, when passed with a question mark. So I just added this to my config : stick on urlp(JSESSIONID) table back_jsessionid stick store-request urlp(JSESSIONID) table back_jsessionid It should cover all cases now. I currently have the following config, I think it will stick on JSESSIONID, either in the cookie (so request header, also capturing the cookie in reply header) or as a request URL parameter. Am I missing something ? backend front [stick-table for another purpose] [other stuff] # sticky session on JSESSIONID option http-buffer-request stick on urlp(JSESSIONID,;) table front_jsessionid stick on cookie(JSESSIONID) table front_jsessionid stick store-response cookie(JSESSIONID) table front_jsessionid stick store-request cookie(JSESSIONID) table front_jsessionid stick store-request urlp(JSESSIONID,;) table front_jsessionid backend front_jsessionid stick-table type string len 24 size 10m expire 1h peers prod
Re: appsession replacement in 1.6
Hi Aleks, and thanks again for your help. Concerning this point : On 11/16/2015 11:16 PM, Aleksandar Lazic wrote: As described here http://git.haproxy.org/?p=haproxy-1.6.git;a=blob;f=doc/configuration.txt;h=45d1aacfbe0d2d53193f7956a0dd03e5f8151ea6;hb=HEAD#l5043 option http-buffer-request maybe you should stick on the header ;-) OK I added "option http-buffer-request", it will help for sure ! What do you mean "stick on the header" ? I currently have the following config, I think it will stick on JSESSIONID, either in the cookie (so request header, also capturing the cookie in reply header) or as a request URL parameter. Am I missing something ? backend front [stick-table for another purpose] [other stuff] # sticky session on JSESSIONID option http-buffer-request stick on urlp(JSESSIONID,;) table front_jsessionid stick on cookie(JSESSIONID) table front_jsessionid stick store-response cookie(JSESSIONID) table front_jsessionid stick store-request cookie(JSESSIONID) table front_jsessionid stick store-request urlp(JSESSIONID,;) table front_jsessionid backend front_jsessionid stick-table type string len 24 size 10m expire 1h peers prod
Re: appsession replacement in 1.6
Hi Aleks, On 11/10/2015 10:56 PM, Aleksandar Lazic wrote: Dear Sylvain Faivre, Am 10-11-2015 12:48, schrieb Sylvain Faivre: On 11/10/2015 12:00 AM, Aleksandar Lazic wrote: Hi Sylvain Faivre. Am 09-11-2015 17:31, schrieb Sylvain Faivre: [snipp] So, I've got this so far : backend http stick-table type string len 24 size 10m expire 1h peers prod stick on urlp(JSESSIONID,;) stick on cookie(JSESSIONID) Does this seem right ? The help for "stick on" tells it defines a request pattern, so I guess this would not match JSESSIONID cookie ou url parameter set in the reply ? I have no java server here to test this commands but with this commands haproxy does not warn you about some config errors ;-). ### backend dest01 mode http stick-table type string len 24 size 10m expire 1h peers prod stick on urlp(JSESSIONID,;) stick on cookie(JSESSIONID) stick store-response cookie(JSESSIONID) # stick store-response res.hdr(JSESSIONID,;) stick store-request cookie(JSESSIONID) stick store-request urlp(JSESSIONID,;) server srv_dest01 dest01.com:80 ### I have not seen a good option to read the JSESSIONID from the response header in case it is not in a cookie. Have anyone I idea?! Please can you post a full response header which was created from the app or appserver when the app or appserver have detected that the client does not allow cookies. [snipp] In fact, the server sends the JSESSIONID as a cookie even if the client does not support cookies, *and* it adds the JSESSIONID as a URL parameter in all links, so this should be all right. This would be helpfully to see the full response. Maybe some appserver behaves different. As far as I know, there is no way for the server to detect if the client has cookies enabled, by looking only at the first request from that client. According to a quick Google search, the most common ways to detect cookies support are either to use Javascript (so client-side check) or to reply with a redirect response with the cookie set, then when processing the redirected URL, check if the client sent the cookie along with the request (so this case will be covered by the proposed HAproxy settings). I don't feel comfortable giving our application server version on a public list, but I will send it to you in private. Here are the headers from a client request and server reply, with a brand new profile on the client (with cookies disabled) : - request : GET /front/url1.do?m=booking&langcode=FR HTTP/1.1 Host: redacted.host.domain User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Connection: keep-alive - reply : HTTP/1.1 200 OK Date: Mon, 16 Nov 2015 15:25:41 GMT Content-Type: text/html;charset=ISO-8859-15 Set-Cookie: JSESSIONID=uNmYNvgUME5-8LYPzimsCg__.8d15fc; Path=/front Vary: Accept-Encoding Content-Encoding: gzip Transfer-Encoding: chunked And here is a URL sample from the body of the reply. You will notice that the jsessionid is there twice, first one after a semicolon and second one after a question mark. I am not sure if this comes from the application server of from our custom code. <tt>src="<a rel="nofollow" href="https://redacted.host.domain/front/url2.do;jsessionid=uNmYNvgUME5-8LYPzimsCg__.8d15fc?jsessionid=uNmYNvgUME5-8LYPzimsCg__.8d15fc&langcode=FR"">https://redacted.host.domain/front/url2.do;jsessionid=uNmYNvgUME5-8LYPzimsCg__.8d15fc?jsessionid=uNmYNvgUME5-8LYPzimsCg__.8d15fc&langcode=FR"</a>; </tt><tt>language="JavaScript"> I just tried your config on a test server, and the session IDs are rightly recorded in the table, whether the client accepts cookies or not. Perfect. I still have some test cases to run, I will check this next week and report back if needed. Oh yes please tell us the results so that we can add this as migration example for appsession. OK, I will. This will not go into production yet, we still need to run it on a test environment for at least 3 weeks... Best regards. Sylvain
Re: appsession replacement in 1.6
On 11/10/2015 12:00 AM, Aleksandar Lazic wrote: Hi Sylvain Faivre. Am 09-11-2015 17:31, schrieb Sylvain Faivre: Hi, Sorry I'm late on this discussion, following this thread : https://marc.info/?l=haproxy&m=143345620219498&w=2 We are using appsession with HAproxy 1.5 like this : Thanks ;-) backend http appsession JSESSIONID len 24 timeout 1h request-learn We would like to be able to do the same thing with HAproxy 1.6. If it is possible, we'd like to catch the JSESSIONID in cookies and URL parameters, either in the request or in the response. I tried to use the info posted previously by Aleksandar, but I encountered several problems : - using "stick on" in frontend section fails with : 'stick' ignored because frontend 'web' has no backend capability. - using "stick store-response urlp(JSESSIONID,;)" in backend section fails with : 'stick': fetch method 'urlp' extracts information from 'HTTP request headers', none of which is available for 'store-response'. So, I've got this so far : backend http stick-table type string len 24 size 10m expire 1h peers prod stick on urlp(JSESSIONID,;) stick on cookie(JSESSIONID) Does this seem right ? The help for "stick on" tells it defines a request pattern, so I guess this would not match JSESSIONID cookie ou url parameter set in the reply ? I have no java server here to test this commands but with this commands haproxy does not warn you about some config errors ;-). ### backend dest01 mode http stick-table type string len 24 size 10m expire 1h peers prod stick on urlp(JSESSIONID,;) stick on cookie(JSESSIONID) stick store-response cookie(JSESSIONID) # stick store-response res.hdr(JSESSIONID,;) stick store-request cookie(JSESSIONID) stick store-request urlp(JSESSIONID,;) server srv_dest01 dest01.com:80 ### I have not seen a good option to read the JSESSIONID from the response header in case it is not in a cookie. Have anyone I idea?! Please can you post a full response header which was created from the app or appserver when the app or appserver have detected that the client does not allow cookies. cheers Aleks Thank you Aleks. In fact, the server sends the JSESSIONID as a cookie even if the client does not support cookies, *and* it adds the JSESSIONID as a URL parameter in all links, so this should be all right. I just tried your config on a test server, and the session IDs are rightly recorded in the table, whether the client accepts cookies or not. I still have some test cases to run, I will check this next week and report back if needed. Regards, Sylvain
appsession replacement in 1.6
Hi, Sorry I'm late on this discussion, following this thread : https://marc.info/?l=haproxy&m=143345620219498&w=2 We are using appsession with HAproxy 1.5 like this : backend http appsession JSESSIONID len 24 timeout 1h request-learn We would like to be able to do the same thing with HAproxy 1.6. If it is possible, we'd like to catch the JSESSIONID in cookies and URL parameters, either in the request or in the response. I tried to use the info posted previously by Aleksandar, but I encountered several problems : - using "stick on" in frontend section fails with : > 'stick' ignored because frontend 'web' has no backend capability. - using "stick store-response urlp(JSESSIONID,;)" in backend section fails with : > 'stick': fetch method 'urlp' extracts information from 'HTTP request headers', none of which is available for 'store-response'. So, I've got this so far : backend http stick-table type string len 24 size 10m expire 1h peers prod stick on urlp(JSESSIONID,;) stick on cookie(JSESSIONID) Does this seem right ? The help for "stick on" tells it defines a request pattern, so I guess this would not match JSESSIONID cookie ou url parameter set in the reply ? Regards, Sylvain
Re: use several gpc's ?
On 11/05/2015 03:30 PM, Baptiste wrote: On Thu, Nov 5, 2015 at 2:48 PM, Sylvain Faivre wrote: Hi, Is there a way to use several gpc's ? I already use gpc0 to track client IPs generating too many errors, and I need to use another counter to track client IPs requesting some pages too fast. Here are the relevant parts of my current setup : frontend web stick-table type ip size 500k expire 5m store gpc0 tcp-request content track-sc1 src http-request deny if !i_internal { sc1_get_gpc0 gt 0 } backend front stick-table type ip size 100k expire 5m store http_err_rate(10s) tcp-request content track-sc2 src acl error_rate_abuse sc2_http_err_rate gt 10 acl mark_as_abuser sc1_inc_gpc0 gt 0 reqtarpit . if error_rate_abuse !whitelist mark_as_abuser And I'm trying to add something like this to the frontend : stick-table type ip size 50k expire 24h store gpc0_rate(60s) acl pages_info path_sub -i info.php acl too_many_info_requests sc0_gpc0_rate() gt 50 acl mark_seen_pages_info sc0_inc_gpc0 gt 0 tcp-request content track-sc0 src if pages_info http-request deny if mark_seen_pages_info too_many_info_requests But I'm afraid that I will not be able to distinguish the info stored in gpc0 for the error count and for the requests count... What am I missing here ? Hi Sylvain, Which version of HAProxy are you using? With 1.6, there are some converters that may be used to get rid of using gpc while counting errors. It means you would store abuser client IP in a dedicated table and simply check if the IP is there: http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#in_table I have on my TODO to write such type of article on the blog. Some kind of DDOS protection with HAProxy 1.6. Baptiste We are using HAproxy 1.5, upgrading to 1.6 shouldn't be a huge problem. I guess I'll wait for your article, since I'm not sure I understand everything about all this table stuff. So, with HAproxy 1.5, one cannot have two types of DDOS protection at the same time ? (against flag offenders who send too many requests, and those whose requests cause too many errors)
use several gpc's ?
Hi, Is there a way to use several gpc's ? I already use gpc0 to track client IPs generating too many errors, and I need to use another counter to track client IPs requesting some pages too fast. Here are the relevant parts of my current setup : frontend web stick-table type ip size 500k expire 5m store gpc0 tcp-request content track-sc1 src http-request deny if !i_internal { sc1_get_gpc0 gt 0 } backend front stick-table type ip size 100k expire 5m store http_err_rate(10s) tcp-request content track-sc2 src acl error_rate_abuse sc2_http_err_rate gt 10 acl mark_as_abuser sc1_inc_gpc0 gt 0 reqtarpit . if error_rate_abuse !whitelist mark_as_abuser And I'm trying to add something like this to the frontend : stick-table type ip size 50k expire 24h store gpc0_rate(60s) acl pages_info path_sub -i info.php acl too_many_info_requests sc0_gpc0_rate() gt 50 acl mark_seen_pages_info sc0_inc_gpc0 gt 0 tcp-request content track-sc0 src if pages_info http-request deny if mark_seen_pages_info too_many_info_requests But I'm afraid that I will not be able to distinguish the info stored in gpc0 for the error count and for the requests count... What am I missing here ?
log SSL/TLS protocol version
Hello, We use Haproxy in front of HTTP servers, SSL termination is done on HAproxy. Is there a way to have HAproxy log the SSL or TLS protocol version (TLS 1.0 / 1.1 / 1.2) or specific cipher that was used for requests ? I know this is negociated between each client and the HAproxy server, but I would like to know which clients use outdated protocols. Thanks.
Re: add header or query parameter when redirecting
On 06/03/2015 04:02 PM, Baptiste wrote: On Wed, Jun 3, 2015 at 11:58 AM, Sylvain Faivre wrote: Hello, I use the redirect directive to redirect users from old sites to a new site, eg: redirect prefix http://new-site.com code 301 if old-site I would like to redirect requests from many old sites to the same new site, so I need a way to add info about the old host in the new redirected request. I'm looking for a way to add a header to the redirected request to identify the host, for example : X-Orig-Site: old-site-123.com Is this possible ? I guess I can't add a header to the request with HAproxy, since HAproxy only sends a new Location header to the browser, and the browser sets the headers. So, is there a way to alter the location sent in the redirect, to include « &orig-site=old-site-123.com » ? I think I'm missing something here. Should I user « http-request redirect » instead of « redirect prefix » ? By the way, I tried to use the set-cookie option for this, but it was a bad idea : redirect prefix http://new-site.com code 301 set-cookie ORIG=%[hdr(host)] if old_site This doesn't work for two reasons : 1. The « %[hdr(host)] » part is send literally in the request : Set-Cookie: ORIG=%[hdr(host)]; path=/; 2. The request sent to new-site.com doesn't seem to include this cookie Sylvain Hi Sylvain, The only "good way" to do what you want to achieve, is to use a query string parameter and http-request and http-response rules coupled to a few sections... Basically, haproxy is not able to modify the headers sent by a redirect rule. So the trick here, is to perform the redirect in a dummy frontend section used as a server in a dedicated backend and insert a header in the response, like this: backend be_redirect http-request capture req.hdr(host),word(1,:),lower len 32 http-response replace-value Location (.*) \1&orig-site=%[capture.req.hdr(0)] if { res.hdr(Location) -m sub ? } http-response replace-value Location (.*) \1?orig-site=%[capture.req.hdr(0)] if !{ res.hdr(Location) -m sub ? } server dummy_redirect 127.0.0.1:8001 frontend fe_dummy_redirect bind 127.0.0.1:8001 http-request redirect prefix http://new-site.com code 301 Note that this configuration needs HAProxy 1.6 (latest snapshot). Baptiste Hi Baptiste, Unfortunately, we are not willing to upgrade to HAproxy 1.6 just yet, so we are going to use another solution for this redirect (change DNS records to resolve old hostnames to the new web server). Thank you for the info anyway, it may be useful for another time. Sylvain
add header or query parameter when redirecting
Hello, I use the redirect directive to redirect users from old sites to a new site, eg: redirect prefix http://new-site.com code 301 if old-site I would like to redirect requests from many old sites to the same new site, so I need a way to add info about the old host in the new redirected request. I'm looking for a way to add a header to the redirected request to identify the host, for example : X-Orig-Site: old-site-123.com Is this possible ? I guess I can't add a header to the request with HAproxy, since HAproxy only sends a new Location header to the browser, and the browser sets the headers. So, is there a way to alter the location sent in the redirect, to include « &orig-site=old-site-123.com » ? I think I'm missing something here. Should I user « http-request redirect » instead of « redirect prefix » ? By the way, I tried to use the set-cookie option for this, but it was a bad idea : redirect prefix http://new-site.com code 301 set-cookie ORIG=%[hdr(host)] if old_site This doesn't work for two reasons : 1. The « %[hdr(host)] » part is send literally in the request : Set-Cookie: ORIG=%[hdr(host)]; path=/; 2. The request sent to new-site.com doesn't seem to include this cookie Sylvain
understanding HAproxy stats
Hello, I have trouble understanding stats reports from our HAproxy servers, can anyone please shed some light on this ? 1. On a backend with only one server, scur(BACKEND) > scur(server). How can this be ? # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime, backend_b,server_a,0,0,1,1,1,92433,167970344,127389583,,0,,0,0,0,0,UP,1,1,0,512,1,20,1,,6373,backend_a/server_a,2,3,,290,92432,0,0,0,0,00,0,0,,,12858,2,530,13826, backend_b,BACKEND,48,142,49,143,410,92481,167970344,127389583,0,0,,0,0,0,0,UP,1,1,0,,0,88182,0,,1,20,0,,6373,,1,0,,650,92432,0,0,0,0,0,0,0,0,0,0,183,,,12858,2,530,13826, 2. Same HAproxy instance, on another backend, smax(BACKEND) > slim(server_a) + slim(server_b). Again, how is this possible ? It seems like smax(BACKEND) = smax(server_a) + smax(server_b) + qmax(BACKEND) ? # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime, backend_c,server_a,0,0,0,3,3,95541,232232183,241897167,,0,,0,0,0,0,UP,1,1,0,512,1,19,1,,95507,backend_a/server_a,2,0,,50,95541,0,0,0,0,01,0,2,,,0,1,706,2028, backend_c,server_b,0,0,0,3,3,95587,232144919,241789273,,0,,0,0,0,0,UP,1,1,0,512,1,19,2,,95546,backend_a/server_b,2,0,,50,95587,0,0,0,0,00,0,3,,,0,2,675,2082, backend_c,BACKEND,0,3,0,9,410,191128,464377102,483686440,0,0,,0,0,0,0,UP,2,2,0,,0,89074,0,,1,19,0,,191053,,1,0,,100,191128,0,0,0,0,1,0,0,0,0,0,2,,,0,1,678,2059, 3. qmax stays at zero on each backend server, even though qmax(BACKEND) > 0. Are queue stats not maintained for each server ? # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime, backend_c,server_a,0,0,0,3,3,95541,232232183,241897167,,0,,0,0,0,0,UP,1,1,0,512,1,19,1,,95507,backend_a/server_a,2,0,,50,95541,0,0,0,0,01,0,2,,,0,1,706,2028, backend_c,server_b,0,0,0,3,3,95587,232144919,241789273,,0,,0,0,0,0,UP,1,1,0,512,1,19,2,,95546,backend_a/server_b,2,0,,50,95587,0,0,0,0,00,0,3,,,0,2,675,2082, backend_c,BACKEND,0,3,0,9,410,191128,464377102,483686440,0,0,,0,0,0,0,UP,2,2,0,,0,89074,0,,1,19,0,,191053,,1,0,,100,191128,0,0,0,0,1,0,0,0,0,0,2,,,0,1,678,2059, 4. On another HAproxy instance, on a backend with two servers, we always have scur=slim on both servers, but qcur stays at zero. The application servers tell us their threads are ready (zero busy thread). Is there any way to show more info about the sessions which HAproxy thinks are in use ? # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime, backend_a,server_a,0,0,12,12,12,90,114944,68086,,0,,0,0,0,0,UP,1,1,0,18,6,67872,1015,128,1,9,1,,88,,2,0,,8,L7OK,200,5,0,78,0,0,0,0,00,0,501099,OK,,3,1,42,55, backend_a,server_b,0,0,12,12,12,72,87204,47593,,0,,0,0,0,0,UP,1,1,0,25,6,67842,1064,128,1,9,2,,72,,2,0,,7,L7OK,200,4,0,60,0,0,0,0,00,0,501461,OK,,0,1,28,36, backend_a,BACKEND,0,3,24,27,410,181,223139,119707,0,0,,5,0,0,0,UP,2,2,0,,6,67872,862,,1,9,0,,160,,1,0,,150,138,0,0,19,0,5,0,0,0,0,0,501099,,,3,1,66,86, BTW, we are running HAproxy 1.5.9 from the vbernat Ubuntu PPA. Thanks in advance. Sylvain
Send client to a specific backend if header found in previous reply from server
Hello, This is a followup to my post and Baptiste's answer on 20141120. Unfortunately, I couldn't get this to work. I am trying to achieve the following : when a response from the application server contains a header named "X-test", send the following requests from the client IP to another backend. The goal is to send clients who abuse the servers to a slower queue. Here is my current config : In the frontend: stick-table type ip size 50k expire 5m store gpc0 tcp-request content track-sc2 src #use_backend slow if { sc2_get_gpc0 gt 0 } In the backend: acl mark_as_CPUusage sc2_inc_gpc0 gt 0 acl unmark_as_CPUusage sc2_clr_gpc0 gt 0 tcp-response content accept if { res.hdr(X-test) -m found } mark_as_CPUusage #tcp-response content accept if ! { res.hdr(X-test) -m found } unmark_as_CPUusage I used "capture response header" to confirm that the application server sends the "X-test" header for some clients. By querying the stats through unix socket commands, I get gpc0=0 for every client IP, so this doesn't seem to work. Any idea about what I'm missing ? Sylvain.
Send client to a specific backend if header found in previous reply from server
Hi, I am trying to achieve the following : when a response from the application server contains a header named "X-test", send the following requests from the client IP to another backend for 5 minutes. The goal is to send clients who abuse the servers to a slower queue. Here is what I got so far : In the frontend: stick-table type ip size 100k expire 5m store gpc1 tcp-request content track-sc1 src use_backend slow if { sc1_get_gpc1 gt 0 } In the backend: acl mark_as_high_usage sc1_inc_gpc1 gt 0 ??? if res.hdr_cnt(X-test) mark_as_high_usage Does this look good so far ? I am wondering what to use in place of the ???, because no action is to be taken in the backend, this serves only as a way to use the ACL and mark the IP using gpc1 so that the frontend sends its further connections to another backend. Thanks in advance. Sylvain Faivre.