Re: Problem with commit 4448925930655dec57847ed41a34a24a8169d053
Hi Thomas, On Wed, Jan 15, 2014 at 01:57:13PM +0100, Thomas Heil wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, Iam running HAPRoxy with eglibc and commit 4448925930655dec57847ed41a34a24a8169d053, aka BUILD/MINOR: listener: remove a glibc warning on accept4() introduces an fatal error on building. I got it as well yesterday when nuilding for x86_64 and fixed it with commit 2317976 (BUILD: listener: fix recent accept4() again). Thanks, Willy
Difference frontend/backend to listen?
Hi, I got two configurations the should do the same. One is based on a frontend/backend layout the second does it with just listen. The listen configuratiuon is working fine but the fontend/backend causes a problem on the backend. It looks like some request string is missing because the customer application is not able to resolve some lookup which seems to be header related. Is anybody able to tell me whats the difference in these two configurations? global log /dev/loglocal6 #log /dev/log local6 notice chroot /var/lib/haproxy user haproxy group haproxy daemon maxconn 5 stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m defaults log global modehttp option dontlognull option dontlog-normal retries 2 option redispatch timeout connect 5000 timeout client 5 timeout server 12 timeout http-request 5000 timeout http-keep-alive 5000 option http-server-close http-check disable-on-404 http-send-name-header X-Target-Server default-server minconn 1024 maxconn 4096 monitor-net 192.168.xxx.xxx/32 listen application9000 0.0.0.0:9000 balance leastconn mode tcp option tcplog option httpchk GET /check.php HTTP/1.0 http-check expect string Hello World server xcmsphp01.xxx 10.0.4.4:9000 check port 80 server xcmsphp02.xxx 10.0.4.7:9000 check port 80 server xcmsphp03.xxx 10.0.4.3:9000 check port 80 listen application80 0.0.0.0:80 balance roundrobin option forwardfor option httplog monitor-uri /haproxymon option httpchk GET /index.html HTTP/1.1\r\nHost:\ monitoring\r\nConnection:\ close http-check expect string Welcome home acl site_dead nbsrv lt 2 monitor fail if site_dead server xcmsfrontend01.xxx 10.2.2.1:80 check server xcmsfrontend02.xxx 10.2.2.2:80 check global log /dev/loglocal6 #log /dev/log local6 notice chroot /var/lib/haproxy user haproxy group haproxy daemon maxconn 5 stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m defaults log global modehttp option dontlognull option dontlog-normal retries 2 option redispatch timeout connect 5000 timeout client 5 timeout server 12 timeout http-request 5000 timeout http-keep-alive 5000 default-server minconn 1024 maxconn 4096 monitor-net 192.168.xxx.xxx/32 frontend http-in bind *:80 option http-server-close option forwardfor option httplog monitor-uri /haproxymon acl site_dead nbsrv(http-out) lt 2 monitor fail if site_dead default_backend http-out backend http-out balance roundrobin http-send-name-header X-Target-Server option forwardfor option http-server-close #option accept-invalid-http-response http-check disable-on-404 option httpchk GET /index.html HTTP/1.1\r\nHost:\ monitoring\r\nConnection:\ close http-check expect string Welcome home server xcmsfrontend01.xxx 10.2.2.1:80 check server xcmsfrontend02.xxx 10.2.2.2:80 check frontend php-in bind *:9000 mode tcp option tcplog http-send-name-header X-Target-Server default_backend php-out backend php-out balance leastconn mode tcp option tcplog http-send-name-header X-Target-Server option httpchk GET /up.php HTTP/1.0 http-check expect string Hello World server xcmsphp01.xxx 10.0.4.4:9000 check port 80 server xcmsphp02.xxx 10.0.4.7:9000 check port 80 server xcmsphp03.xxx 10.0.4.3:9000 check port 80 Regards, Florian
haproxy-1.5-dev21 and firefox POST (shibboleth-sp) problems
Hi, I'm testing haproxy-1.5-dev19/21 to lb php application(apache/moodle 2.6) moodle is setup to use shibboleth-authentication (https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPConfiguration). Login is all https (client - haproxy - apache+mod_ssl): Login works (haproxy-ss-20131228) ok on IE, chrome and safari, but fails with firefox. When I changed the the connection from haproxy to apache to use http (no https (removed ssl from server line)) I managed to capture packet logs on the apache host. Chrome+haproxy sends whole request to apache (and it works), but with firefox+haproxy apache sees request like this: POST /Shibboleth.sso/SAML2/POST HTTP/1.1 Host: mdl26.uef.fi User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: fi,en-us;q=0.7,en;q=0.3 Accept-Encoding: gzip, deflate Referer: https://idp.uef.fi/idp/profile/SAML2/Redirect/SSO Cookie: __utma=cookie cleanedup; MoodleSessionmdl26=moodle session cookie cleaned; _shibstate_1389859501_c8b0=cookie removed; Content-Type: application/x-www-form-urlencoded Content-Length: 13099 X-Forwarded-For: 10.0.28.54 Connection: close RelayState=cookie%3A1389859501_c8b0SAMLResponse=PD94bWwg ... here's about 7500 bytes from the SAMLResponse mxPREw4bW1QdW9LdzM5V2dxNktiTU8vdWZ2WVlhelpETzh4T2pPawpBVEtlN0szTUtReVh2MS9UdnpDeUFYVzB5S0hMbVBQQlM0ZlRjRG1yZHTTP/1.1 500 Internal Server Error Date: Thu, 16 Jan 2014 08:05:14 GMT Server: Apache Expires: Wed, 01 Jan 1997 12:00:00 GMT Cache-Control: private,no-store,no-cache,max-age=0 Content-Length: 966 Connection: close Content-Type: text/html; charset=UTF-8 --- HERE I removed most of the HTML --- p class=erroropensaml#58;#58;BindingException at (https#58;//mdl26.uef.fi/Shibboleth.sso/SAML2/POST)/p pUnable to decode base64 in POST binding message./p /body /html 05acXphQjI5aXd5Wlk1V1ZkRFc5Y0xEc0 rest of the SAMLResponse So it looks like that firefox+haproxy sends only partial SAMLResponse to apache and sends the rest after apache/shibboleth sp sends back error 500. If I use older 1.5-dev snapshot (haproxy-ss-20131031) then login works on firefox (same config). With latest snapshot haproxy-ss-20140116 the ssl backend doesn't work at all. All requests get 408 error from haproxy. Here's my test config: global log /dev/log local2 info chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 8000 userhaproxy group haproxy daemon spread-checks 3 # turn on stats unix socket stats socket /var/lib/haproxy/stats stats timeout 2m defaults modehttp log global option httplog option dontlognull option redispatch no option accept-invalid-http-request no option accept-invalid-http-response retries 3 timeout http-request10s timeout queue 30s timeout connect 4s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 8192 no option socket-stats frontend http_mdl bind 10.0.0.201:80 name demo3 bind 10.0.0.202:80 name demo4 option tcp-smart-accept option contstats option forwardfor except 127.0.0.0/8 errorfile 403 /etc/haproxy/errors/403.html errorfile 504 /etc/haproxy/errors/504.html default_backend BE_http frontend https_mdl bind 10.0.0.201:443 name demo3ssl ssl crt /etc/haproxy/demo3.pem bind 10.0.0.202:443 name demo4ssl ssl crt /etc/haproxy/demo4.pem option forceclose option contstats option forwardfor except 127.0.0.0/8 errorfile 403 /etc/haproxy/errors/403.html errorfile 504 /etc/haproxy/errors/504.html default_backend BE_http-ssl backend BE_http balance roundrobin option tcp-smart-connect option httpchk HEAD / HTTP/1.0\r\n http-check disable-on-404 no option persist stats enable stats realm removed stats auth info:removed stats uri /statsuri #removed stats hide-version stats refresh 120s default-server inter 5s downinter 15s rise 2 error-limit 10 fall 2 on-error fail-check server mdldemo1 10.0.0.151:80 check observe layer7 server mdldemo2 10.0.0.152:80 check observe layer7 backend BE_http-ssl option forceclose timeout server 120s # long timeout to receive shib error w/firefox default-server inter 15s downinter 20s rise 2 server mdldemo1 10.0.0.151:443 maxconn 256 ssl force-sslv3 check # second server disabled for testing (also stick on src removed for # testing). #server mdldemo2 10.0.0.152:443 maxconn 256 ssl force-sslv3 check
URL path in backend servers
Hello, Is it possible to forward an incoming request to the *backend *by retaining the URL path in *http *mode?. Using *ACLs *I was able to categorize the incoming requests to different rulesets, but on using a specific *backend *for a certain URL parth, I could not figure out how to send the request to the underlying server *with *the URL path?. I figured out how to redirect a request, but that is not what I wanted. So, if HAProxy receives a request on, say *http://192.168.5.10:80/url/path http://192.168.5.10/url/path* , I want the request to be sent to the underlying server as*http://133.168.5.20:80/url/path http://133.168.5.20/url/path.* Does anyone know how to do this?. Any help is much appreciated!. PS: Question on ServerFault : http://serverfault.com/questions/566862/haproxy-url-pattern-forwarding Thanks, Rakesh
Re: URL path in backend servers
Rakesh - I replied to your identical question about this, yesterday, suggesting what you could do to help yourself diagnose your problem. Please don't start new threads for the same question. Jonathan
How to test HAProxy ?
Hi I'm involved in testing the port of HAProxy 1.4 (.24) on Ubuntu on PPC/LE. For now, I've found nothing about how to check that the basic features and the important features of HAProxy do work fine in this environment. The README file says nothing about testing. The Makefile file provides no test or tests entry. The examples directory does not contain explanations and instructions for testing. The tests directory does not contain explanations and instructions for testing. I do not have time to get skills about HAProxy or about any tool that makes use of HAProxy. I simply need to get configuration and test instructions. What do you suggest/recommend me to do for testing the most important features of HAProxy, in 1 week, knowing that I know nothing about HAProxy ? Thanks Tony
Re: Difference frontend/backend to listen?
Hi Florian, Found a a minor difference, not sure if it is the issue.? - The 9000 backend checks up.php versus check.php. - Also I don't think http-send-name-header does anything in 'tcp mode'.. If thats not it, maybe someone else has a clue. :) p.s. You might want to configure a stats page to see if servers are properly checked as 'up' by haproxy. Greets PiBa-NL Florian Engelmann schreef op 16-1-2014 12:29: Hi, I got two configurations the should do the same. One is based on a frontend/backend layout the second does it with just listen. The listen configuratiuon is working fine but the fontend/backend causes a problem on the backend. It looks like some request string is missing because the customer application is not able to resolve some lookup which seems to be header related. Is anybody able to tell me whats the difference in these two configurations? global log /dev/loglocal6 #log /dev/loglocal6 notice chroot /var/lib/haproxy user haproxy group haproxy daemon maxconn 5 stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m defaults log global modehttp option dontlognull option dontlog-normal retries 2 option redispatch timeout connect 5000 timeout client 5 timeout server 12 timeout http-request 5000 timeout http-keep-alive 5000 option http-server-close http-check disable-on-404 http-send-name-header X-Target-Server default-server minconn 1024 maxconn 4096 monitor-net 192.168.xxx.xxx/32 listen application9000 0.0.0.0:9000 balance leastconn mode tcp option tcplog option httpchk GET /check.php HTTP/1.0 http-check expect string Hello World server xcmsphp01.xxx 10.0.4.4:9000 check port 80 server xcmsphp02.xxx 10.0.4.7:9000 check port 80 server xcmsphp03.xxx 10.0.4.3:9000 check port 80 listen application80 0.0.0.0:80 balance roundrobin option forwardfor option httplog monitor-uri /haproxymon option httpchk GET /index.html HTTP/1.1\r\nHost:\ monitoring\r\nConnection:\ close http-check expect string Welcome home acl site_dead nbsrv lt 2 monitor fail if site_dead server xcmsfrontend01.xxx 10.2.2.1:80 check server xcmsfrontend02.xxx 10.2.2.2:80 check global log /dev/loglocal6 #log /dev/loglocal6 notice chroot /var/lib/haproxy user haproxy group haproxy daemon maxconn 5 stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m defaults log global modehttp option dontlognull option dontlog-normal retries 2 option redispatch timeout connect 5000 timeout client 5 timeout server 12 timeout http-request 5000 timeout http-keep-alive 5000 default-server minconn 1024 maxconn 4096 monitor-net 192.168.xxx.xxx/32 frontend http-in bind *:80 option http-server-close option forwardfor option httplog monitor-uri /haproxymon acl site_dead nbsrv(http-out) lt 2 monitor fail if site_dead default_backend http-out backend http-out balance roundrobin http-send-name-header X-Target-Server option forwardfor option http-server-close #option accept-invalid-http-response http-check disable-on-404 option httpchk GET /index.html HTTP/1.1\r\nHost:\ monitoring\r\nConnection:\ close http-check expect string Welcome home server xcmsfrontend01.xxx 10.2.2.1:80 check server xcmsfrontend02.xxx 10.2.2.2:80 check frontend php-in bind *:9000 mode tcp option tcplog http-send-name-header X-Target-Server Wont work in tcpmode default_backend php-out backend php-out balance leastconn mode tcp option tcplog http-send-name-header X-Target-Server Wont work for tcp mode. option httpchk GET /up.php HTTP/1.0 Should this be check.php v.s. up.php? http-check expect string Hello World server xcmsphp01.xxx 10.0.4.4:9000 check port 80 server xcmsphp02.xxx 10.0.4.7:9000 check port 80 server xcmsphp03.xxx 10.0.4.3:9000 check port 80 Regards, Florian
issue with acl pattern -m match on a string starting with space or containing a comma, with 1.5-dev21
Hi, Using HAProxy 1.5-dev21 i'm having trouble getting it to match my user-agent with an acl that uses -m pattern matching.. The browser is Chrome 31.0.1650.63 which sends useragent string: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36 My test ACLs, of which only ACL21 and ACL31 are matched with the result below: *ACLexact*= A *ACLbeg*= B, 1 *ACLend*= C, 1 I would expect at least 2 the ACLbeg acls and ACL2 to be also matched, also i dont understand why ACL32 is not matched as the leading space seems to be correctly escaped.? The acl's used/tried..: reqadd ACLexact:\ A reqadd ACLbeg:\ B reqadd ACLend:\ C acl ACL1 hdr(User-Agent) Mozilla/5.0\ (Windows\ NT\ 6.1;\ WOW64)\ AppleWebKit/537.36\ (KHTML\,\ like\ Gecko)\ Chrome/32.0.1700.76\ Safari/537.36 reqadd ACLexact:\ 1 if ACL1 acl ACL2 hdr(User-Agent) -m str Mozilla/5.0\ (Windows\ NT\ 6.1;\ WOW64)\ AppleWebKit/537.36\ (KHTML\,\ like\ Gecko)\ Chrome/32.0.1700.76\ Safari/537.36 reqadd ACLexact:\ 2 if ACL2 acl ACL21 hdr(User-Agent) -m beg Mozilla/5.0\ (Windows\ NT\ 6.1;\ WOW64)\ AppleWebKit/537.36\ (KHTML reqadd ACLbeg:\ 1 if ACL21 acl ACL22 hdr(User-Agent) -m beg Mozilla/5.0\ (Windows\ NT\ 6.1;\ WOW64)\ AppleWebKit/537.36\ (KHTML, reqadd ACLbeg:\ 2 if ACL22 acl ACL23 hdr(User-Agent) -m beg Mozilla/5.0\ (Windows\ NT\ 6.1;\ WOW64)\ AppleWebKit/537.36\ (KHTML\, reqadd ACLbeg:\ 3 if ACL23 acl ACL31 hdr(User-Agent) -m end like\ Gecko)\ Chrome/32.0.1700.76\ Safari/537.36 reqadd ACLend:\ 1 if ACL31 acl ACL32 hdr(User-Agent) -m end \ like\ Gecko)\ Chrome/32.0.1700.76\ Safari/537.36 reqadd ACLend:\ 2 if ACL32 acl ACL33 hdr(User-Agent) -m end ,\ like\ Gecko)\ Chrome/32.0.1700.76\ Safari/537.36 reqadd ACLend:\ 3 if ACL33 acl ACL34 hdr(User-Agent) -m end \,\ like\ Gecko)\ Chrome/32.0.1700.76\ Safari/537.36 reqadd ACLend:\ 4 if ACL34 HAPROXY Version used: HA-Proxy version 1.5-dev21-6b07bf7 +2013/12/17 Copyright 2000-2013 Willy Tarreau w...@1wt.eu Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -O2 -pipe -fno-strict-aliasing -DFREEBSD_PORTS OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 Did i do something wrong, or can you give it a test.? Thanks. Thanks for the great product! Greets PiBa-NL
Re: Bug report for latest dev release, 1.5.21, segfault when using http expect string x and large 404 page (includes GDB output)
Hi Willy,
Le 15/01/2014 01:08, Willy Tarreau a écrit :
On Tue, Jan 14, 2014 at 12:25:37PM -0800, Steve Ruiz wrote:
Patched and confirmed in our environment that this is now working / seems
to have fixed the issue. Thanks!
Great, many thanks to you both guys. We've got rid of another pretty
old bug, these are the ones that make me the happiest once fixed!
I'm currently unpacking my laptop to push the fix so that it appears
in todays snapshot.
Excellent work!
I fear there are some more work to do on this patch.
I made some tests on ssl and it looks to be broken since this commit :-(
The shortest configuration I could find to reproduce the issue is :
listen test
bind 0.0.0.0:443 ssl crt cert.pem
mode http
timeout server 5s
timeout client 5s
When a request is received by haproxy, the cpu raises to 100% in a
epoll_wait loop (timeouts are here to prevent an unlimited loop).
$ curl -k https://localhost/
...
epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 200, 0) = 1
epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 200, 0) = 1
epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 200, 0) = 1
epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 200, 0) = 1
epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 200, 0) = 1
...
The same issue occurs when a server is declared.
The same also occurs when the proxy is in clear http and a server is in
https :
listen test
bind 0.0.0.0:80
mode http
timeout server 5s
timeout client 5s
server ssl_backend 127.0.0.1:443 ssl
$ curl http://localhost/
...
epoll_wait(3, {}, 200, 0) = 0
epoll_wait(3, {}, 200, 0) = 0
epoll_wait(3, {}, 200, 0) = 0
epoll_wait(3, {}, 200, 0) = 0
epoll_wait(3, {}, 200, 0) = 0
...
--
Cyril Bonté
optimizing TLS time to first byte
Hey all. I've spent some time looking into HAProxy (1.5-dev21) + TLS performance and stumbled across a few areas where I think we could make some improvements. In particular I'm interested in time to first byte, as that's a critical piece for interactive traffic: time to first paint in browsers, time to first frame for video, etc. - (1) Certificates that exceed 4KB require an extra RTT even with IW10: HA ships the first 4KB then pauses and waits for client ACK before proceeding to send remainder of the certificate. At a minimum, this results in an extra handshake RTT. You can see this in action here: WPT run: *http://www.webpagetest.org/result/140116_VW_3bd95a5cfb7e667498ef13b59639b9bf/2/details/ http://www.webpagetest.org/result/140116_VW_3bd95a5cfb7e667498ef13b59639b9bf/2/details/* tcpdump: http://www.webpagetest.org/result/140116_VW_3bd95a5cfb7e667498ef13b59639b9bf/2.cap I believe this is the same exact issue as fixed in nginx here: https://github.com/nginx/nginx/commit/e52bddaaa90e64b2291f6e58ef1a2cff71604f6a#diff-0584d16332cf0d6dd9adb990a3c76a0cR539 - (2) HA allows you to tune max_record_size - yay. That said, using a static value introduces an inherent tradeoff between latency and throughput -- smaller records are good for latency, but hurt server throughput by adding bytes and CPU overhead. It would be great if we could implement a smarter strategy: http://mailman.nginx.org/pipermail/nginx-devel/2013-December/004703.html http://mailman.nginx.org/pipermail/nginx-devel/2014-January/004748.html Would love to hear any thoughts on this. The advantage of above strategy is that it can give you good performance out-of-the-box and without sacrificing performance for different kinds of traffic (which is especially relevant for a proxy...). Ilya
Re: Bug report for latest dev release, 1.5.21, segfault when using http expect string x and large 404 page (includes GDB output)
Cyril is correct - I simply waited for a segfault, but didn't actually test
through the load balancer. I'm using SSL on haproxy, and yes, when I try to
hit a web page behind haproxy, CPU spins at 100% for a good while.
Steve Ruiz
Manager - Hosting Operations
Mirth
ste...@mirth.com ste...@mirthcorp.com
On Thu, Jan 16, 2014 at 1:48 PM, Cyril Bonté cyril.bo...@free.fr wrote:
Hi Willy,
Le 15/01/2014 01:08, Willy Tarreau a écrit :
On Tue, Jan 14, 2014 at 12:25:37PM -0800, Steve Ruiz wrote:
Patched and confirmed in our environment that this is now working / seems
to have fixed the issue. Thanks!
Great, many thanks to you both guys. We've got rid of another pretty
old bug, these are the ones that make me the happiest once fixed!
I'm currently unpacking my laptop to push the fix so that it appears
in todays snapshot.
Excellent work!
I fear there are some more work to do on this patch.
I made some tests on ssl and it looks to be broken since this commit :-(
The shortest configuration I could find to reproduce the issue is :
listen test
bind 0.0.0.0:443 ssl crt cert.pem
mode http
timeout server 5s
timeout client 5s
When a request is received by haproxy, the cpu raises to 100% in a
epoll_wait loop (timeouts are here to prevent an unlimited loop).
$ curl -k https://localhost/
...
epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 200, 0) = 1
epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 200, 0) = 1
epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 200, 0) = 1
epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 200, 0) = 1
epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 200, 0) = 1
...
The same issue occurs when a server is declared.
The same also occurs when the proxy is in clear http and a server is in
https :
listen test
bind 0.0.0.0:80
mode http
timeout server 5s
timeout client 5s
server ssl_backend 127.0.0.1:443 ssl
$ curl http://localhost/
...
epoll_wait(3, {}, 200, 0) = 0
epoll_wait(3, {}, 200, 0) = 0
epoll_wait(3, {}, 200, 0) = 0
epoll_wait(3, {}, 200, 0) = 0
epoll_wait(3, {}, 200, 0) = 0
...
--
Cyril Bonté
--
CONFIDENTIALITY NOTICE: The information contained in this electronic
transmission may be confidential. If you are not an intended recipient, be
aware that any disclosure, copying, distribution or use of the information
contained in this transmission is prohibited and may be unlawful. If you
have received this transmission in error, please notify us by email reply
and then erase it from your computer system.
sFlow patch for upstream
Hi, Base on original sFlow patch: https://github.com/sflow/haproxy/commit/b0058a4b344bb61d05182db291f76453eaaea301 I've made patch for patch :) to work with upstream HAProxy version. https://github.com/ljagiello/haproxy/commit/1a93d1ff693388dd96a0428fc28e43dae8ee4267 If anyone is also using it. Cheers -- Łukasz Jagiełło lukaszatjagiellodotorg
how to force curernt session when active is down
Hi, I am pretty new to haproxy and trying to do ha setup for rabbitmq. Setup is pretyt simple: listen rabbitmq 192.168.69.106:5672 modetcp balance leastconn option tcplog option tcpka server rabbit01 192.168.69.107:5672 check inter 1000 downinter 5000 fall 1 on-error mark-down server rabbit02 192.168.69.108:5672 check inter 1000 downinter 5000 fall 1 on-error mark-down backup Everything seems to work well, but I noticed one strange behaviour. If I gracefully shut down rabbit01, message producer (client) connected to 192.168.69.106:5672 notices broken connection pretty much straight away, attempts a reconnect, hits rabbit02 and all is well. However if I simply turn off rabbit01, message producer (client) notices broken connection only in around 30 seconds or so. And while the break is undetected it keeps pushing messages thinking there is still someone on another side. Now, from the statistic report I can see that as soon as I turn off the box, rabbit01 is marked as DOWN, but its Sessions Current is still set to 1? In the logs this comes up pretty much immediately after the hard turn off: Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is DOWN, reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 backup servers left. Running on backup. 1 sessions active, 0 requeued, 0 remaining in queue. After around 30 seconds, the connection is detected as broken and failover happens successfully. At that time Sessions Current is set to 0 fo rabbit01. Tried option nolinger and option abortonclose, but no luck. Any help would be much appreciated... Cheers, Andrei
Re: how to force curernt session when active is down
Hi! That's related to rabbiit's default heartbeat timeout. You can set it to lower value when connecting. Also there is a way to check if message was actually delivered to the broker, take a look at rabbiit's docs. пятница, 17 января 2014 г. пользователь Andrei Chevenkov написал: Hi, I am pretty new to haproxy and trying to do ha setup for rabbitmq. Setup is pretyt simple: listen rabbitmq 192.168.69.106:5672 modetcp balance leastconn option tcplog option tcpka server rabbit01 192.168.69.107:5672 check inter 1000 downinter 5000 fall 1 on-error mark-down server rabbit02 192.168.69.108:5672 check inter 1000 downinter 5000 fall 1 on-error mark-down backup Everything seems to work well, but I noticed one strange behaviour. If I gracefully shut down rabbit01, message producer (client) connected to 192.168.69.106:5672 notices broken connection pretty much straight away, attempts a reconnect, hits rabbit02 and all is well. However if I simply turn off rabbit01, message producer (client) notices broken connection only in around 30 seconds or so. And while the break is undetected it keeps pushing messages thinking there is still someone on another side. Now, from the statistic report I can see that as soon as I turn off the box, rabbit01 is marked as DOWN, but its Sessions Current is still set to 1? In the logs this comes up pretty much immediately after the hard turn off: Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is DOWN, reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 backup servers left. Running on backup. 1 sessions active, 0 requeued, 0 remaining in queue. After around 30 seconds, the connection is detected as broken and failover happens successfully. At that time Sessions Current is set to 0 fo rabbit01. Tried option nolinger and option abortonclose, but no luck. Any help would be much appreciated... Cheers, Andrei
Re: how to force curernt session when active is down
Dmitry, thank you for the reply, but I would imagine that haproxy would close all sessions on the DOWN nodes, regardless of the type of client connecting and the protocol? Can this be enforced? I did implement producer ack, but that slows down publishes big time. Have also tried heartbeat and that did help, but this is initiated by the client and I would like to see if haproxy can manage this. I.e. close all sessions on a node that is marked as DOWN. On 17/01/2014 8:09 pm, Dmitriy Samsonov dmitriy.samso...@gmail.com wrote: Hi! That's related to rabbiit's default heartbeat timeout. You can set it to lower value when connecting. Also there is a way to check if message was actually delivered to the broker, take a look at rabbiit's docs. пятница, 17 января 2014 г. пользователь Andrei Chevenkov написал: Hi, I am pretty new to haproxy and trying to do ha setup for rabbitmq. Setup is pretyt simple: listen rabbitmq 192.168.69.106:5672 modetcp balance leastconn option tcplog option tcpka server rabbit01 192.168.69.107:5672 check inter 1000 downinter 5000 fall 1 on-error mark-down server rabbit02 192.168.69.108:5672 check inter 1000 downinter 5000 fall 1 on-error mark-down backup Everything seems to work well, but I noticed one strange behaviour. If I gracefully shut down rabbit01, message producer (client) connected to 192.168.69.106:5672 notices broken connection pretty much straight away, attempts a reconnect, hits rabbit02 and all is well. However if I simply turn off rabbit01, message producer (client) notices broken connection only in around 30 seconds or so. And while the break is undetected it keeps pushing messages thinking there is still someone on another side. Now, from the statistic report I can see that as soon as I turn off the box, rabbit01 is marked as DOWN, but its Sessions Current is still set to 1? In the logs this comes up pretty much immediately after the hard turn off: Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is DOWN, reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 backup servers left. Running on backup. 1 sessions active, 0 requeued, 0 remaining in queue. After around 30 seconds, the connection is detected as broken and failover happens successfully. At that time Sessions Current is set to 0 fo rabbit01. Tried option nolinger and option abortonclose, but no luck. Any help would be much appreciated... Cheers, Andrei
