Re: Geotargeting and Server DOWN problem
Thank you for your answer. Sorry Cyril I've missed the messeges between my textlines somehow, no offense, just plain laziness. Should have read the complete text before I asked for further advice. I'll check your hints now and will reconfigure my system hope it helps. Thank you again. Best regards Sebastian On 30.01.2012 07:20, Baptiste wrote: Hi Sebastian, I think Cyril's email is full of advices for you, explaining you where your problem may be. Basically you have several options for your configuration. Either you turn all your listeneners to tcp, removing the http health checks, HAProxy will do basic tcp. Or you can also splitting your configuration based on the frontend port: http vs https. If you need affinity per user on http and https, you can use this configuration example: http://blog.exceliance.fr/2011/07/12/send-user-to-the-same-backend-for-both-http-and-https/ And the second example below will show you how to do persistence based on the SSL id: http://blog.exceliance.fr/2011/07/04/maintain-affinity-based-on-ssl-session-id/ cheers On Mon, Jan 30, 2012 at 1:14 AM, Sebastian Fohleri...@far-galaxy.com wrote: Where is the problem? As I already said before, it's my first haproxy config. I only used hardware loadbalancing before. Any help would be appreciated. Best regards Sebastian On 29.01.2012 23:27, Cyril Bonté wrote: Hi Sebastian, Is this really your configuration ? I'm afraid to say that it can't work the way it is done (or I missed something). For those reasons : Le 29/01/2012 14:09, Sebastian Fohler a écrit : Sure no problem, here is my config: the points the points # cat haproxy.cfg global maxconn 9 log 127.0.0.1 local0 uid 80 gid 80 nbproc 1 chroot /var/empty daemon listen adserver.adworxs.local bind 192.168.10.68:80 bind 192.168.10.68:443 mode http First of all, haproxy doesn't understand HTTPS natively. I guess that if you bind on port 443 this is because you want haproxy to receive HTTPS, but it can't. If you want to use haproxy in HTTP mode, you'll have to decode HTTPS into plain text HTTP with a ssl termination such as stunnel or stud. = Don't bind on port 443 with mode http enabled. log global option dontlognull option httpclose option forwardfor maxconn 1 clitimeout 3 balance roundrobin contimeout 3 srvtimeout 3 retries 10 option httpchk HEAD /www/admin/index.php HTTP/1.0 stats enable stats uri /haproxy?stats stats realm adserver.adworxs.local stats auth admin:password stats show-node stats refresh s server adserver1 192.168.10.61:80 check inter 1000 weight 10 server adserver1 192.168.10.61:443 check inter 1000 weight 10 Here, several issues occur : - You shouldn't use several times the same server name, this is maybe what made your diagnostics difficult in the stats page and in the logs. (In your first post, you said : The first problem I had was, while checking the haproxy stats, that they show every backendserver is at least the same time DOWN as it is UP) - You've mixed HTTP servers and HTTPS ones in the backend : haproxy will load balance the incoming connections on them (in round robin as you've configured it some lines before). Those connecting to port 443 will probably be always DOWN so you'll think it works if you only send HTTP requests. But as soon as you try to send HTTPS requests, you'll get in trouble. - You can't use check on your https server because you've configured option httpchk : haproxy will try to send plain text HTTP where the server is waiting for HTTPS. server adserver2 192.168.10.62:80 check inter 1000 weight 10 server adserver2 192.168.10.62:443 check inter 1000 weight 10 server adserver4 192.168.10.67:80 check inter 1000 weight 120 server adserver4 192.168.10.67:443 check inter 1000 weight 120 server adserver5 192.168.10.66:80 check inter 1000 weight 120 server adserver5 192.168.10.66:443 check inter 1000 weight 120 ...same issues for each adserver listen adserver.dev.adworxs.local bind 192.168.10.70:80 bind 192.168.10.70:443 mode http log global option dontlognull option httpclose option forwardfor maxconn 6000 clitimeout 3 balance leastconn contimeout 3 srvtimeout 3 retries 10 option httpchk HEAD /www/admin/index.php HTTP/1.0 stats enable stats uri /haproxy?stats stats realm adserver.dev.adworxs.local stats auth admin:password stats show-node stats refresh m server adserver1.dev 192.168.10.41:80 check inter 10 weight 100 server adserver1.dev 192.168.10.41:443 check inter 10 weight 100 and same issues for this proxy. I'd suggest you to remove every lines containing a reference to port 443 (binds and servers) and add stunnel or stud in front of haproxy, configured to receive incoming HTTPS request on port 443 and forward the plain text HTTP request to haproxy on port 80. Hope this helps. -- Sebastian Fohler Far-Galaxy Networks Web: http://www.far-galaxy.de Tel.: +49 (0)173 8796211 E-Mail: i...@far-galaxy.de
Re: Geotargeting and Server DOWN problem
Sure no problem, here is my config: # cat haproxy.cfg global maxconn 9 log 127.0.0.1 local0 uid 80 gid 80 nbproc 1 chroot /var/empty daemon listen adserver.adworxs.local bind192.168.10.68:80 bind192.168.10.68:443 modehttp log global option dontlognull option httpclose option forwardfor maxconn 1 clitimeout 3 balance roundrobin contimeout 3 srvtimeout 3 retries 10 option httpchk HEAD /www/admin/index.php HTTP/1.0 stats enable stats uri /haproxy?stats stats realm adserver.adworxs.local stats auth admin:password stats show-node stats refresh s server adserver1 192.168.10.61:80 check inter 1000 weight 10 server adserver1 192.168.10.61:443 check inter 1000 weight 10 server adserver2 192.168.10.62:80 check inter 1000 weight 10 server adserver2 192.168.10.62:443 check inter 1000 weight 10 server adserver4 192.168.10.67:80 check inter 1000 weight 120 server adserver4 192.168.10.67:443 check inter 1000 weight 120 server adserver5 192.168.10.66:80 check inter 1000 weight 120 server adserver5 192.168.10.66:443 check inter 1000 weight 120 listen adserver.dev.adworxs.local bind192.168.10.70:80 bind192.168.10.70:443 modehttp log global option dontlognull option httpclose option forwardfor maxconn 6000 clitimeout 3 balance leastconn contimeout 3 srvtimeout 3 retries 10 option httpchk HEAD /www/admin/index.php HTTP/1.0 stats enable stats uri /haproxy?stats stats realm adserver.dev.adworxs.local stats auth admin:password stats show-node stats refresh m server adserver1.dev 192.168.10.41:80 check inter 10 weight 100 server adserver1.dev 192.168.10.41:443 check inter 10 weight 100 Thank you in advance. Best regards Sebastian On 27.01.2012 20:54, Sander Klein wrote: On 27.01.2012 16:01, Sebastian Fohler wrote: Sorry just found out that I definitly do an active check. But for some reason every second refresh of my stats shows the servers down. Any idea why that could be? The servers are definitly up all that time. Hmz, I don't know. It think it's helpful if you post more info like your haproxy config. Greets, Sander
Re: Geotargeting and Server DOWN problem
Another thing which might help to understand the problem: Jan 29 14:24:57 haproxy[61441]: Server adserver.adworxs.local/adserver2 is DOWN, reason: Socket error, info: Operation not permitted, check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jan 29 14:25:07 haproxy[61441]: Server adserver.adworxs.local/adserver2 is UP, reason: Layer7 check passed, code: 200, info: OK, check duration: 610ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Jan 29 14:25:11 haproxy[61441]: Server adserver.adworxs.local/adserver2 is DOWN, reason: Layer7 timeout, check duration: 1009ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jan 29 14:25:18 haproxy[61441]: Server adserver.adworxs.local/adserver2 is UP, reason: Layer7 check passed, code: 200, info: OK, check duration: 341ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Jan 29 14:25:34 haproxy[61441]: Server adserver.adworxs.local/adserver2 is DOWN, reason: Socket error, info: Operation not permitted, check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jan 29 14:26:03 haproxy[61441]: Server adserver.adworxs.local/adserver4 is UP, reason: Layer7 check passed, code: 200, info: OK, check duration: 100ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Jan 29 14:26:03 haproxy[61441]: Server adserver.adworxs.local/adserver5 is UP, reason: Layer7 check passed, code: 200, info: OK, check duration: 144ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Jan 29 14:26:04 haproxy[61441]: Server adserver.adworxs.local/adserver2 is UP, reason: Layer7 check passed, code: 200, info: OK, check duration: 553ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Jan 29 14:26:15 haproxy[61441]: Server adserver.adworxs.local/adserver2 is DOWN, reason: Layer7 timeout, check duration: 1019ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jan 29 14:26:26 haproxy[61441]: Server adserver.adworxs.local/adserver4 is DOWN, reason: Socket error, info: Operation not permitted, check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jan 29 14:26:26 haproxy[61441]: Server adserver.adworxs.local/adserver5 is DOWN, reason: Socket error, info: Operation not permitted, check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jan 29 14:26:26 haproxy[61441]: proxy adserver.adworxs.local has no server available! Jan 29 14:26:28 haproxy[61441]: Server adserver.adworxs.local/adserver4 is UP, reason: Layer7 check passed, code: 200, info: OK, check duration: 152ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Jan 29 14:26:28 haproxy[61441]: Server adserver.adworxs.local/adserver5 is UP, reason: Layer7 check passed, code: 200, info: OK, check duration: 235ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Best regards Sebastian On 27.01.2012 20:54, Sander Klein wrote: On 27.01.2012 16:01, Sebastian Fohler wrote: Sorry just found out that I definitly do an active check. But for some reason every second refresh of my stats shows the servers down. Any idea why that could be? The servers are definitly up all that time. Hmz, I don't know. It think it's helpful if you post more info like your haproxy config. Greets, Sander
Re: Geotargeting and Server DOWN problem
Hi Sebastian, On Sun, Jan 29, 2012 at 02:09:57PM +0100, Sebastian Fohler wrote: server adserver1.dev 192.168.10.41:80 check inter 10 weight 100 server adserver1.dev 192.168.10.41:443 check inter 10 weight 100 The check intervals are in milliseconds, you're hamering your servers by checking them 100 times a second and only allowing them 10 ms to respond. So when they're up, some traffic is sent, their response time increases and next check sees them down, so the traffic is not sent anymore and the response time falls below 10 ms and this is doing that in loops. Regards, Willy
Re: Geotargeting and Server DOWN problem
Hi Sebastian, Is this really your configuration ? I'm afraid to say that it can't work the way it is done (or I missed something). For those reasons : Le 29/01/2012 14:09, Sebastian Fohler a écrit : Sure no problem, here is my config: the points the points # cat haproxy.cfg global maxconn 9 log 127.0.0.1 local0 uid 80 gid 80 nbproc 1 chroot /var/empty daemon listen adserver.adworxs.local bind 192.168.10.68:80 bind 192.168.10.68:443 mode http First of all, haproxy doesn't understand HTTPS natively. I guess that if you bind on port 443 this is because you want haproxy to receive HTTPS, but it can't. If you want to use haproxy in HTTP mode, you'll have to decode HTTPS into plain text HTTP with a ssl termination such as stunnel or stud. = Don't bind on port 443 with mode http enabled. log global option dontlognull option httpclose option forwardfor maxconn 1 clitimeout 3 balance roundrobin contimeout 3 srvtimeout 3 retries 10 option httpchk HEAD /www/admin/index.php HTTP/1.0 stats enable stats uri /haproxy?stats stats realm adserver.adworxs.local stats auth admin:password stats show-node stats refresh s server adserver1 192.168.10.61:80 check inter 1000 weight 10 server adserver1 192.168.10.61:443 check inter 1000 weight 10 Here, several issues occur : - You shouldn't use several times the same server name, this is maybe what made your diagnostics difficult in the stats page and in the logs. (In your first post, you said : The first problem I had was, while checking the haproxy stats, that they show every backendserver is at least the same time DOWN as it is UP) - You've mixed HTTP servers and HTTPS ones in the backend : haproxy will load balance the incoming connections on them (in round robin as you've configured it some lines before). Those connecting to port 443 will probably be always DOWN so you'll think it works if you only send HTTP requests. But as soon as you try to send HTTPS requests, you'll get in trouble. - You can't use check on your https server because you've configured option httpchk : haproxy will try to send plain text HTTP where the server is waiting for HTTPS. server adserver2 192.168.10.62:80 check inter 1000 weight 10 server adserver2 192.168.10.62:443 check inter 1000 weight 10 server adserver4 192.168.10.67:80 check inter 1000 weight 120 server adserver4 192.168.10.67:443 check inter 1000 weight 120 server adserver5 192.168.10.66:80 check inter 1000 weight 120 server adserver5 192.168.10.66:443 check inter 1000 weight 120 ...same issues for each adserver listen adserver.dev.adworxs.local bind 192.168.10.70:80 bind 192.168.10.70:443 mode http log global option dontlognull option httpclose option forwardfor maxconn 6000 clitimeout 3 balance leastconn contimeout 3 srvtimeout 3 retries 10 option httpchk HEAD /www/admin/index.php HTTP/1.0 stats enable stats uri /haproxy?stats stats realm adserver.dev.adworxs.local stats auth admin:password stats show-node stats refresh m server adserver1.dev 192.168.10.41:80 check inter 10 weight 100 server adserver1.dev 192.168.10.41:443 check inter 10 weight 100 and same issues for this proxy. I'd suggest you to remove every lines containing a reference to port 443 (binds and servers) and add stunnel or stud in front of haproxy, configured to receive incoming HTTPS request on port 443 and forward the plain text HTTP request to haproxy on port 80. Hope this helps. -- Cyril Bonté
Re: Geotargeting and Server DOWN problem
Where is the problem? As I already said before, it's my first haproxy config. I only used hardware loadbalancing before. Any help would be appreciated. Best regards Sebastian On 29.01.2012 23:27, Cyril Bonté wrote: Hi Sebastian, Is this really your configuration ? I'm afraid to say that it can't work the way it is done (or I missed something). For those reasons : Le 29/01/2012 14:09, Sebastian Fohler a écrit : Sure no problem, here is my config: the points the points # cat haproxy.cfg global maxconn 9 log 127.0.0.1 local0 uid 80 gid 80 nbproc 1 chroot /var/empty daemon listen adserver.adworxs.local bind 192.168.10.68:80 bind 192.168.10.68:443 mode http First of all, haproxy doesn't understand HTTPS natively. I guess that if you bind on port 443 this is because you want haproxy to receive HTTPS, but it can't. If you want to use haproxy in HTTP mode, you'll have to decode HTTPS into plain text HTTP with a ssl termination such as stunnel or stud. = Don't bind on port 443 with mode http enabled. log global option dontlognull option httpclose option forwardfor maxconn 1 clitimeout 3 balance roundrobin contimeout 3 srvtimeout 3 retries 10 option httpchk HEAD /www/admin/index.php HTTP/1.0 stats enable stats uri /haproxy?stats stats realm adserver.adworxs.local stats auth admin:password stats show-node stats refresh s server adserver1 192.168.10.61:80 check inter 1000 weight 10 server adserver1 192.168.10.61:443 check inter 1000 weight 10 Here, several issues occur : - You shouldn't use several times the same server name, this is maybe what made your diagnostics difficult in the stats page and in the logs. (In your first post, you said : The first problem I had was, while checking the haproxy stats, that they show every backendserver is at least the same time DOWN as it is UP) - You've mixed HTTP servers and HTTPS ones in the backend : haproxy will load balance the incoming connections on them (in round robin as you've configured it some lines before). Those connecting to port 443 will probably be always DOWN so you'll think it works if you only send HTTP requests. But as soon as you try to send HTTPS requests, you'll get in trouble. - You can't use check on your https server because you've configured option httpchk : haproxy will try to send plain text HTTP where the server is waiting for HTTPS. server adserver2 192.168.10.62:80 check inter 1000 weight 10 server adserver2 192.168.10.62:443 check inter 1000 weight 10 server adserver4 192.168.10.67:80 check inter 1000 weight 120 server adserver4 192.168.10.67:443 check inter 1000 weight 120 server adserver5 192.168.10.66:80 check inter 1000 weight 120 server adserver5 192.168.10.66:443 check inter 1000 weight 120 ...same issues for each adserver listen adserver.dev.adworxs.local bind 192.168.10.70:80 bind 192.168.10.70:443 mode http log global option dontlognull option httpclose option forwardfor maxconn 6000 clitimeout 3 balance leastconn contimeout 3 srvtimeout 3 retries 10 option httpchk HEAD /www/admin/index.php HTTP/1.0 stats enable stats uri /haproxy?stats stats realm adserver.dev.adworxs.local stats auth admin:password stats show-node stats refresh m server adserver1.dev 192.168.10.41:80 check inter 10 weight 100 server adserver1.dev 192.168.10.41:443 check inter 10 weight 100 and same issues for this proxy. I'd suggest you to remove every lines containing a reference to port 443 (binds and servers) and add stunnel or stud in front of haproxy, configured to receive incoming HTTPS request on port 443 and forward the plain text HTTP request to haproxy on port 80. Hope this helps.
Re: Geotargeting and Server DOWN problem
Hi Sebastian, I think Cyril's email is full of advices for you, explaining you where your problem may be. Basically you have several options for your configuration. Either you turn all your listeneners to tcp, removing the http health checks, HAProxy will do basic tcp. Or you can also splitting your configuration based on the frontend port: http vs https. If you need affinity per user on http and https, you can use this configuration example: http://blog.exceliance.fr/2011/07/12/send-user-to-the-same-backend-for-both-http-and-https/ And the second example below will show you how to do persistence based on the SSL id: http://blog.exceliance.fr/2011/07/04/maintain-affinity-based-on-ssl-session-id/ cheers On Mon, Jan 30, 2012 at 1:14 AM, Sebastian Fohler i...@far-galaxy.com wrote: Where is the problem? As I already said before, it's my first haproxy config. I only used hardware loadbalancing before. Any help would be appreciated. Best regards Sebastian On 29.01.2012 23:27, Cyril Bonté wrote: Hi Sebastian, Is this really your configuration ? I'm afraid to say that it can't work the way it is done (or I missed something). For those reasons : Le 29/01/2012 14:09, Sebastian Fohler a écrit : Sure no problem, here is my config: the points the points # cat haproxy.cfg global maxconn 9 log 127.0.0.1 local0 uid 80 gid 80 nbproc 1 chroot /var/empty daemon listen adserver.adworxs.local bind 192.168.10.68:80 bind 192.168.10.68:443 mode http First of all, haproxy doesn't understand HTTPS natively. I guess that if you bind on port 443 this is because you want haproxy to receive HTTPS, but it can't. If you want to use haproxy in HTTP mode, you'll have to decode HTTPS into plain text HTTP with a ssl termination such as stunnel or stud. = Don't bind on port 443 with mode http enabled. log global option dontlognull option httpclose option forwardfor maxconn 1 clitimeout 3 balance roundrobin contimeout 3 srvtimeout 3 retries 10 option httpchk HEAD /www/admin/index.php HTTP/1.0 stats enable stats uri /haproxy?stats stats realm adserver.adworxs.local stats auth admin:password stats show-node stats refresh s server adserver1 192.168.10.61:80 check inter 1000 weight 10 server adserver1 192.168.10.61:443 check inter 1000 weight 10 Here, several issues occur : - You shouldn't use several times the same server name, this is maybe what made your diagnostics difficult in the stats page and in the logs. (In your first post, you said : The first problem I had was, while checking the haproxy stats, that they show every backendserver is at least the same time DOWN as it is UP) - You've mixed HTTP servers and HTTPS ones in the backend : haproxy will load balance the incoming connections on them (in round robin as you've configured it some lines before). Those connecting to port 443 will probably be always DOWN so you'll think it works if you only send HTTP requests. But as soon as you try to send HTTPS requests, you'll get in trouble. - You can't use check on your https server because you've configured option httpchk : haproxy will try to send plain text HTTP where the server is waiting for HTTPS. server adserver2 192.168.10.62:80 check inter 1000 weight 10 server adserver2 192.168.10.62:443 check inter 1000 weight 10 server adserver4 192.168.10.67:80 check inter 1000 weight 120 server adserver4 192.168.10.67:443 check inter 1000 weight 120 server adserver5 192.168.10.66:80 check inter 1000 weight 120 server adserver5 192.168.10.66:443 check inter 1000 weight 120 ...same issues for each adserver listen adserver.dev.adworxs.local bind 192.168.10.70:80 bind 192.168.10.70:443 mode http log global option dontlognull option httpclose option forwardfor maxconn 6000 clitimeout 3 balance leastconn contimeout 3 srvtimeout 3 retries 10 option httpchk HEAD /www/admin/index.php HTTP/1.0 stats enable stats uri /haproxy?stats stats realm adserver.dev.adworxs.local stats auth admin:password stats show-node stats refresh m server adserver1.dev 192.168.10.41:80 check inter 10 weight 100 server adserver1.dev 192.168.10.41:443 check inter 10 weight 100 and same issues for this proxy. I'd suggest you to remove every lines containing a reference to port 443 (binds and servers) and add stunnel or stud in front of haproxy, configured to receive incoming HTTPS request on port 443 and forward the plain text HTTP request to haproxy on port 80. Hope this helps.
Re: Geotargeting and Server DOWN problem
On 27.01.2012 16:01, Sebastian Fohler wrote: Sorry just found out that I definitly do an active check. But for some reason every second refresh of my stats shows the servers down. Any idea why that could be? The servers are definitly up all that time. Hmz, I don't know. It think it's helpful if you post more info like your haproxy config. Greets, Sander
Geotargeting and Server DOWN problem
I'm trying to setup a loadbalancing configuration with four backend server on nginx basis. The first problem I had was, while checking the haproxy stats, that they show every backendserver is at least the same time DOWN as it is UP, how can this be, and what could be the problem? Another problem I have is that the backend servers are using geotargeting to deliver specific content to specific country locations, since the haproxy loadbalancing has always the same ip the seem to be some confusion with the geotargeting after activating haproxy. Can someone help me to find out which configuration would be need to overcome these obstacles? Thank you in advance. Best regards Sebastian -- Sebastian Fohler Far-Galaxy Networks Web: http://www.far-galaxy.de Tel.: +49 (0)173 8796211 E-Mail: i...@far-galaxy.de
Re: Geotargeting and Server DOWN problem
Hi, On 26.01.2012 18:45, Sebastian Fohler wrote: I'm trying to setup a loadbalancing configuration with four backend server on nginx basis. The first problem I had was, while checking the haproxy stats, that they show every backendserver is at least the same time DOWN as it is UP, how can this be, and what could be the problem? Are you doing active check against the backend servers using haproxy? Another problem I have is that the backend servers are using geotargeting to deliver specific content to specific country locations, since the haproxy loadbalancing has always the same ip the seem to be some confusion with the geotargeting after activating haproxy. You might use real ip (http://wiki.nginx.org/HttpRealIpModule) and the haproxy 'option forwardfor' to solve the geotargetting problem. Greets, Sander
