Re: make haproxy notice that backend server ip has changed
On 08/05/2011 06:51 AM, Julien Vehent wrote: On Fri, 05 Aug 2011 01:08:22 +0300, Piavlo wrote: Hi Jens, I'm using names which resolve to internal EC2 addresses in haproxy configs - the /etc/hosts of all instances are auto updated then new instance is added/removed. But the problem manifests then the instance is stoped and then started - this makes the internal ip to change. I can also use DNS CNAMES to public instance ip with very low TTL - which get auto updated then instance boots by using route53 - but it's still the same problem - the ip changes in DNS and not in /etc/hosts (getnameinfo does not really care from where from the name is resolved) - in both cases haproxy will not know it has changed since it probably uses getnameinfo once only on startup/reload. And never later rechecks if the ip has changed. Hi, Haproxy resolves names into addresses at startup, so using names is just an ugly way, and probably confusing, to define an ip address. Names are less confusing and less ugly for me. I understand I can use a combination of automation tool like puppet/chef with zookeper like tool to rebuild the haproxy configuration and reload haproxy - in some situations like add/removal of backend server - that's unavoidable. But why do a reload of haproxy in other situations (much more common in my use case and loose statistics and possibly some connections) if there could be a config option that tells haproxy to re-resolve name to ip - then backend health check fails? The only reliable way to work with IP addresses in EC2, without messing with the whole /etc/hosts and applications reload, is to use static LAN addresses, like in a real network. And you can do that just fine in a VPC environment. DO NOT use elastic IPs for internal traffic. Elastic IPs are routed through the external network of EC2, so you will get charged $$$, your interconnections will be slower and you don't even know where the hell your packets are going I never said that I'm using elastic IPs. But I don't think it matters if it's an elastic/static ip or just a normal public ip which can/will change on stop/start of ec2 instance. There is a well know trick in EC2 - if you do dns lookup on public ec2 hostname from within ec2 - you will get and internal ip instead of public ip. So you are not charged because you are effectively working with internal ip's - if you have a CNAME to public A records - it ends up resolving to internal ec2 ip from within ec2 and to public ip from outside of ec2. Thanks Alex Just the result of my personal experience... :) Julien
Re: erratic X-Forwarded-For patch for stunnel
**
Hi,
are you using httpclose in haproxy in the frontend for the ssl portion of
haproxy ? Willy has talk about other ways to solve this yesterday but just
to do a test you could put option httpclose in this frontend.
most of the time there is only 192.168.134.222 the IP of haproxy) It's the
ip of stunnel to I imagine ?
You can see more then 1 X-Forwarded-For in the log it's cumulative... But
you can tell haproxy to not include X-Forwarded-For when stunnel already put
the client ip with a option like this:
option forwardfor except 10.222.0.0/27
for me this is the subnet of the ssl offloader 10.222.0.0/27.
SO the way I understand it
Client - stunnel (add the client ip to X-For) ---
haproxy (will not add X-For) - apache1
- apache2
Client -- ---
haproxy (will add X-For) - apache1
- apache2
Then you need to decide if you will be using option httpclose or what was
discuss yesterday
From Willy;
So if you need stunnel to provide the IP to haproxy, you have two
solutions :
- either disable keep-alive using option httpclose on haproxy so that
it forces stunnel to reopen a new connection for each request and to
add the header to each of them ;
- or make use of the send-proxy patch for stunnel, which is compatible
with the accept-proxy feature of haproxy. This is the preferred solution
because instead of mangling the beginning of the HTTP request, stunnel
then informs haproxy about the source address in an out-of-band fashion,
which makes it compatible with keep-alive.
Bye.
e 2011-08-05 05:45, Damien Hardy a écrit :
Hello,
I patched the debian stunnel4 package for squeeze
# aptitude install devscripts build-essential fakeroot
# apt-get build-dep stunnel4
# apt-get source stunnel4
# wget
http://haproxy.1wt.eu/download/patches/stunnel-4.29-xforwarded-for.diff
# cd stunnel4-4.29/
# patch -p1 -i ../stunnel-4.29-xforwarded-for.diff
# debuild -us -uc
# dpkg -i ../stunnel4_4.29-1_amd64.deb
change my conf /etc/stunnel/stunnel.conf as :
[...]
[https]
accept = 192.168.134.222:443
connect = 192.168.134.222:4430
TIMEOUTclose = 0
xforwardedfor = yes
change my conf /etc/haproxy/haproxy.conf as :
listen sslsite
bind 192.168.134.222:4430
balance roundrobin
cookie SRV insert indirect nocache
capture request header X-Forwarded-For len 256
rspirep ^Location:\ http://(.*)Location:\ https://\1
server vexft04 192.168.16.55:80 cookie ahD2Fiel check inter 5000 fall 3
server vexft05 192.168.16.50:80 cookie ifaop7Ge check inter 5000 fall 3
server vexft06 192.168.128.52:80 cookie aina1oRo check inter 5000 fall
3
server vexft07 192.168.128.53:80 cookie ohQuai5g check inter 5000 fall
3
But X-Forwarded-For header is inconstantly set in logs as :
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.218] sslsite
sslsite/vexft04 0/0/0/250/250 200 3865 -
- --NI 1/1/0/1/0 0/0 {10.147.28.20} GET /admin/AdmInscriptionPro.shtml
HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.468] sslsite
sslsite/vexft04 31/0/1/1/33 200 471 - -
--VN 1/1/0/1/0 0/0 {} GET /css/admin/master.css HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.502] sslsite
sslsite/vexft04 173/0/0/5/178 200 2018 -
- --VN 1/1/0/1/0 0/0 {} GET /css/lightwindow.css HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.680] sslsite
sslsite/vexft04 56/0/1/1/58 200 573 - -
--VN 1/1/0/1/0 0/0 {} GET /css/sIFR-screen.css HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.739] sslsite
sslsite/vexft04 64/0/1/1/66 200 722 - -
--VN 1/1/0/1/0 0/0 {} GET /css/niftyCorners.css HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.805] sslsite
sslsite/vexft04 3/0/1/11/16 200 28961 - -
--VN 1/1/0/1/0 0/0 {} GET /script/aculous/prototype.js HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43922[05/Aug/2011:11:23:54.832] sslsite
sslsite/vexft04 0/0/0/1/1 200 2071 - -
--VN 4/4/3/4/0 0/0 {10.147.28.20} GET /script/espace-pro.js HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43920[05/Aug/2011:11:23:54.831] sslsite
sslsite/vexft04 0/0/0/2/2 200 1811 - -
--VN 4/4/2/3/0 0/0 {10.147.28.20} GET /script/niftyCorners.js HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43924[05/Aug/2011:11:23:54.832] sslsite
sslsite/vexft04 0/0/0/2/2 200 739 - -
--VN 6/6/3/4/0 0/0 {10.147.28.20} GET /script/niftyDeclare.js HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43928[05/Aug/2011:11:23:54.834] sslsite
sslsite/vexft04 0/0/0/1/1 200 604 - -
--VN 6/6/2/3/0 0/0 {10.147.28.20} GET /script/admin/menu_admin.js HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.821] sslsite
sslsite/vexft04 7/0/0/7/14 200 13798 - -
--VN 6/6/2/3/0 0/0 {} GET /script/aculous/lightwindow.js HTTP/1.1
Fwd: erratic X-Forwarded-For patch for stunnel
(this ML need a reply-to header :)
For the conclusion :
-- Forwarded message --
From: Damien Hardy damienhardy@gmail.com
Date: 2011/8/5
Subject: Re: erratic X-Forwarded-For patch for stunnel
To: Guillaume Bourque guillaume.bour...@gmail.com
Good point for you.
I was running with option http-server-close as global configuration.
Now with option httpclose it get the X-Forwarded-For for every request.
Thank you a lot.
--
Damien
2011/8/5 Guillaume Bourque guillaume.bour...@gmail.com
**
Hi,
are you using httpclose in haproxy in the frontend for the ssl portion of
haproxy ? Willy has talk about other ways to solve this yesterday but just
to do a test you could put option httpclose in this frontend.
most of the time there is only 192.168.134.222 the IP of haproxy) It's
the ip of stunnel to I imagine ?
You can see more then 1 X-Forwarded-For in the log it's cumulative... But
you can tell haproxy to not include X-Forwarded-For when stunnel already put
the client ip with a option like this:
option forwardfor except 10.222.0.0/27
for me this is the subnet of the ssl offloader 10.222.0.0/27.
SO the way I understand it
Client - stunnel (add the client ip to X-For) ---
haproxy (will not add X-For) - apache1
- apache2
Client -- ---
haproxy (will add X-For) - apache1
- apache2
Then you need to decide if you will be using option httpclose or what was
discuss yesterday
From Willy;
So if you need stunnel to provide the IP to haproxy, you have two
solutions :
- either disable keep-alive using option httpclose on haproxy so that
it forces stunnel to reopen a new connection for each request and to
add the header to each of them ;
- or make use of the send-proxy patch for stunnel, which is compatible
with the accept-proxy feature of haproxy. This is the preferred solution
because instead of mangling the beginning of the HTTP request, stunnel
then informs haproxy about the source address in an out-of-band fashion,
which makes it compatible with keep-alive.
Bye.
e 2011-08-05 05:45, Damien Hardy a écrit :
Hello,
I patched the debian stunnel4 package for squeeze
# aptitude install devscripts build-essential fakeroot
# apt-get build-dep stunnel4
# apt-get source stunnel4
# wget
http://haproxy.1wt.eu/download/patches/stunnel-4.29-xforwarded-for.diff
# cd stunnel4-4.29/
# patch -p1 -i ../stunnel-4.29-xforwarded-for.diff
# debuild -us -uc
# dpkg -i ../stunnel4_4.29-1_amd64.deb
change my conf /etc/stunnel/stunnel.conf as :
[...]
[https]
accept = 192.168.134.222:443
connect = 192.168.134.222:4430
TIMEOUTclose = 0
xforwardedfor = yes
change my conf /etc/haproxy/haproxy.conf as :
listen sslsite
bind 192.168.134.222:4430
balance roundrobin
cookie SRV insert indirect nocache
capture request header X-Forwarded-For len 256
rspirep ^Location:\ http://(.*)Location:\ https://\1
server vexft04 192.168.16.55:80 cookie ahD2Fiel check inter 5000 fall
3
server vexft05 192.168.16.50:80 cookie ifaop7Ge check inter 5000 fall
3
server vexft06 192.168.128.52:80 cookie aina1oRo check inter 5000
fall 3
server vexft07 192.168.128.53:80 cookie ohQuai5g check inter 5000
fall 3
But X-Forwarded-For header is inconstantly set in logs as :
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.218] sslsite sslsite/vexft04
0/0/0/250/250 200 3865 -
- --NI 1/1/0/1/0 0/0 {10.147.28.20} GET /admin/AdmInscriptionPro.shtml
HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.468] sslsite sslsite/vexft04
31/0/1/1/33 200 471 - -
--VN 1/1/0/1/0 0/0 {} GET /css/admin/master.css HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.502] sslsite sslsite/vexft04
173/0/0/5/178 200 2018 -
- --VN 1/1/0/1/0 0/0 {} GET /css/lightwindow.css HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.680] sslsite sslsite/vexft04
56/0/1/1/58 200 573 - -
--VN 1/1/0/1/0 0/0 {} GET /css/sIFR-screen.css HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.739] sslsite sslsite/vexft04
64/0/1/1/66 200 722 - -
--VN 1/1/0/1/0 0/0 {} GET /css/niftyCorners.css HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.805] sslsite sslsite/vexft04
3/0/1/11/16 200 28961 - -
--VN 1/1/0/1/0 0/0 {} GET /script/aculous/prototype.js HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43922[05/Aug/2011:11:23:54.832] sslsite sslsite/vexft04
0/0/0/1/1 200 2071 - -
--VN 4/4/3/4/0 0/0 {10.147.28.20} GET /script/espace-pro.js HTTP/1.1
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43920[05/Aug/2011:11:23:54.831] sslsite sslsite/vexft04
0/0/0/2/2 200 1811 - -
--VN 4/4/2/3/0 0/0
Re: Fwd: erratic X-Forwarded-For patch for stunnel
On Fri, Aug 05, 2011 at 03:21:31PM +0200, Damien Hardy wrote: (this ML need a reply-to header :) No, because reply-to makes it harder to reply to individual people, and incites responders to reply only to the list, which is the best way to lose track of threads, as most of us are not constantly watching the list. Thanks for the forward anyway ;-) Willy
[ANNOUNCE] haproxy 1.4.16
Hi all, Since 1.4.15 was released 2 months ago, very few minor bugs were detected. They were so minor that it was worth waiting for other ones to be found, but after some time, there wasn't any point making users wait any more, so I released 1.4.16. A few minor improvements were also made based on feedback from users. Among the changes, MySQL checks now support Mysqld versions after 5.5, health checks support for multi-packet response has been fixed, the HTTP 200 status can be configured for monitor responses, a new http-no-delay option has been added to work around buggy HTTP implementations that assume packet-based transfers, chunked-encoded transfers have been optimised a bit, the stats interface now support URL-encoded forms, and halog correctly handles truncated files. Quite honnestly, there is no real emergency to upgrade but it makes sense for new deployments and for packagers. The usual links apply and I have even built both the Linux and Solaris binaries : site index : http://haproxy.1wt.eu/ sources: http://haproxy.1wt.eu/download/1.4/src/ changelog : http://haproxy.1wt.eu/download/1.4/src/CHANGELOG binaries : http://haproxy.1wt.eu/download/1.4/bin/ I'll check if some backports are needed and will issue a 1.3 soon too. Have fun, Willy
unknown keyword 'userlist' in '****' section
Hello, So I'm trying to setup a userlist in haproxy.cfg using the following instructions: http://code.google.com/p/haproxy-docs/wiki/Userlists however no matter where in the .cfg file I put the userlist directive in it keeps giving me errors: unknown keyword 'userlist' in 'frontend' section unknown keyword 'userlist' out of section. unknown keyword 'userlist' in 'backend' section unknown keyword 'userlist' in 'defaults' section unknown keyword 'userlist' in 'global' section Version of Pound is 2.4.3 Can someone help? TIA
Re: unknown keyword 'userlist' in '****' section
correction, Version is HA-Proxy version 1.3.15.2 On Fri, Aug 5, 2011 at 1:09 PM, Tom Sztur tsz...@gmail.com wrote: Hello, So I'm trying to setup a userlist in haproxy.cfg using the following instructions: http://code.google.com/p/haproxy-docs/wiki/Userlists however no matter where in the .cfg file I put the userlist directive in it keeps giving me errors: unknown keyword 'userlist' in 'frontend' section unknown keyword 'userlist' out of section. unknown keyword 'userlist' in 'backend' section unknown keyword 'userlist' in 'defaults' section unknown keyword 'userlist' in 'global' section Version of Pound is 2.4.3 Can someone help? TIA
Re: unknown keyword 'userlist' in '****' section
On Fri, Aug 5, 2011 at 1:10 PM, Tom Sztur tsz...@gmail.com wrote: correction, Version is HA-Proxy version 1.3.15.2 Userlist is not an option in 1.3. See your version's documentation: http://haproxy.1wt.eu/download/1.3/doc/configuration.txt
Re: make haproxy notice that backend server ip has changed
On 08/05/2011 06:53 PM, Willy Tarreau wrote: On Fri, Aug 05, 2011 at 11:17:16AM +0300, Piavlo wrote: But why do a reload of haproxy in other situations (much more common in my use case and loose statistics and possibly some connections) if there could be a config option that tells haproxy to re-resolve name to ip - then backend health check fails? It's not a matter of config option. You're supposed to run haproxy inside a chroot. It will then not have access to the resolver. There are simple ways to make the resolver work inside chroot without making the chroot less secure. I could ask the question the other direction : why try to resolve a name to IP when a check fails, there is no reason why a server would have its address changed without the admin being responsible for it. I don't agree that admin is supposed to be responsible for it directly at all. Say backend server crashes/enters bad state - this is detected and new ec2 instance is automatically spawned and autoconfigured to replace the failed backend ec2 instance- which is optionally terminated. The /etc/hosts of all relevent ec2 instances is auto updated (or DNS with 60 seconds ttl is updated - by the way the 60 seconds ttl works great withing ec2). There is no admin person involved - all is done automatically. Also, in your case it would not fix the issue : resolving when the server goes down will bring you the old address, and only after caches expires it would bring the new one. If /etc/hosts is updated locally the is no need to wait for cache expiration. And if /etc/hosts is auto updated by appropriate tool - going one more step of restarting/reloading haproxy is not a problem at all - but this is what I want to avoid. If instead for example i could send a command to haproxy control socket to re-resolve all the names (or better just specific name) configured in haproxy - it would be much better - as since /etc/hosts is already updated it would resolve to correct ip address. BTW afaiu adding/removing backends/frontends dynamically on the fly through some api / socket - is not something that is ever planned to be supported in haproxy? Thanks Alex In the mean time, someone else might have got the old address before you have a new one, so this means that it is still possible that only the old address is used. There is no way to reach a reliable behaviour with unreliable configuration processes. Regards, Willy
Re: make haproxy notice that backend server ip has changed
Why don't you edit the haproxy conf directly and reload it ? If you have the new IP and are going to update the /etc/hosts, what is stopping you from doing a sed on the backend's ip in haproxy.cfg ? Or, you could just run in a VPC and stop doing weird stuff with your networking ;) Julien Or using some kind of haproxy conf template with some keyword you replace using sed with IPs you would get from the hosts file? with inotify, you can get updated each time hosts file change, then you generate a new haproxy conf from your template and you ask haproxy to reload it :) brilliant ! cheers
Re: cookie-less sessions
Hi Hank Actually stick on URL param should work with client which does not support cookies. is the first reply a 30[12] ? How is they user aware of the jsid or how is he supposed to send his jsid to the server? Do you have a X-Forwarded-For on your proxy or can you setup one? cheers
Re: make haproxy notice that backend server ip has changed
On Fri, Aug 5, 2011 at 11:58 PM, Willy Tarreau w...@1wt.eu wrote: Hi Baptiste, On Fri, Aug 05, 2011 at 11:53:40PM +0200, Baptiste wrote: Or using some kind of haproxy conf template with some keyword you replace using sed with IPs you would get from the hosts file? with inotify, you can get updated each time hosts file change, then you generate a new haproxy conf from your template and you ask haproxy to reload it :) Once again, if the host is in /etc/hosts, then you don't need to touch the config anymore. Simply reload it so that it resolves the hosts again. cheers, Willy Why make things easy when you can make them complicated cheers
