Re: make haproxy notice that backend server ip has changed
On Sat, Aug 06, 2011 at 02:42:45AM +0300, Piavlo wrote: Well certainly aws has it's limitations which force you to design a very different infrastructure than you would in normal datacenter environment. IMHO this is the great thing about those limitations as you are forced to start thinking differently and end up using a set of well known and established tools to overcome those limitations. Surely, getting rid of everything which had worked fine for ages and limiting oneself to use lazy and naive approaches like DNS because it's probably good enough to offer high availability is a way to think differently. But it's not the way I conceive reliable infrastructures. I'm talking mainly about monitoring/automation/deployment tools centralized coordination service tools - so that you can automatically react to any change in the infrastructure. Changes should not happen often, so you can expect that they come with a minor cost. Or you have something which makes your servers die several times a minute and you need to fix that before considering adding servers. With those tools you don't really care if some server ip changes - the ip only changes if you stop/start and start ec2 instance. If you reboot ec2 instance the ip does not change. But normally you would not really stop/start instance - this really happens then something bad happens to the instance, so that you need to reboot it, but reboot does not always works since there might be hardware problem on the server hosting this ec2 instance. So you need to stop it and then start - then you start it will start on different hardware server. Fine. In the real world, when a server is dead, one guy comes with a master, reinstalls it on another hardware and restores its configuration. The IP is taken back and everything magically works again. In the VPC you should be able to do that too when you decide to replace a faulty instance. But you don't really need to all this stuff manually. If some ec2 instance is sick this is detected and propagated through the centralized coordination service to the relevant parties. Here I think you need to define sick. For me, a sick server is one that needs a stop/start or reboot sequence to be fine again. Otherwise it's considered dead and needs at least repair, at worst replacement. Repair is covered by high availability. In case of replacement, you can keep the IP, so seen from the LB it's just a repair. Then you can decide to start a service from a failed instance on another already running ec2 instance or start new instance configure itself and start the service. If the already running instance was OK, why was it not integrated in the LB farm then ? The old failed instance can be just killed or suspended. (So VPC or normal datacenter will not help here - since the service will be running on different instance/server with different ip - yes you could use a floating ip in normal datacenter but you would not want to do that for every backend especially then backend are automatically added/removed. No but here you're already describing corner cases, I see a lot of if here to reach that case, and at this point I think that a simple reload is the smallest operation to complete the process ! You would normally use floating ip for the frontend). Then service is active again on another/new instance - this is again propagated through the centralized coordination service. Then you automatically update needed stuff on relevant instances - like in this specific case update /etc/hosts and restart/reload haproxy. (All I wanted is to avoid haproxy restart/reload - there is no technical problem at all to do the restart). And of course all this is done automatically without human intervention. So you realize that you're saying you lose a server, you look for another compatible server, you find one which is doing nothing useful, you decide to install the service on it, you start it, you update all /etc/hosts and the only thing you don't want to do is to reload a process, which represents less than 0.01% of all the operations that have been performed automatically for you ! I don't buy that, that does not make any sense to me, I'm sorry. For me, it's comparable to the guy who would absolutely want to be able to power his servers on batteries before moving the rack to another city, so that he can avoid a shutdown+restart sequence which would kill its impressive uptimes. From where I stand I see no unreliability problem with aws - the normal datacenter is just unreliable for me as aws. I don't need the normal datacenter or the VPC. The usage of those tools and the other aws features make aws much more attractive and reliable than normal datacenter. Quite frankly, given the way you consider reliability, I fail to understand why you insist on using a load balancer. Why not advertise all your servers with the DNS and let the monitoring mechanism automatically
Re: cookie-less sessions
On 8/5/11 3:01 PM, Baptiste wrote: Hi Hank Actually stick on URL param should work with client which does not support cookies. is the first reply a 30[12] ? So you are saying that stick on URL param reads the outgoing 302 and saves the URL param from that in the stick table on 1.5? f so, great then problem solved. If it doesn't save it on the way out from the initial redirect then it won't help. Is the same supposed to happen with balance url_param on 1.4? If not, I will switch to 1.5. If it is supposed to, it doesn't, afaict. How is they user aware of the jsid or how is he supposed to send his jsid to the server? 302 to the URL with the jsid URL param. Thanks Do you have a X-Forwarded-For on your proxy or can you setup one? cheers
Re: cookie-less sessions
On Sat, Aug 6, 2011 at 8:51 AM, Hank A. Paulson h...@spamproof.nospammail.net wrote: On 8/5/11 3:01 PM, Baptiste wrote: Hi Hank Actually stick on URL param should work with client which does not support cookies. is the first reply a 30[12] ? So you are saying that stick on URL param reads the outgoing 302 and saves the URL param from that in the stick table on 1.5? f so, great then problem solved. If it doesn't save it on the way out from the initial redirect then it won't help. Is the same supposed to happen with balance url_param on 1.4? If not, I will switch to 1.5. If it is supposed to, it doesn't, afaict. How is they user aware of the jsid or how is he supposed to send his jsid to the server? 302 to the URL with the jsid URL param. Thanks Do you have a X-Forwarded-For on your proxy or can you setup one? cheers Well, I'm thinking of something, let me run some tests and I'll come back to you with a good or a bad news. cheers
Re: cookie-less sessions
Hi Baptiste, On Sat, Aug 06, 2011 at 09:24:08AM +0200, Baptiste wrote: On Sat, Aug 6, 2011 at 8:51 AM, Hank A. Paulson h...@spamproof.nospammail.net wrote: On 8/5/11 3:01 PM, Baptiste wrote: Hi Hank Actually stick on URL param should work with client which does not support cookies. is the first reply a 30[12] ? So you are saying that stick on URL param reads the outgoing 302 and saves the URL param from that in the stick table on 1.5? f so, great then problem solved. If it doesn't save it on the way out from the initial redirect then it won't help. Is the same supposed to happen with balance url_param on 1.4? If not, I will switch to 1.5. If it is supposed to, it doesn't, afaict. How is they user aware of the jsid or how is he supposed to send his jsid to the server? 302 to the URL with the jsid URL param. Thanks Do you have a X-Forwarded-For on your proxy or can you setup one? cheers Well, I'm thinking of something, let me run some tests and I'll come back to you with a good or a bad news. Right now I see no way to do that. We'd need to extract the url_param from the Location header, this would be a new pattern. I think it's not too hard to implement. We already have url_param for the request, we could have hdr_url_param(header_name) or something like this. Regards, Willy
Re: cookie-less sessions
On 8/6/11 12:32 AM, Willy Tarreau wrote: Hi Baptiste, On Sat, Aug 06, 2011 at 09:24:08AM +0200, Baptiste wrote: On Sat, Aug 6, 2011 at 8:51 AM, Hank A. Paulson h...@spamproof.nospammail.net wrote: On 8/5/11 3:01 PM, Baptiste wrote: Hi Hank Actually stick on URL param should work with client which does not support cookies. is the first reply a 30[12] ? So you are saying that stick on URL param reads the outgoing 302 and saves the URL param from that in the stick table on 1.5? f so, great then problem solved. If it doesn't save it on the way out from the initial redirect then it won't help. Is the same supposed to happen with balance url_param on 1.4? If not, I will switch to 1.5. If it is supposed to, it doesn't, afaict. How is they user aware of the jsid or how is he supposed to send his jsid to the server? 302 to the URL with the jsid URL param. Thanks Do you have a X-Forwarded-For on your proxy or can you setup one? cheers Well, I'm thinking of something, let me run some tests and I'll come back to you with a good or a bad news. Right now I see no way to do that. We'd need to extract the url_param from the Location header, this would be a new pattern. I think it's not too hard to implement. We already have url_param for the request, we could have hdr_url_param(header_name) or something like this. Regards, Willy but it would have to be a combo thing, right? - set up stick table entry on outgoing 302 with url_param blah in location header then check incoming requests for url_param blah and continue to stick those based on the entry created by the initial outgoing response. I don't know if you'd do that as 2 rules or one. This would be a great and wondrous thing for cookie-less clients and decoupled servers - not to mention it may be a unique feature among existing load balancing products.
Re: cookie-less sessions
On Sat, Aug 6, 2011 at 9:32 AM, Willy Tarreau w...@1wt.eu wrote: Hi Baptiste, On Sat, Aug 06, 2011 at 09:24:08AM +0200, Baptiste wrote: On Sat, Aug 6, 2011 at 8:51 AM, Hank A. Paulson h...@spamproof.nospammail.net wrote: On 8/5/11 3:01 PM, Baptiste wrote: Hi Hank Actually stick on URL param should work with client which does not support cookies. is the first reply a 30[12] ? So you are saying that stick on URL param reads the outgoing 302 and saves the URL param from that in the stick table on 1.5? f so, great then problem solved. If it doesn't save it on the way out from the initial redirect then it won't help. Is the same supposed to happen with balance url_param on 1.4? If not, I will switch to 1.5. If it is supposed to, it doesn't, afaict. How is they user aware of the jsid or how is he supposed to send his jsid to the server? 302 to the URL with the jsid URL param. Thanks Do you have a X-Forwarded-For on your proxy or can you setup one? cheers Well, I'm thinking of something, let me run some tests and I'll come back to you with a good or a bad news. Right now I see no way to do that. We'd need to extract the url_param from the Location header, this would be a new pattern. I think it's not too hard to implement. We already have url_param for the request, we could have hdr_url_param(header_name) or something like this. Regards, Willy Actually, I was thinking of the stuff you developed to replace the appsession by stick tables. A dirty workaround could be: - configure the application to send a response with a set-cookie and the 302/Location header, both with the same ID value. - on HAProxy, match the set-cookie of the response and store it in a stick table - when the client send request, match the URL param against the table set up above I agree that learning a URL param from a header would be cleaner ;) Hank, these options are currently not available in HAProxy, even in 1.5-dev6. I guess it will be released in the next 1.5-dev. cheers
Re: cookie-less sessions
I made it work on our Aloha load-balancer (4.1.2) :) PHP code on the server: cookie.php : ?php session_start(); header(Location: /?ID= . session_id()); echo apache_getenv(SERVER_ADDR); ? test script.php: ?php echo apache_getenv(SERVER_ADDR); ? it creates a set-cookie with cookie name PHPSESSID and redirect the use with a URL param ID (which has the same value as the cookie id). HAProxy configuration, on my backend configuration: stick-table type string len 32 size 10K stick store-response set-cookie(PHPSESSID) stick on url_param(ID) The test: First request: $ curl -D - http://aloha/cookie.php HTTP/1.1 302 Found Date: Sat, 06 Aug 2011 09:13:45 GMT Server: Apache/2.2.16 (Debian) X-Powered-By: PHP/5.3.3-7+squeeze3 Set-Cookie: PHPSESSID=8a28f7089e9d70c3375505c9620472db; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Location: /?ID=8a28f7089e9d70c3375505c9620472db Vary: Accept-Encoding Content-Length: 68 Content-Type: text/html 192.168.10.101 Second request, with the ID on URL: $ curl -D - http://aloha/script.php?ID=8a28f7089e9d70c3375505c9620472db HTTP/1.1 200 OK Date: Sat, 06 Aug 2011 09:15:21 GMT Server: Apache/2.2.16 (Debian) X-Powered-By: PHP/5.3.3-7+squeeze3 Vary: Accept-Encoding Content-Length: 17 Content-Type: text/html 192.168.10.101 Note that I'm using the same backend, since the IP printed is the same. Let's have a look at the table now: echo show table bk_myappli | socat unix-connect:/var/run/haproxy.stat stdio # table: bk_myappli, type: string, size:10240, used:5 0x14b5694: key=8a28f7089e9d70c3375505c9620472db use=0 exp=0 server_id=1 Hank, as said before, it's not yet in HAProxy. I can't speak on behalf of Willy, but you can ask him to kindly include it on the next 1.5-dev release :) cheers
Re: cookie-less sessions
On Sat, Aug 6, 2011 at 12:50 PM, Willy Tarreau w...@1wt.eu wrote: On Sat, Aug 06, 2011 at 11:27:53AM +0200, Baptiste wrote: I made it work on our Aloha load-balancer (4.1.2) :) Baptiste, you should stop taking hardware with you during holidays, it's too much temptation ;-) Since we have the KVM version, I can work at anytime, anywhere... It's bad Anyway, I'm always keen to help people :) cheers
Re: Automate backend registration
EC2 auto registration Common practice is to do this the right scale / IBM way. When a backend server is launched via autoscaling then it makes an ssh remote command call to the loadbalancer calling an api script that adds the backend to the running haproxy configuration. By all means steal the one from the loadbalancer.org ec2 appliance and copy or modify it... We re happy to release the script as gpl. You can get the details from our web site. On Thursday, 4 August 2011, Jens Bräuer jens.brae...@numberfour.eu wrote: Hi Holger, thanks for the answer. I already assemble HA-Proxy configuration with the help of Puppet. Now things require a bit more logic fe. I dont want to restart HA-Proxy if the config that ships with the RPM does not change. I'll definitely have a look at your script - thanks for the pointer. CU, Jens On 03.08.2011, at 23:22, Holger Just wrote: Jens, Many people have a script that builds a working configuration file from various bits and pieces. As the actual needed configuration typically isn't something which follows a common path but depends on the environment and the actual applications and a thousand other bits, there isn't a standard here. But it really isn't hard to throw together a small shell/perl/python whatever script which concatenates the final config file from various pieces or uses some templating language of your chosen language. An script we use is https://github.com/finnlabs/haproxy. It consists of a python script which assembles the config file from a certain directory structure. This script is then called before a start/reload of the haproxy in the init script. So basically, you need to create your script for generating your Haproxy configuration, hook it into your init script and then, as a post-install in your RPMs put the configuration in place for your configuration-file-creating-script and reload Haproxy. To enable/disable previously registered backend components, you might be able to use the socket, but that usage is rather limited and mainly intended for maintenance, not for actual configuration changes. Hope that helps and sorry if that was a bit recursive :) Holger On 2011-08-03 22:52, Jens Bräuer wrote: Hi Baptiste, sorry for my wording. But you are right, with registration I mean - add ACL - add use_backend - add backend section so to sum it up make haproxy aware of a new application. There might be cases there I want to only add a server to existing backend, but that would be the second/third step. The use-case is that I have HA-Proxy running and do a yum/apt-get install and the RPM should come with everything to integrate with HA-Proxy. I am sure that there must be some tool out there.. ;-) Cheers, Jens On 03.08.2011, at 20:24, Baptiste wrote: Hi Jens, What do you mean by registration? Is that make haproxy aware of the freshly deployed application ? cheers On Wed, Aug 3, 2011 at 5:46 PM, Jens Bräuer jens.brae...@numberfour.eu wrote: Hi HA-Proxy guys, I wonder whats the current state of the art to automate the registration of backend. My setup runs in on EC2 and I run HA-Proxy in front of local applications to easy administration. So a typical config file would be like this. frontend http bind *:8080 acl is-auth path_beg /auth acl is-core path_beg /core use_backend authif is-auth use_backend coreif is-core backend auth server auth-1 localhost:7778 check backend core server core-1 localhost:1 check All applications are installed via RPMs and I would like couple the installation with the backend registration. I like to do this as want to configure everything in one place (the RPM) and the number of installed applications may vary from host to host. I'd really appreciate hint where I can find tools or whats the current state to handle this kind of task. Cheers, Jens -- Regards, Malcolm Turnbull. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
