Redirect_prefix question
Hi Guys, I have a wildcard SSL certificate which I wish to do a redirect from http to https whilst maintaining the original subdomain requested, eg: http://sub.domain.com/page should be redirected to https://sub.domain.com/page I can't do this with redirect_prefix (it seems to require I know the destination in advance) Can anyone point me in the right direction for this? Using haproxy 1.3.19 on a Centos 5 machine. Cheers Chris This message was sent using IMP, the Internet Messaging Program.
Question relating to the errors show on the HAProxy stats screen and 503 errors.
Under the errors section what exactly does the Resp.section mean? Does it mean any response that is not a 200 OK? I have a customer with a lot of check errors, a lot of Resp. errors and too many Down errors. Origional config was: server mm5 173.45.238.119:80 weight 1 cookie mm5 server mm6 173.45.227.82:80 weight 1 cookie mm6 I've changed it to the following for more testing and enabled logging at info level. server mm5 173.45.238.119:80 weight 1 cookie mm5 check inter 2 fall 3 rise 1 server mm6 173.45.227.82:80 weight 1 cookie mm6 check inter 2 fall 2 rise 1 They are getting 503 errors and I assume that means that HAProxy is passing traffic to the real server , getting an error then telling the client 503? or do I misunderstand that? Current load is between 200-400 concurrent sessions. Also I just realised that the option redispatch was not set, would this option reduce the number of 503 errors? -- Regards, Malcolm Turnbull. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Re: dynamic weights based on actual server load
2009/10/16 Craig cr...@haquarter.de: Hi, a patch (set weight/get weight) I imagined some days ago was integrated just 6hrs after I had thought about it (Willy must be reading them!). I've written a simple (exchangable) interface that prints out a servers load and a client to read it. I plan to read the load from all servers and adjust the weight dynamically according to the load so that a very busy server gets less queries and the whole farm is more balanced. I plan to smoothen the increasing/decreasing a bit so that there aren't too great jumps with the weight, I want to implement a policy of something like oh that server can do 50% more, lets just increase the weight by 25% and check again in a minute. I hope this will autobalance servers with different hardware quite well, so that you don't have to guess or do performance tests to get the weights properly. Some python code is already finished (partly because I'd like to practise a bit) but I didn't continue yet, because I'd like to hear your opionions about this. Hi I'm sure lot of ppl (including me) will be interested in that ( I would probably have to write something similar in a month or so). By load you mean /proc/loadavg ? If yes then u might want to include some kind of per-server tuning of multiplier because obviously on 16 core server loadavg of 10 would be moderately loaded while on 4 core server i'd be overload Regards Mariusz
Re: dynamic weights based on actual server load
For the code you are developing, if you make the interface general enough so that parameters can be added or removed that would be good. Telnet/text/memcached style protocols seem popular to allow easy debugging/monitoring. So if your protocol says a machine has to send a load info bundle like: SS:8cbed340118ddf87e2d8ca4352006572 SYSID: blah1 SAMPLETIME: 2009-10-14-22-00-03 CPU: 83.23343455 NETI: 134238.0232 NETO: 492283.6549 DISK: 433.232 ES:8cbed340118ddf87e2d8ca4352006572 that would give you a generic record format that anyone could create a client or modify your client and add/remove load parameter info fields. Then they just take their list of load parameters and wrap them in the header and footer with the id of that sample record being the md5sum of the included data. SS = start sample record/ES = end sample record Then you could have a separate/pluggable module or process that takes that info and maybe pulls other data from system history files, etc and decides what to set the weight to for that server. You could provide a default weight setter engine that uses a simple algo based on just CPU load or something and others could fill in more complex/custom engines if desired. You might want to check out feedbackd: http://ozlabs.org/~jk/projects/feedbackd/ and his paper on Using Dynamic Feedback to Optimise Load Balancing Decisions http://redfishsoftware.com.au/projects/feedbackd/lca-paper.pdf to get some ideas. It is probably possible to just modify feedbackd to emit haproxy set weight commands. More interesting, I think would be to combine a multiple load parameter (active connections, CPU, net in/out bytes, net in/out packets, disk io, etc) feedback system with the ideas from the NetBSD neural network scheduler, creating an ai based dynamic load balancing system. http://softlayer.dl.sourceforge.net/project/nnsched/docs/thesis/nnsched.pdf This is more possible now that we have multi core systems that would have some idle CPU resources available for the ai compute load. On 10/16/09 10:29 AM, Craig wrote: Hi, a patch (set weight/get weight) I imagined some days ago was integrated just 6hrs after I had thought about it (Willy must be reading them!). I've written a simple (exchangable) interface that prints out a servers load and a client to read it. I plan to read the load from all servers and adjust the weight dynamically according to the load so that a very busy server gets less queries and the whole farm is more balanced. I plan to smoothen the increasing/decreasing a bit so that there aren't too great jumps with the weight, I want to implement a policy of something like oh that server can do 50% more, lets just increase the weight by 25% and check again in a minute. I hope this will autobalance servers with different hardware quite well, so that you don't have to guess or do performance tests to get the weights properly. Some python code is already finished (partly because I'd like to practise a bit) but I didn't continue yet, because I'd like to hear your opionions about this. Am I mad? ;) Best Regards, Craig
Re: dynamic weights based on actual server load
Hello, Before anyone goes too far into creating things here, I should probably point out that I have already written a system that handles this feature based upon CPU, Memory and Network usage, it was my University Dissertation! Since that has only just been marked, I haven't previously been able to discuss or release any information of this to the community. In simple terms, I have created a fairly simple daemon that monitors the CPU, Memory and Network usage of servers using the SNMP protocol. The statistics gathered are stored in a MySQL database, which allows the graphical web user interface to graph these statistics. Whilst I can go into further details at a later date, I'd like to request some assistance from someone in the community who has more experience with patching the HAProxy code. As I'd like to see a patch created that integrates the protocol for performing the dynamic weight update, currently the code I have only works under 1.3.14.11, due to the changes made to the Unix socket in later versions, this needs to be tweaked before it can be re-used by the community. Given time, its possible that the entire package (daemon and UI) can be released as an add-on to HAProxy. Cheers, Robert. On 16 Oct 2009, at 21:30, Mariusz Gronczewski wrote: 2009/10/16 Craig cr...@haquarter.de: Hi, a patch (set weight/get weight) I imagined some days ago was integrated just 6hrs after I had thought about it (Willy must be reading them!). I've written a simple (exchangable) interface that prints out a servers load and a client to read it. I plan to read the load from all servers and adjust the weight dynamically according to the load so that a very busy server gets less queries and the whole farm is more balanced. I plan to smoothen the increasing/decreasing a bit so that there aren't too great jumps with the weight, I want to implement a policy of something like oh that server can do 50% more, lets just increase the weight by 25% and check again in a minute. I hope this will autobalance servers with different hardware quite well, so that you don't have to guess or do performance tests to get the weights properly. Some python code is already finished (partly because I'd like to practise a bit) but I didn't continue yet, because I'd like to hear your opionions about this. Hi I'm sure lot of ppl (including me) will be interested in that ( I would probably have to write something similar in a month or so). By load you mean /proc/loadavg ? If yes then u might want to include some kind of per-server tuning of multiplier because obviously on 16 core server loadavg of 10 would be moderately loaded while on 4 core server i'd be overload Regards Mariusz
Re: dynamic weights based on actual server load
On Fri, Oct 16, 2009 at 01:36:52PM -0700, Hank A. Paulson wrote: For the code you are developing, if you make the interface general enough so that parameters can be added or removed that would be good. Telnet/text/memcached style protocols seem popular to allow easy debugging/monitoring. So if your protocol says a machine has to send a load info bundle like: SS:8cbed340118ddf87e2d8ca4352006572 SYSID: blah1 SAMPLETIME: 2009-10-14-22-00-03 CPU: 83.23343455 NETI: 134238.0232 NETO: 492283.6549 DISK: 433.232 ES:8cbed340118ddf87e2d8ca4352006572 It's a lot better to return these information in HTTP headers, because that can be added on top of any other resource that haproxy would check. (...) It is probably possible to just modify feedbackd to emit haproxy set weight commands. Don't do that ! The set weight is for the human, not for an automaton. The most interesting usage is set weight xx/xx 0 to disable a server before operating on it, or set weight xx/xx 50% to off-load it a bit if you see it start swapping. If you send an automated tool on it, it will constantly change your manually assigned values. More interesting, I think would be to combine a multiple load parameter (active connections, CPU, net in/out bytes, net in/out packets, disk io, etc) feedback system with the ideas from the NetBSD neural network scheduler, creating an ai based dynamic load balancing system. http://softlayer.dl.sourceforge.net/project/nnsched/docs/thesis/nnsched.pdf This is more possible now that we have multi core systems that would have some idle CPU resources available for the ai compute load. if doing some computations on that few parameters consumes even a measurable amount of CPU, then they are clearly wrong. We're not doing fractals here. Combining 3-4 parameters should not results on billions of operations. Willy
Re: Redirect_prefix question
Hi Chris, On Fri, Oct 16, 2009 at 02:50:44PM +0100, ch...@sargy.co.uk wrote: Hi Guys, I have a wildcard SSL certificate which I wish to do a redirect from http to https whilst maintaining the original subdomain requested, eg: http://sub.domain.com/page should be redirected to https://sub.domain.com/page I can't do this with redirect_prefix (it seems to require I know the destination in advance) Can anyone point me in the right direction for this? This would mean extract part of the Host: request header to build the Location: response header. We planned to implement that at one moment but it's still not done because there was no use of it yet. Do you have a lot of domains in your wildcard certificate ? In a first time you could write as many redirect rules as you have domains, if that's not too many. For instance : acl name1 hdr(host) -i name1.domain.com acl name2 hdr(host) -i name2.domain.com acl name3 hdr(host) -i name3.domain.com redirect prefix https://name1.domain.com if name1 redirect prefix https://name2.domain.com if name2 redirect prefix https://name3.domain.com if name3 See ? Using haproxy 1.3.19 on a Centos 5 machine. Be careful with 1.3.19, it has a bug causing a random crash if you forget to set some timeouts on some instances. So be careful to have a defaults section with at least client/server/connect timeouts set. Regards, Willy
Re: OT: murmurhash
On Fre 16.10.2009 06:32, Willy Tarreau wrote: Hi Aleks, On Fri, Oct 16, 2009 at 12:56:02AM +0200, Aleksandar Lazic wrote: Dear Listmember, does anybody have seen or used this 'new' hash? http://murmurhash.googlepages.com/ http://murmurhash.googlepages.com/MurmurHash2A.cpp For me it looks very promising ;-) I had already downloaded version 2 8 months ago but I was not interested because it assumes int-aligned input data, which is obviously wrong for text. On x86 this will not be a problem since the processor is able to fetch unaligned data, but on most RISC processors, you'll get an alignment fault when trying to fetch unaligned data. However, it might be interesting to test it with the consistent hash to place the servers. And if it's well spread, then we can re-implement it with bytes on input, so that it works everywhere. I will take a look in the next year ;-) You know this issues is a 'timekiller' ;-) BR Aleks
