Re: dynamic weights based on actual server load
Hi, On Mon, Oct 19, 2009 at 06:10:24PM +0300, Balcoes wrote: Hi. This discussion got me thinking would it be possible for HAproxy to query a PHP/Perl/Python...-script that returns some kind of numerical information concerning the server's load. For example HAproxy could check periodically http://server_name/load_status where the 'load_status'-script would return something like performance_index 0.92 Ideally the performance_index should be 1.00, but under heavy load the index would be lower. HAproxy could then multiply the basic server weight with this performance_index. For example there could be three servers server Aserver 192.168.0.1:80 cookie Acookie check inter 2000 weight 100 server Bserver 192.168.0.2:80 cookie Bcookie check inter 2000 weight 100 server Cserver 192.168.0.3:80 cookie Ccookie check inter 2000 weight 80 If all servers are working optimally under light load all of them would return a performance_index of 1.00. However if server A should be under heavy load, it would return performance_index 0.5. The server B is somewhat less loaded returning performance_index 0.8 and C lightly loaded returning 0.95 the dynamic weights would be server A: 100 * 0.5 = 50 server B: 100 * 0.8 = 80 server C:80 * 0.95 = 76 The beauty is that every administrator could tweak the 'load_status'-script at his/her heart's will taking into consideration just his/hers own server farm's typical use and properties by choosing what performance data (CPU usage, CPU load, io_wait, average/maximum response time...) he/she wants to use when calculating the performance_index. Best, B This is exactly what was planned when the dynamic weights were implementated in 1.3.15, but finally there was a choice of the customer not to use them that way because of possible disagreements between the people in charge of the servers and the people in charge of the LB in case of issues caused by an unbalanced farm. The most basic principle simply is to check the idle CPU ratio, but then you can announce whatever you want. Also, don't forget that you don't want a value from the server to be immediately turned into a weight. A weight only has a meaning compared to other weights. So you want to adjust each server's weight relative to its available capacity vs global capacity, and its weight vs total weight. It's indeed useless and even undesirable to adjust weights when servers are properly balanced and their load varies homogeneously. Willy
Re: dynamic weights based on actual server load
Hello, On Sat, Oct 17, 2009 at 11:18:24AM +0200, Angelo Höngens wrote: Just read this thread, and I thought I would give my humble opinion on this: As a hosting provider we use both windows and unix backends, en we use haproxy to balance requests across sites on a per-site backend (with squid in front of haproxy). What I would love to see, is dynamic balancing based on the round-trip time of the health check. So when a backend is slower to respond, the weight should go down (slowly), so the faster servers would get more requests. Now that's a feature I'd love to see.. And then there would not be anything to configure on the backend (we don't always have control over the backend application) Having already seen this on another equipment about 5-6 years ago, I can tell you this does not work at all. The reason is simple : the health checks should always be fast on a server, and their response time almost never tells anything about the server's remaining capacity. Some people even use static files as health checks. What is needed though is to measure real traffic's response time. The difficulty comes from the fact that if you lower the weight too much, there is too little traffic to measure a reduced response time, and it is important to be able to bound the window in which the weight evolves. Regards, Willy
Re: dynamic weights based on actual server load
Hi On Sun, 18 Oct 2009 08:26:37 +0200, Willy Tarreau w...@1wt.eu wrote: Hello, On Sat, Oct 17, 2009 at 11:18:24AM +0200, Angelo Höngens wrote: Just read this thread, and I thought I would give my humble opinion on this: As a hosting provider we use both windows and unix backends, en we use haproxy to balance requests across sites on a per-site backend (with squid in front of haproxy). What I would love to see, is dynamic balancing based on the round-trip time of the health check. So when a backend is slower to respond, the weight should go down (slowly), so the faster servers would get more requests. Now that's a feature I'd love to see.. And then there would not be anything to configure on the backend (we don't always have control over the backend application) Having already seen this on another equipment about 5-6 years ago, I can tell you this does not work at all. The reason is simple : the health checks should always be fast on a server, and their response time almost never tells anything about the server's remaining capacity. Some people even use static files as health checks. What is needed though is to measure real traffic's response time. The difficulty comes from the fact that if you lower the weight too much, there is too little traffic to measure a reduced response time, and it is important to be able to bound the window in which the weight evolves. For healthcheck-based loadbalancing healthchecks had to be something like make-some-number-crunching then make some-database-reading and quite long (because shorter requests tend to be more random). So you will have either have long time between checks or, when checks will be more often, lot of load on server thanks to health-checks. I think loadbalancing should be both done by request length and server load, but then it would have to be some kind of long term log analysing of one often used part of page, for example: 1.If server load (simplest measure will be loadavg/cores) is below 80%, increase weigth 2.If average request time of http://example.org/index.php is less than 90% of target_request_time, increase weight a bit. 3.If avg. req. time is more than target_request_time * 1.1 decrease weight a bit 4. every x minutes if weigth will be less than 50 add 1 if more subtract 50 (so values won't be drifting to max or 0 over time) target request time would be some predefined value or (better) calculated average from all nodes + 50% so u won't have situation where every node weigth is skyrocketing or falling down because of small load/overload Regards Mariusz -- Mariusz Gronczewski (XANi) xani...@gmail.com GnuPG: 0xEA8ACE64 http://devrandom.pl signature.asc Description: PGP signature
RE: dynamic weights based on actual server load
Just read this thread, and I thought I would give my humble opinion on this: As a hosting provider we use both windows and unix backends, en we use haproxy to balance requests across sites on a per-site backend (with squid in front of haproxy). What I would love to see, is dynamic balancing based on the round-trip time of the health check. So when a backend is slower to respond, the weight should go down (slowly), so the faster servers would get more requests. Now that's a feature I'd love to see.. And then there would not be anything to configure on the backend (we don't always have control over the backend application) Just my 2 cents. -- With kind regards, Angelo Höngens Systems Administrator -- NetMatch tourism internet software solutions Ringbaan Oost 2b 5013 CA Tilburg T: +31 (0)13 5811088 F: +31 (0)13 5821239 mailto:a.hong...@netmatch.nl http://www.netmatch.nl -- -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: vrijdag 16 oktober 2009 22:59 To: Hank A. Paulson Cc: Craig; haproxy@formilux.org Subject: Re: dynamic weights based on actual server load On Fri, Oct 16, 2009 at 01:36:52PM -0700, Hank A. Paulson wrote: For the code you are developing, if you make the interface general enough so that parameters can be added or removed that would be good. Telnet/text/memcached style protocols seem popular to allow easy debugging/monitoring. So if your protocol says a machine has to send a load info bundle like: SS:8cbed340118ddf87e2d8ca4352006572 SYSID: blah1 SAMPLETIME: 2009-10-14-22-00-03 CPU: 83.23343455 NETI: 134238.0232 NETO: 492283.6549 DISK: 433.232 ES:8cbed340118ddf87e2d8ca4352006572 It's a lot better to return these information in HTTP headers, because that can be added on top of any other resource that haproxy would check. (...) It is probably possible to just modify feedbackd to emit haproxy set weight commands. Don't do that ! The set weight is for the human, not for an automaton. The most interesting usage is set weight xx/xx 0 to disable a server before operating on it, or set weight xx/xx 50% to off-load it a bit if you see it start swapping. If you send an automated tool on it, it will constantly change your manually assigned values. More interesting, I think would be to combine a multiple load parameter (active connections, CPU, net in/out bytes, net in/out packets, disk io, etc) feedback system with the ideas from the NetBSD neural network scheduler, creating an ai based dynamic load balancing system. http://softlayer.dl.sourceforge.net/project/nnsched/docs/thesis/nnsched.pdf This is more possible now that we have multi core systems that would have some idle CPU resources available for the ai compute load. if doing some computations on that few parameters consumes even a measurable amount of CPU, then they are clearly wrong. We're not doing fractals here. Combining 3-4 parameters should not results on billions of operations. Willy
Re: dynamic weights based on actual server load
2009/10/16 Craig cr...@haquarter.de: Hi, a patch (set weight/get weight) I imagined some days ago was integrated just 6hrs after I had thought about it (Willy must be reading them!). I've written a simple (exchangable) interface that prints out a servers load and a client to read it. I plan to read the load from all servers and adjust the weight dynamically according to the load so that a very busy server gets less queries and the whole farm is more balanced. I plan to smoothen the increasing/decreasing a bit so that there aren't too great jumps with the weight, I want to implement a policy of something like oh that server can do 50% more, lets just increase the weight by 25% and check again in a minute. I hope this will autobalance servers with different hardware quite well, so that you don't have to guess or do performance tests to get the weights properly. Some python code is already finished (partly because I'd like to practise a bit) but I didn't continue yet, because I'd like to hear your opionions about this. Hi I'm sure lot of ppl (including me) will be interested in that ( I would probably have to write something similar in a month or so). By load you mean /proc/loadavg ? If yes then u might want to include some kind of per-server tuning of multiplier because obviously on 16 core server loadavg of 10 would be moderately loaded while on 4 core server i'd be overload Regards Mariusz
Re: dynamic weights based on actual server load
For the code you are developing, if you make the interface general enough so that parameters can be added or removed that would be good. Telnet/text/memcached style protocols seem popular to allow easy debugging/monitoring. So if your protocol says a machine has to send a load info bundle like: SS:8cbed340118ddf87e2d8ca4352006572 SYSID: blah1 SAMPLETIME: 2009-10-14-22-00-03 CPU: 83.23343455 NETI: 134238.0232 NETO: 492283.6549 DISK: 433.232 ES:8cbed340118ddf87e2d8ca4352006572 that would give you a generic record format that anyone could create a client or modify your client and add/remove load parameter info fields. Then they just take their list of load parameters and wrap them in the header and footer with the id of that sample record being the md5sum of the included data. SS = start sample record/ES = end sample record Then you could have a separate/pluggable module or process that takes that info and maybe pulls other data from system history files, etc and decides what to set the weight to for that server. You could provide a default weight setter engine that uses a simple algo based on just CPU load or something and others could fill in more complex/custom engines if desired. You might want to check out feedbackd: http://ozlabs.org/~jk/projects/feedbackd/ and his paper on Using Dynamic Feedback to Optimise Load Balancing Decisions http://redfishsoftware.com.au/projects/feedbackd/lca-paper.pdf to get some ideas. It is probably possible to just modify feedbackd to emit haproxy set weight commands. More interesting, I think would be to combine a multiple load parameter (active connections, CPU, net in/out bytes, net in/out packets, disk io, etc) feedback system with the ideas from the NetBSD neural network scheduler, creating an ai based dynamic load balancing system. http://softlayer.dl.sourceforge.net/project/nnsched/docs/thesis/nnsched.pdf This is more possible now that we have multi core systems that would have some idle CPU resources available for the ai compute load. On 10/16/09 10:29 AM, Craig wrote: Hi, a patch (set weight/get weight) I imagined some days ago was integrated just 6hrs after I had thought about it (Willy must be reading them!). I've written a simple (exchangable) interface that prints out a servers load and a client to read it. I plan to read the load from all servers and adjust the weight dynamically according to the load so that a very busy server gets less queries and the whole farm is more balanced. I plan to smoothen the increasing/decreasing a bit so that there aren't too great jumps with the weight, I want to implement a policy of something like oh that server can do 50% more, lets just increase the weight by 25% and check again in a minute. I hope this will autobalance servers with different hardware quite well, so that you don't have to guess or do performance tests to get the weights properly. Some python code is already finished (partly because I'd like to practise a bit) but I didn't continue yet, because I'd like to hear your opionions about this. Am I mad? ;) Best Regards, Craig
Re: dynamic weights based on actual server load
Hello, Before anyone goes too far into creating things here, I should probably point out that I have already written a system that handles this feature based upon CPU, Memory and Network usage, it was my University Dissertation! Since that has only just been marked, I haven't previously been able to discuss or release any information of this to the community. In simple terms, I have created a fairly simple daemon that monitors the CPU, Memory and Network usage of servers using the SNMP protocol. The statistics gathered are stored in a MySQL database, which allows the graphical web user interface to graph these statistics. Whilst I can go into further details at a later date, I'd like to request some assistance from someone in the community who has more experience with patching the HAProxy code. As I'd like to see a patch created that integrates the protocol for performing the dynamic weight update, currently the code I have only works under 1.3.14.11, due to the changes made to the Unix socket in later versions, this needs to be tweaked before it can be re-used by the community. Given time, its possible that the entire package (daemon and UI) can be released as an add-on to HAProxy. Cheers, Robert. On 16 Oct 2009, at 21:30, Mariusz Gronczewski wrote: 2009/10/16 Craig cr...@haquarter.de: Hi, a patch (set weight/get weight) I imagined some days ago was integrated just 6hrs after I had thought about it (Willy must be reading them!). I've written a simple (exchangable) interface that prints out a servers load and a client to read it. I plan to read the load from all servers and adjust the weight dynamically according to the load so that a very busy server gets less queries and the whole farm is more balanced. I plan to smoothen the increasing/decreasing a bit so that there aren't too great jumps with the weight, I want to implement a policy of something like oh that server can do 50% more, lets just increase the weight by 25% and check again in a minute. I hope this will autobalance servers with different hardware quite well, so that you don't have to guess or do performance tests to get the weights properly. Some python code is already finished (partly because I'd like to practise a bit) but I didn't continue yet, because I'd like to hear your opionions about this. Hi I'm sure lot of ppl (including me) will be interested in that ( I would probably have to write something similar in a month or so). By load you mean /proc/loadavg ? If yes then u might want to include some kind of per-server tuning of multiplier because obviously on 16 core server loadavg of 10 would be moderately loaded while on 4 core server i'd be overload Regards Mariusz
Re: dynamic weights based on actual server load
On Fri, Oct 16, 2009 at 01:36:52PM -0700, Hank A. Paulson wrote: For the code you are developing, if you make the interface general enough so that parameters can be added or removed that would be good. Telnet/text/memcached style protocols seem popular to allow easy debugging/monitoring. So if your protocol says a machine has to send a load info bundle like: SS:8cbed340118ddf87e2d8ca4352006572 SYSID: blah1 SAMPLETIME: 2009-10-14-22-00-03 CPU: 83.23343455 NETI: 134238.0232 NETO: 492283.6549 DISK: 433.232 ES:8cbed340118ddf87e2d8ca4352006572 It's a lot better to return these information in HTTP headers, because that can be added on top of any other resource that haproxy would check. (...) It is probably possible to just modify feedbackd to emit haproxy set weight commands. Don't do that ! The set weight is for the human, not for an automaton. The most interesting usage is set weight xx/xx 0 to disable a server before operating on it, or set weight xx/xx 50% to off-load it a bit if you see it start swapping. If you send an automated tool on it, it will constantly change your manually assigned values. More interesting, I think would be to combine a multiple load parameter (active connections, CPU, net in/out bytes, net in/out packets, disk io, etc) feedback system with the ideas from the NetBSD neural network scheduler, creating an ai based dynamic load balancing system. http://softlayer.dl.sourceforge.net/project/nnsched/docs/thesis/nnsched.pdf This is more possible now that we have multi core systems that would have some idle CPU resources available for the ai compute load. if doing some computations on that few parameters consumes even a measurable amount of CPU, then they are clearly wrong. We're not doing fractals here. Combining 3-4 parameters should not results on billions of operations. Willy
