For the code you are developing, if you make the interface general enough so
that parameters can be added or removed that would be good.
Telnet/text/memcached style protocols seem popular to allow easy
debugging/monitoring.
So if your protocol says a machine has to send a load info bundle like:
SS:8cbed340118ddf87e2d8ca4352006572
SYSID: blah1
SAMPLETIME: 2009-10-14-22-00-03
CPU: 83.23343455
NETI: 134238.0232
NETO: 492283.6549
DISK: 433.232
ES:8cbed340118ddf87e2d8ca4352006572
that would give you a generic record format that anyone could create a client
or modify your client and add/remove load parameter info fields. Then they
just take their list of load parameters and wrap them in the header and footer
with the id of that sample record being the md5sum of the included data. SS =
start sample record/ES = end sample record
Then you could have a separate/pluggable module or process that takes that
info and maybe pulls other data from system history files, etc and decides
what to set the weight to for that server. You could provide a default "weight
setter" engine that uses a simple algo based on just CPU load or something and
others could fill in more complex/custom engines if desired.
You might want to check out feedbackd:
http://ozlabs.org/~jk/projects/feedbackd/
and his paper on "Using Dynamic Feedback to Optimise Load Balancing Decisions"
http://redfishsoftware.com.au/projects/feedbackd/lca-paper.pdf to get some
ideas.
It is probably possible to just modify feedbackd to emit haproxy "set weight"
commands.
More interesting, I think would be to combine a multiple load parameter
(active connections, CPU, net in/out bytes, net in/out packets, disk io, etc)
feedback system with the ideas from the NetBSD neural network scheduler,
creating an ai based dynamic load balancing system.
http://softlayer.dl.sourceforge.net/project/nnsched/docs/thesis/nnsched.pdf
This is more possible now that we have multi core systems that would have some
idle CPU resources available for the ai compute load.
On 10/16/09 10:29 AM, Craig wrote:
Hi,
a patch (set weight/get weight) I imagined some days ago was integrated
just 6hrs after I had thought about it (Willy must be reading them!).
I've written a simple (exchangable) interface that prints out a servers
load and a client to read it. I plan to read the load from all servers
and adjust the weight dynamically according to the load so that a very
busy server gets less queries and the whole farm is more balanced. I
plan to "smoothen" the increasing/decreasing a bit so that there aren't
too great jumps with the weight, I want to implement a policy of
something like "oh that server can do 50% more, lets just increase the
weight by 25% and check again in a minute". I hope this will autobalance
servers with different hardware quite well, so that you don't have to
guess or do performance tests to get the weights properly.
Some python code is already finished (partly because I'd like to
practise a bit) but I didn't continue yet, because I'd like to hear your
opionions about this.
Am I mad? ;)
Best Regards,
Craig
