Hi all,
here is probably the largest update we ever had, it's composed of 345
patches!
Some very difficult changes had to be made and as usual when such changes
happen, they take a lot of time due to the multiple attempts at getting
them right, and as time goes, people submit features :-)
After two weeks spent doing only fixes, I thought it was time to issue dev20.
I'm sure I'll forget a large number of things, but the main features of this
version include the following points (in merge order) :
- optimizations (splicing, polling, etc...) : a few percent CPU could be
saved ;
- memory : the connections and applets are now allocated only when needed.
Additionally, some structures were reorganized to avoid fragmentation on
64-bit systems. In practice, an idle session size has dropped from 1936
bytes to 1296 bytes (-640 bytes, or -33%).
- samples : all sample fetch expressions now support a comma-delimited
list of converters. This is also true in ACLs, so that it becomes
possible to do things like :
# convert to lower case and use fast tree indexing
acl known_domain hdr(host),lower -f huge-domain-list.lst
- a lot of code has been deduplicated in the tracked counters, it's now
possible to use sc_foo_bar(1, args) instead of sc1_foo_bar(args). Doing
so has simplified the code and makes life of APIs easier.
- it's now possible to look up a tracked key from another table. This allows
to retrieve multiple counters for the same key.
- several hash algorithms are provided, and it is possible to select them
per backend. This high quality work was done at Tumblr by Bhaskar Maddala.
- agent-checks: this new feature was merged and replaced the lb-agent-chk.
Some changes are still planned but feedback is welcome. The goal of this
agent is to retrieve soem weight information from a server independantly
of the service health. A typical usage would consist in reporting the
server's idle percentage as an estimate of the possible weight. This work
was done by Simon Horman for Loadbalancer.org.
- samples : more automatic conversions between types are supported, making
it easier to stick to any parameter. The types are much more dynamic now.
Some improvements are still pending. This work was done by Thierry Fournier
at Exceliance.
- map : a new type of converter appeared : maps. A map matches a key from
a file just like ACLs do, and replaces this value with the value associated
with the key on the same line of the file. As it is a converter, it can be
used in any sample expression. The first usage consists in geolocation,
where networks are associated with country codes. Maps may be consulted,
deleted, updated and filled from the CLI. Some will probably use this to
program actions or emulate ACLs without even reloading a config. This
work was also achieved by Thierry Fournier, and reviewed by Cyril Bonté
who developped the original Geoip patchset for 1.4 and 1.5.
- http-request redirect now supports log-format like expressions, just like
http-request add-header. This allows to emit strings extracted from the
request (host header, country code from a map, ...). Thierry again here.
- checks: tcp-check supports send/expect sequences with strings/regex/binary.
Thus it now becomes possible to check unsupported protocols, even binary.
This work is from Baptiste Assmann.
- keep-alive: the dynamic allocation of the connection and applet in the
session now allows to reuse or kill a connection that was previously
associated with the session. Thus we now have a very basic support for
keep-alive to the servers. There is even an option to relax the load
balancing to try to keep the same connection. Right now we don't do
any connection sharing so the main use is for static servers and for
far remote servers or those which require the broken NTLM auth. That
said, the performance tests I have run show an increase from 71000
connections per second to 150000 keep-alive requests per second running
on one core of a Xeon E5 3.6 GHz. This doubled to 300k requests per
second with two cores. I didn't test above, I lacked injection tools :-)
One good point is that it will help people assemble haproxy and varnish
together with haproxy doing the consistent hash and varnish caching after
it.
As most of you know, server-side keep-alive is the condition to release 1.5.
Now we have it, we'll be able to improve on it but it's basically working.
I expect to release 1.5-final around January and mostly focus on chasing
bugs till there. So I'd like to set a feature freeze. I know it doesn't
mean much considering that we won't stop contribs. But I don't want to
merge another large patch set before the release. Ideally there will not
be any dev21 version. Reality probably is that we'll have to issue one
because people will inevitably report annoying bugs that were not reported
in snapshots.
I still have to perform difficult changes on the health checks system to
adapt the agent-check after identifying some limitations caused by the
original design we decided on a few months ago.
Another set of pending changes concerns the polling. Yes I know, each time
I touch the pollers I break things. But I need to do them, as now with
keep-alive it becomes obvious that we waste too much time enabling and
disabling polling because we miss one flag ("this FD last returned EAGAIN").
The good point is that it will simplify the connection management and checks.
If these points are done quick enough, I'll see if I can implement a very
basic support for server connection sharing connection (without which I
still consider keep-alive as not a huge improvement).
Note to users of 1.5-dev19 who hesitate to test this one : 71 bugs were fixed
since dev19. Others probably added, but hopefully less.
Usual links below :
Site index : http://haproxy.1wt.eu/
Sources : http://haproxy.1wt.eu/download/1.5/src/devel/
Changelog : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
Cyril's HTML doc :
http://cbonte.github.com/haproxy-dconv/configuration-1.5.html
I'm not appending the changelog, it's too large.
Have fun and please report issues here,
Willy
