On Thu, 01 Sep 2011 03:09:56 +0200, Krzysztof Olędzki wrote:
Your concern is very valid and I think this is a moment where you
should take advantage of HAProxy, so you came to the right place. ;)
Each active session on HAProxy does not cost too much (much less than
on http server), so you may use "http-server-close" mode. You will
provide keep-alive to clients and only to clients - http requests
between LB and http server(s) will be handled without keep-alive.
HAproxy also gives you possibility to transparently distribute
requests (and load) between two or more servers, without additional
dns records.
As a matter of fact, I've been around for some time :) We are already
using haproxy but not as a web front end. Our architecture will look
like that once I had the secondary lighttpd
+-----------+
+--------+ +----> tomcat |
+-------->lighttpd+ | +-----------+
keep +--------| | +-----------+
alive +->+----------+----> tomcat |
|haproxy + +-----------+
+--------+->+----------| +-----------+
+-------->lighttpd| |----> tomcat |
keep +--------+ | +-----------+
alive | +-----------+
+----> tomcat |
+-----------+
We do not do any keepalive passed lighttpd. Everything is non-keepalive
because we need the x-forwarded-for header on each request for tomcat.
(that's yet another story)
Anyway, the reason why I don't want to put haproxy in front of
everything right now is because we have a shit load of rewrite rules and
it will take me forever to convert all of that into haproxy acl's
syntax.
Plus, whether it's lighttpd or haproxy, the memory impact on the kernel
is going to be the same.
Right now, I'm considering putting a keepalive timeout at 5 seconds.
Maybe even less.
As a side question: do you know where I can find the information
regarding connection size and conntrack size in the kernel ? (other
than
printf size_of(sk_buff) :p).
Conntrack has nothing to do with sk_buff. However, you are able to
find this information with for example:
# egrep "(#|connt)" /proc/slabinfo
Very nice ! I'll read about slabinfo. Here is the result from the
current lighttpd server
# name <active_objs> <num_objs> <objsize> <objperslab>
<pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata
<active_slabs> <num_slabs> <sharedavail>
ip_conntrack_expect 0 0 136 28 1 : tunables 120 60
8 : slabdata 0 0 0
ip_conntrack 23508 39767 304 13 1 : tunables 54 27
8 : slabdata 3059 3059 15
It should be around 208 bytes on x86 and 264 bytes on x86_64 (2 x
longer pointers), but this is not all. Each conntrack can have some
additional data attached, which is known as "extends". Currently
there
are 5 possible extends:
- helper - struct nf_conn_help: 16 bytes (x86) / 24 bytes (x86_64)
- nat - struct nf_conn_nat: 16 bytes (x86) / 24 bytes (x86_64)
- acct - struct nf_conn_counter: 16 bytes
- ecache - struct nf_conntrack_ecache: 16 bytes
- zone - struct nf_conntrack_zone: 2 bytes
So, as you can see, in the worst case there can be 66 / 82 more bytes
allocated with each conntrack and this goes into kmalloc-x slab that
rounds it into 2^n bytes.
That's for conntrack only ? This is pretty low (346 bytes max per
connection), I thought conntrack would consume more than that.
What about the sk_buff and other structure? I didn't dive in the
networking layer for some time. I don't need an exact number, but just
an idea of how much memory we are talking about.
I know some of that is configurable (read and write buffers in sysctl,
etc...) I'll get to that at some point and put some reasonable
parameters.
Thanks for your help :)
Julien
