Claudio Jeker wrote:
On Wed, May 09, 2007 at 06:41:27PM -0400, Daniel Ouellet wrote:
Hi,
I am passing my finding around for the configuration of sysctl.conf to
remove bottleneck I found in httpd as I couldn't get more then 300 httpd
process without crapping out badly and above that, the server simply got
out of wack.
<SNIP>
===========================
sysctl.conf changes.
kern.seminfo.semmni=1024
kern.seminfo.semmns=4096
kern.shminfo.shmall=16384
kern.maxclusters=12000
What does netstat -m tell you about the peak usage of clusters is it
really that high?
I will do an other series of tests in the next few days and be sure of
it before putting my foot in my mouth. But at 10000, I was getting drops
in my test setup.
kern.maxproc=2048 # Increase for the process limits.
kern.maxfiles=5000
kern.shminfo.shmmax=67108864
kern.somaxconn=2048
Is httpd really so slow in accepting sockets that you had to increase this
by factor 16? Is httpd actually doing a listen with such a large number?
Yes, I was doing tests using a few clients and pushing the server at
2000 parallel connections to test with. That was in lab test and in real
life, I assume that half should be fine. But I wanted to be safe. So,
place for review on my side.
net.bpf.bufsize=524288
As tedu@ pointed out this has nothing todo with your setup.
Agreed before and was removed after more reading. You are right.
net.inet.ip.maxqueue=1278
Are you sure you need to tune the IP fragment queue? You are using TCP
which does PMTU discovery and sets the DF flag by default so no IP
fragments should be seen at all unless you borked something else.
With smaller queue I was getting slower responses and drop. May be a
need a better way to verify this situation for a fact.
net.inet.ip.portfirst=32768
net.inet.ip.redirect=0
This has no effect unless you enable forwarding.
Was removed as well.
net.inet.tcp.keepinittime=10
net.inet.tcp.keepidle=30
net.inet.tcp.keepintvl=30
These values are super aggressive especially the keepidle and keepintvl
values are doubtful for your test. Is your benchmark using SO_KEEPALIVE? I
doubt that and so these two values have no effect and are actually
counterproductive (you are sending more packets for idle sessions).
Yes, aggressive I was/am. Keep Alive was/is in use yes. I will have more
to play with in lab and see if I was to aggressive and look like you
would think I am. The default value give me not as good results however.
More tests needed specifically on this and I will do so. May be the
defaults are fine, I will see if I can find a way to be more objective
about these values.
net.inet.tcp.mssdflt=1452
This is another knob that should not be changed unless you really know
what you are doing. The mss calculation uses this value as safe default
that is always accepted. Pushing that up to this value may have unpleasant
sideeffects for people behind IPSec tunnels. The used mss is the max
between mssdflt and the MTU of the route to the host minus IP and TCP
header.
I will review and read more on it. I based my changes on results seen
with the setup under heavy load. There is always place for improvements.
This gives me more to consider and will do so.
net.inet.tcp.recvspace=65535
net.inet.tcp.sendspace=65535
net.inet.tcp.rstppslimit=400
net.inet.tcp.synbucketlimit=420
net.inet.tcp.syncachelimit=20510
If you need to tune the syncache in such extrem ways you should consider
to adjust TCP_SYN_HASH_SIZE and leave synbucketlimit as is. The
synbucketlimit is here to limit attacks to the hash list by overloading
the bucket list. On your system it may be necessary to traverse 420 nodes
on a lookup. Honestly the syncachelimit and synbucketlimit knob are totaly
useless. If anything we should allow to resize the hash and calculate the
both limits from there.
Interesting! I will retest with that in mind. Didn't see that
explication in my reading so far. Thanks for this!
You are most helpful and this gives me something to research more and I
sure appreciates your time in passing the informations.
Looks like a few more days of testing needed.
Many thanks!
Daniel
