On 30-12-2009 14:04, Marcus Herou wrote:
Hi Willy, thanks for your answer it got filtered, that's why I missed it
for two weeks.
Let's start with describing the service.
We are hosting javascripts of the sizes up to 20K and serve flash and
image banners as well which of course are larger. That is basically it..
Ad Serving.
On the LB's we have about 2MByte/s per LB = 2x2MByte/s = 4MByte/s
~30MBit/s at peak, that is not the issue.
I've created a little script which parse the active connections from
the HAProxy stat interface and plots it into Cacti, it peaks at 100
(2x100) connections per machine which is very little in your world I guess.
I've attached a plot of tcp-connections as well. Nothing fancy there
either besides that the number of TIME_WAIT sockets are in the 1
range (log scale)
Here's the problem:
Everyother day I receive alarms from Pingdom that the service is not
available and if I watch the syslog I get at about the same timings
hints about possible SYN flood. At the same timings we receive emails
from sites using us that our service is damn slow.
What I feel is that we get hickups on the LB's somehow and that
requests get queued. If I count the number of rows in the access logs on
the machines behind the LB it decreases at the same timings and with the
same factor on each machine (perhaps 10-20%) leading me to think that
the narrow point is not on the backend side.
Maybe interesting, maybe not: I had some problems like this as well and
in my case I think this was caused by the limited number of outgoing
ports from my proxy machines..
I don't use connection keep-alives, and I think my balancers were
reusing ports faster than the backend windows machines could handle or
something like that.. Anyway, after I changed my FreeBSD's available
outgoing ports range, all problems were solved again.
Here's my sysctl for my FreeBSD 7.2 machines, but as Willy said, this
might not work for everyone.
kern.maxfiles=65535
kern.maxfilesperproc=32767
kern.ipc.maxsockbuf=16777216
kern.ipc.somaxconn=32768
net.inet.tcp.rfc1323=1
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.inflight.enable=0
net.inet.tcp.hostcache.expire=1
net.inet.ip.portrange.first=1024
net.inet.ip.portrange.last=65535
net.inet.ip.portrange.hifirst=49152
net.inet.ip.portrange.hilast=65535
@Willy or someone else, feel free to comment on these settings if you
see something strange.
--
With kind regards,
Angelo Höngens
systems administrator
MCSE on Windows 2003
MCSE on Windows 2000
MS Small Business Specialist
--
NetMatch
tourism internet software solutions
Ringbaan Oost 2b
5013 CA Tilburg
+31 (0)13 5811088
+31 (0)13 5821239
a.hong...@netmatch.nl
www.netmatch.nl
--
