On Mon, Mar 16, 2009 at 04:01:44PM -0400, John Marrett wrote:
> This is a somewhat off topic question, though I know that Willy (and
> hopefully other haproxy mailing list readers) have a good deal of
> expertise in the matter.
>
> We have been experiencing some performance issues with machines (running
> the latest RHEL4 kernel) handling large volumes of ftp traffic over our
> gigabit network.
Well, RHEL4 is based on an extremely old kernel which has never been
reknown for its reliability nor performance :-/
> We see indications of queue overruns, as follows:
>
> netstat -s
> [...]
> 403509572 segments received
> [...]
> 6015285 packets pruned from receive queue because of socket buffer
> overrun
You should probably slightly increase tcp_rmem (not too much, or you'll
go OOM). Since your kernel is based on the old buggy scheduler, it is
also possible that haproxy sometimes does not get CPU for several seconds
if its usage is close to 50% (one of the numerous design bugs of early
2.6 kernels).
> 67132 packets pruned from receive queue
> [...]
you should be able to reduce these by increasing netdev_max_backlog I
think.
> We also see indications of packet loss at the driver level:
>
> ifconfig -a
> [...]
> RX packets:17314770645 errors:0 dropped:98578 overruns:0
> frame:0
Same as above, probably netdev_max_backlog.
> - What would you recommend in terms of tuning, compared to the default
> values?
It's not easy to tell param X or Y. It depends a lot on the workload,
the NIC driver, etc... Usually you need to iterate over changes.
> - Are there any references on the subject that could be considered
> definitive? There's certainly no shortage of conflicting advice...
I once noticed that all tuning designed for benchmarks is pure crap.
They use huge network buffers that do not even fit into memory, and
disable some feature without even understanding what they're useful
to.
Here are a few rules of thumb :
- if you experience a high session rate, you need a large SYN
backlog (certainly more than the default 1024). You must then
tweak tcp_max_syn_backlog accordingly (at least the same value
as the higher frontend maxconn). You must also set somaxconn
to at least the same value (128 by default, far too low).
You also need to get large source port ranges (ip_local_port_range),
and preferably enable reusing of timewait sockets (tcp_tw_reuse=1).
- if you experience high bandwidth, you need to ensure that the
default value in tcp_rmem matches your most common buffer size,
and try to get tcp_wmem slightly higher.
- for high session raites with few servers, enable tcp_timestamps,
as this will enable PAWS, and solve session reuse problems.
- for front servers, enable tcp_sack. This will reduce the number
of retransmits. Also, you should reduce tcp_synack_retries so
that SYN floods don't turn your box into a flooder itself. Setting
that one to 2-3 is often enough.
There are other tricks that don't come to mind right now.
> Any advice that you and the other learned haproxy mailing list
> contributers could offer would be much appreciated.
>
> Willy,
>
> I know that you've commented on TCP tuning required to handle large
> volumes of traffic with HAProxy, and wonder if those same techniques
> might help us with our situation. You managed 10 gigabits (of contrived
> benchmark packets, to be sure), surely getting us up to 1 shouldn't be a
> problem ;)
1 gig can be huge, depending on the objects size. Try to stay below 10-20k
sessions per second if you don't want to spend your time tuning your system
or even patch your kernel ;-)
I'd say that running at 10k conn/s and 1 Gbps is completely achievable
on a recent machine, I know several people pushing more than twice that
in production with proper tuning ;-)
Regards,
Willy
