Hi, I'm an admin for a non-profit that runs Tor exit nodes, most of them on Linux currently, but due to problems on our high bandwidth nodes, we decided to migrate one of them to FreeBSD to do some testing. I've been using FreeBSD for quite some years now, longer than Linux, so I figured this would probably be a no-brainer, but turns out, that it isn't.
On FreeBSD I currently manage to push 150-200Mbits with some heavy tuning
applied already, on Linux it is roughly 500Mbits.
Therefor I'm wondering if I'm really running into some limitations here or if
I'm actually doing something wrong.
ifconfig bge0
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c01db<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,POLLING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
As you can see I've compiled polling and currently I'm running on kern.hz=16000
as that's given me the best performance, so far.
This is my netstat output for tcp on ipv4:
netstat -s
tcp:
265086752 packets sent
84255022 data packets (155791599035 bytes)
2244601 data packets (2698410010 bytes) retransmitted
73011 data packets unnecessarily retransmitted
684 resends initiated by MTU discovery
153151729 ack-only packets (0 delayed)
0 URG only packets
19864 window probe packets
16854982 window update packets
8551532 control packets
236720260 packets received
86600847 acks (for 155842386836 bytes)
5062568 duplicate acks
76267 acks for unsent data
138258588 packets (150041170335 bytes) received in-sequence
1502804 completely duplicate packets (562243206 bytes)
44193 old duplicate packets
91821 packets with some dup. data (23693391 bytes duped)
6598536 out-of-order packets (7168950882 bytes)
32074 packets (20848106 bytes) of data after window
7806 window probes
1457705 window update packets
159860 packets received after close
5219 discarded for bad checksums
3 discarded for bad header offset fields
0 discarded because packet too short
1468 discarded due to memory problems
5665849 connection requests
694088 connection accepts
0 bad connection attempts
129 listen queue overflows
9308 ignored RSTs in the windows
3289250 connections established (including accepts)
6334698 connections closed (including 449721 drops)
916420 connections updated cached RTT on close
923786 connections updated cached RTT variance on close
273103 connections updated cached ssthresh on close
354989 embryonic connections dropped
81015541 segments updated rtt (of 56772442 attempts)
9127304 retransmit timeouts
19875 connections dropped by rexmit timeout
21274 persist timeouts
215 connections dropped by persist timeout
4541 Connections (fin_wait_2) dropped because of timeout
10657 keepalive timeouts
0 keepalive probes sent
10657 connections dropped by keepalive
39113689 correct ACK header predictions
121244352 correct data packet header predictions
698461 syncache entries added
21576 retransmitted
14546 dupsyn
0 dropped
694088 completed
0 bucket overflow
0 cache overflow
1046 reset
3338 stale
135 aborted
0 badack
6 unreach
0 zone failures
698461 cookies sent
232 cookies received
173007 hostcache entries added
0 bucket overflow
285122 SACK recovery episodes
584154 segment rexmits in SACK recovery episodes
730380132 byte rexmits in SACK recovery episodes
3053612 SACK options (SACK blocks) received
6689960 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
8236 packets with ECN CE bit set
26367032 packets with ECN ECT(0) bit set
56 packets with ECN ECT(1) bit set
312220 successful ECN handshakes
34178 times ECN reduced the congestion window
My sysctls are roughly equivalent to these:
http://serverfault.com/questions/64356/freebsd-performance-tuning-sysctls-loader-conf-kernel
Any hints?
Do I need to provide more info?
Julian
signature.asc
Description: Message signed with OpenPGP using GPGMail
