What are your iperf command lines?
A standard iperf cmd line won't give line rate, the ones we use here are:-
== Server ==
iperf -s -w 2.5M -l 2.5M
== Client ==
iperf -i 10 -t 20 -c <server> -w 2.5M -l 2.5M
== Tuning ==
We use the following tuning on our machines to achieve line rate Gig on 7.0
amd64
net.inet.tcp.inflight.enable=0
net.inet.tcp.sendspace=65536
kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
Out of curiousity I just tried this very test on an 8.0 box we have here and
was only
able to achieve similar performance to yourself. So it may be the case that
there
has been a significant regression since 7.0, I'll have to do some more tests
when
I have time.
For reference the machines we have tested and get line rate on have the
following
nic's
== Machine #1 7.0-RELEASE amd64 ==
em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0x2000-0x201f mem 0xd8400000-0xd841ffff irq 18 at device 0.0 on
pci6
em0: Using MSI interrupt
em0: Ethernet address: .....
em0: [FILTER]
== Machine #2 7.0-RELEASE amd64 ==
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x2100> mem
0xfc9f0000-0xfc9fffff irq 26 at device 5.0 on pci3
miibus0: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bge0: Ethernet address: ....
bge0: [ITHREAD]
The machine which is currently under performing
== Machine #3 8.0-RELEASE amd64 ==
em0: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x2000-0x201f mem
0xd8400000-0xd841ffff irq 18 at device 0.0 on pci6
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:30:48:33:ec:44
Regards
Steve
----- Original Message -----
From: "Noisex" <[email protected]>
To: <[email protected]>
Sent: Thursday, December 10, 2009 2:28 PM
Subject: RE: FreeBSD TCP tuning and performance
This server is basically used for bandwidth testing (Our company is internet service provider) and we need and our clients
something to measure a internet speed.
Actually on that server is hosted MyConnection SpeedServer (http://www.visualware.com/) for bandwidth tests. Some month ago we
started to clients offer GPON with speed 500Mbit/500mbit.
While we offer DSL technology and speeds was up to 100Mbit it was pretty good results...but now we can't measure a speed on 1Gbps
link...the results is very pure (almost half from real on FreeBSD).
This results aren't only with MyConnection..also with nuttcp and iperf...i have feeling, that FreBSD can't handle window size
send/receive buffer
p.s maybe I should enable Network Pooling/disable interrupts on network card etc things? What could be a recommendations to set up
for maximum performance TCP settings on 1/10Gbps interface with sysctl parameters?
Noisex
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Steven Hartland
Sent: ceturtdiena, 2009. gada 10. decembrī 15:20
To: Noisex; [email protected]
Subject: Re: FreeBSD TCP tuning and performance
What app are you using there and is it setting the send / receive buffers
correctly?
----- Original Message -----
From: "Noisex" <[email protected]>
To: <[email protected]>
Sent: Monday, December 07, 2009 12:41 PM
Subject: FreeBSD TCP tuning and performance
Hi! I have a problem with TCP performance on FBSD boxes with 1Gbps net i-faces
(Broadcom NetXtreme II BCM5708 1000Base-T (B2)).
Currently i use FBSD 7.1 AMD64.
The test lab: 2 x (server-client) HP Proliant G5 DL360 (quad-core/8gb ram, raid
5 SAS).
For net benchmark i used nuttcp and iperf.
The servers (client-server) are in 1 VLAN.
The results on 1Gbps (down & up):
63.4375 MB / 1.00 sec = 532.1332 Mbps
64.3750 MB / 1.00 sec = 540.0426 Mbps
62.8125 MB / 1.00 sec = 526.8963 Mbps
64.5625 MB / 1.00 sec = 541.6318 Mbps
63.9375 MB / 1.00 sec = 536.3595 Mbps
63.7500 MB / 1.00 sec = 534.7566 Mbps
63.0000 MB / 1.00 sec = 528.5003 Mbps
63.5000 MB / 1.00 sec = 532.7150 Mbps
64.0000 MB / 1.00 sec = 536.8586 Mbps
63.5625 MB / 1.00 sec = 533.2452 Mbps
637.6688 MB / 10.02 sec = 533.9108 Mbps 9 %TX 9 %RX 9 host-retrans 0.67
msRTT
25.5625 MB / 1.00 sec = 214.3916 Mbps
30.8750 MB / 1.00 sec = 259.0001 Mbps
29.9375 MB / 1.00 sec = 251.1347 Mbps
27.1875 MB / 1.00 sec = 228.0669 Mbps
30.5000 MB / 1.00 sec = 255.8533 Mbps
30.2500 MB / 1.00 sec = 253.7551 Mbps
26.8125 MB / 1.00 sec = 224.9211 Mbps
30.3750 MB / 1.00 sec = 254.8047 Mbps
30.3750 MB / 1.00 sec = 254.8050 Mbps
30.0625 MB / 1.00 sec = 252.1835 Mbps
292.2155 MB / 10.02 sec = 244.6825 Mbps 10 %TX 12 %RX 0 host-retrans 0.71
msRTT
As you can see down is littlebit more than half of full link speed. And upload
is only 20-25% of full link.
I tried to change a lot sysctl params but without a big results. Currenlty my
entries in /etc/sysctl.conf which regarding to TCP:
#kernel tuning, tcp
kern.ipc.somaxconn=2048
kern.ipc.nmbclusters=32768
kern.ipc.maxsockbuf=8388608
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.inflight.enable=0
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
net.inet.udp.recvspace=65536
net.inet.tcp.inflight.enable=0
net.inet.tcp.rfc1323=1
net.inet.tcp.sack.enable=1
net.inet.tcp.path_mtu_discovery=1
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.recvbuf_inc=524288
Do you have some kind suggestion what i could to change to increase the
performance of TCP?
Besides when i make the benchamrks i run the sniffer to see whats happening
with network..sometimes i saw that window size is
0...does it mean that server can't handle something or recieve buffer size is
to small?
p.s sory for my bad english :)
Noisex
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[email protected]"
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event
of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information
contained in it.
In the event of misdirection, illegible or incomplete transmission please
telephone +44 845 868 1337
or return the E.mail to [email protected].
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[email protected]"
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[email protected]"
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please
telephone +44 845 868 1337
or return the E.mail to [email protected].
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[email protected]"
