On 11 August 2010 15:12, James Peltier <[email protected]> wrote: > Hello fellow OpenBSD'ers. > > I would just like to share some information with the list about our new > firewall/bridge and perhaps get some input as to where I might be able to look > to squeeze some additional performance improvements. I must say though, I am > very impressed with the performance improvements of networking/PF in the > snapshots. > > Parameters: > =========== > bridge: OpenBSD 4.8-BETA (snapshot Aug 5, 2010) > server: CentOS 5.5 w/Updates as of Aug 5, 2010 - head3) > client: Ubuntu 10.04 w/Updates as of today - buckeye) > > iperf options on server/client > ============================== > server: > ------- > iperf -s > > client: > ------- > for count in 1 2 3 4 5; do iperf -i 1 -t 60 -c head3 && sleep 15; done > > > Transfer indicates the amount of data transferred throughout the duration of > the test. Bandwidth indicates the average bandwidth consumed for the test. > > [ ID] Interval Transfer Bandwidth > [ 1] 0.0-60.0 sec 5.28 GBytes 756 Mbits/sec > [ 2] 0.0-60.0 sec 5.20 GBytes 744 Mbits/sec > [ 3] 0.0-60.0 sec 5.12 GBytes 733 Mbits/sec > [ 4] 0.0-60.0 sec 5.30 GBytes 759 Mbits/sec > [ 5] 0.0-60.0 sec 5.08 GBytes 727 Mbits/sec > > So as can be seen here we are seeing data transfer rates of between 85 and > 90MBps. Pretty impressive for an first pass, untweaked configuration. > > However, there are some "unfortunates". During these tests the system was > running at between 80 and 95% interrupt, with the inverse being idle. This > means that either there are some tweaks that I can add to counteract the > interrupts, perhaps a tweak for interrupt mitigation, or that the hardware > is currently not able to handle more than a single gigabit link running at > full capacity. In any case I would like to know what the developers see if > better hardware would help as well as any performance tweaks that may help. > > These "unfortunates" are not really "bad news". The box is certainly up to > the task of dealing with our network traffic. Some tweaking may help and > for a first pass test it is a good baseline to work from and understand > where the bottlenecks are. > > > > Obligitory Configuration Information: > =============================== > > # cat /etc/pf.conf > # See pf.conf(5) for syntax and examples. > # Remember to set net.inet.ip.forwarding=1 and/or net.inet6.ip6.forwarding=1 > # in /etc/sysctl.conf if packets are to be forwarded between interfaces. > > set skip on lo > > # Table definitions > table <bad_hosts> persist > > pass in on vlan300 # to establish keep-state > > # block any host deemed for whatever reason to be bad > block quick from <bad_hosts> > > pass out on vlan300 > > # if a host is found to be connecting more than 100 times within 10 minutes > # add them to bad_hosts table so they can be blocked > pass in proto tcp to any port ssh keep state \ > (max-src-conn-rate 15/5, overload <bad_hosts> flush global) > > # By default, do not permit remote connections to X11 > #block in on ! lo0 proto tcp to port 6000:6010 > > > # cat /etc/sysctl.conf > # $OpenBSD: sysctl.conf,v 1.47 2009/06/09 11:52:54 sthen Exp $ > # > # This file contains a list of sysctl options the user wants set at > # boot time. See sysctl(3) and sysctl(8) for more information on > # the many available variables. > # > net.inet.ip.forwarding=1 # 1=Permit forwarding (routing) of IPv4 packets > #net.inet.ip.mforwarding=1 # 1=Permit forwarding (routing) of IPv4 > multicast packets > #net.inet.ip.multipath=1 # 1=Enable IP multipath routing > #net.inet.icmp.rediraccept=1 # 1=Accept ICMP redirects > #net.inet6.icmp6.rediraccept=0 # 0=Don't accept IPv6 ICMP redirects > #net.inet6.ip6.forwarding=1 # 1=Permit forwarding (routing) of IPv6 packets > #net.inet6.ip6.mforwarding=1 # 1=Permit forwarding (routing) of IPv6 > multicast packets > #net.inet6.ip6.multipath=1 # 1=Enable IPv6 multipath routing > #net.inet6.ip6.accept_rtadv=1 # 1=Permit IPv6 autoconf (forwarding must be 0) > #net.inet.tcp.rfc1323=0 # 0=Disable TCP RFC1323 extensions (for if tcp > is slow) > #net.inet.tcp.rfc3390=0 # 0=Disable RFC3390 for TCP window increasing > #net.inet.esp.enable=0 # 0=Disable the ESP IPsec protocol > #net.inet.ah.enable=0 # 0=Disable the AH IPsec protocol > #net.inet.esp.udpencap=0 # 0=Disable ESP-in-UDP encapsulation > #net.inet.ipcomp.enable=1 # 1=Enable the IPCOMP protocol > #net.inet.etherip.allow=1 # 1=Enable the Ethernet-over-IP protocol > #net.inet.tcp.ecn=1 # 1=Enable the TCP ECN extension > #net.inet.carp.preempt=1 # 1=Enable carp(4) preemption > #net.inet.carp.log=1 # 1=Enable logging of carp(4) packets > #ddb.panic=0 # 0=Do not drop into ddb on a kernel panic > #ddb.console=1 # 1=Permit entry of ddb from the console > #fs.posix.setuid=0 # 0=Traditional BSD chown() semantics > #vm.swapencrypt.enable=0 # 0=Do not encrypt pages that go to swap > #vfs.nfs.iothreads=4 # Number of nfsio kernel threads > #net.inet.ip.mtudisc=0 # 0=Disable tcp mtu discovery > #kern.usercrypto=0 # 0=Disable userland use of /dev/crypto > #kern.splassert=2 # 2=Enable with verbose error messages > #kern.nosuidcoredump=2 # 2=Put suid coredumps in /var/crash > #kern.watchdog.period=32 # >0=Enable hardware watchdog(4) timer if > available > #kern.watchdog.auto=0 # 0=Disable automatic watchdog(4) retriggering > #machdep.allowaperture=2 # See xf86(4) > #machdep.kbdreset=1 # permit console CTRL-ALT-DEL to do a nice halt > > > #pfctl -sa (without states) > INFO: > Status: Enabled for 5 days 03:51:55 Debug: err > > State Table Total Rate > current entries 6292 > searches 4258348201 9549.7/s > inserts 45197611 101.4/s > removals 45191319 101.3/s > Counters > match 2397196364 5375.9/s > bad-offset 0 0.0/s > fragment 441 0.0/s > short 111 0.0/s > normalize 370 0.0/s > memory 14103 0.0/s > bad-timestamp 0 0.0/s > congestion 681 0.0/s > ip-option 447562 1.0/s > proto-cksum 0 0.0/s > state-mismatch 7560 0.0/s > state-insert 0 0.0/s > state-limit 0 0.0/s > src-limit 5 0.0/s > synproxy 0 0.0/s > > TIMEOUTS: > tcp.first 120s > tcp.opening 30s > tcp.established 86400s > tcp.closing 900s > tcp.finwait 45s > tcp.closed 90s > tcp.tsdiff 30s > udp.first 60s > udp.single 30s > udp.multiple 60s > icmp.first 20s > icmp.error 10s > other.first 60s > other.single 30s > other.multiple 60s > frag 30s > interval 10s > adaptive.start 6000 states > adaptive.end 12000 states > src.track 0s > > LIMITS: > states hard limit 10000 > src-nodes hard limit 10000 > frags hard limit 5000 > tables hard limit 1000 > table-entries hard limit 200000 > > TABLES: > bad_hosts > > > > #dmesg > > OpenBSD 4.8-beta (GENERIC.MP) #259: Tue Aug 3 09:06:37 MDT 2010 > [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 3487244288 (3325MB) > avail mem = 3380592640 (3223MB) > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xcff9c000 (46 entries) > bios0: vendor Dell Inc. version "1.4.3" date 05/15/2009 > bios0: Dell Inc. PowerEdge R200 > acpi0 at bios0: rev 2 > acpi0: tables DSDT FACP APIC SPCR HPET MCFG WDAT SLIC ERST HEST BERT EINJ SSDT > SSDT SSDT > acpi0: wakeup devices PCI0(S5) > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Core(TM)2 Duo CPU E7400 @ 2.80GHz, 2800.50 MHz > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,EST,TM2,SSSE3,CX1 6,xTPR,PDCM,SSE4.1,XSAVE,NXE,LONG > > cpu0: 3MB 64b/line 8-way L2 cache > cpu0: apic clock running at 266MHz > cpu1 at mainbus0: apid 1 (application processor) > cpu1: Intel(R) Core(TM)2 Duo CPU E7400 @ 2.80GHz, 2800.10 MHz > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,EST,TM2,SSSE3,CX1 6,xTPR,PDCM,SSE4.1,XSAVE,NXE,LONG > > cpu1: 3MB 64b/line 8-way L2 cache > ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins > ioapic0: misconfigured as apic 0, remapped to apid 2 > ioapic1 at mainbus0: apid 3 pa 0xfec10000, version 20, 24 pins > ioapic1: misconfigured as apic 0, remapped to apid 3 > acpihpet0 at acpi0: 14318179 Hz > acpiprt0 at acpi0: bus 0 (PCI0) > acpiprt1 at acpi0: bus 1 (PEX1) > acpiprt2 at acpi0: bus 2 (SBE0) > acpiprt3 at acpi0: bus 3 (PXHA) > acpiprt4 at acpi0: bus 4 (SBE4) > acpiprt5 at acpi0: bus 5 (SBE5) > acpiprt6 at acpi0: bus 6 (COMP) > acpicpu0 at acpi0: PSS > acpicpu1 at acpi0: PSS > ipmi at mainbus0 not configured > cpu0: Enhanced SpeedStep 2800 MHz: speeds: 2800, 2400, 2133, 1867, 1600 MHz > pci0 at mainbus0 bus 0 > pchb0 at pci0 dev 0 function 0 "Intel 3200/3210 Host" rev 0x01 > ppb0 at pci0 dev 1 function 0 "Intel 3200/3210 PCIE" rev 0x01: apic 2 int 16 > (irq 15) > pci1 at ppb0 bus 1 > em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic 2 int > 16 (irq 15), address 00:15:17:d6:18:d0 > em1 at pci1 dev 0 function 1 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic 2 int > 17 (irq 14), address 00:15:17:d6:18:d1 > ppb1 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02 > pci2 at ppb1 bus 2 > ppb2 at pci2 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09 > pci3 at ppb2 bus 3 > ppb3 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02 > pci4 at ppb3 bus 4 > bge0 at pci4 dev 0 function 0 "Broadcom BCM5721" rev 0x21, BCM5750 C1 > (0x4201): apic 2 int 16 (irq 15), address 00:25:64:3c:c1:0a > brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0 > ppb4 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02 > pci5 at ppb4 bus 5 > bge1 at pci5 dev 0 function 0 "Broadcom BCM5721" rev 0x21, BCM5750 C1 > (0x4201): apic 2 int 17 (irq 14), address 00:25:64:3c:c1:0b > brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0 > uhci0 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 2 int 21 (irq > 11) > uhci1 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 2 int 20 (irq > 10) > uhci2 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 2 int 21 (irq > 11) > ehci0 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 2 int 21 (irq > 11) > usb0 at ehci0: USB revision 2.0 > uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 > ppb5 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92 > pci6 at ppb5 bus 6 > vga1 at pci6 dev 5 function 0 "ATI ES1000" rev 0x02 > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) > wsdisplay0: screen 1-5 added (80x25, vt100 emulation) > radeondrm0 at vga1: apic 2 int 19 (irq 5) > drm0 at radeondrm0 > pcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02 > pciide0 at pci0 dev 31 function 2 "Intel 82801I SATA" rev 0x02: DMA, channel 0 > configured to native-PCI, channel 1 configured to native-PCI > pciide0: using apic 2 int 23 (irq 6) for native-PCI interrupt > wd0 at pciide0 channel 0 drive 0: <WDC WD1602ABKS-18N8A0> > wd0: 16-sector PIO, LBA48, 152587MB, 312500000 sectors > atapiscsi0 at pciide0 channel 0 drive 1 > scsibus0 at atapiscsi0: 2 targets > cd0 at scsibus0 targ 0 lun 0: <TEAC, DVD-ROM DV28SV, D.0L> ATAPI 5/cdrom > removable > wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6 > cd0(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 5 > usb1 at uhci0: USB revision 1.0 > uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > usb2 at uhci1: USB revision 1.0 > uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > usb3 at uhci2: USB revision 1.0 > uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > isa0 at pcib0 > isadma0 at isa0 > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > com0: console > pckbc0 at isa0 port 0x60/5 > pcppi0 at isa0 port 0x61 > spkr0 at pcppi0 > mtrr: Pentium Pro MTRR support > uhub4 at uhub0 port 5 "Cypress Semiconductor USB2 Hub" rev 2.00/0.0b addr 2 > uhidev0 at uhub2 port 2 configuration 1 interface 0 "Avocent Dell 03R874" rev > 1.10/1.00 addr 2 > uhidev0: iclass 3/1 > ukbd0 at uhidev0: 8 modifier keys, 6 key codes, country code 33 > wskbd0 at ukbd0: console keyboard, using wsdisplay0 > uhidev1 at uhub2 port 2 configuration 1 interface 1 "Avocent Dell 03R874" rev > 1.10/1.00 addr 2 > uhidev1: iclass 3/1, 3 report ids > ums0 at uhidev1 reportid 1: 5 buttons, Z dir > wsmouse0 at ums0 mux 0 > uhid0 at uhidev1 reportid 2: input=2, output=0, feature=0 > uhid1 at uhidev1 reportid 3: input=1, output=0, feature=0 > softraid0 at root > root on wd0a swap on wd0b dump on wd0b > > --- > James A. Peltier [email protected] > >
Could you perform the same test using tcpbench between two openbsd boxes ? I never had the chance to test it under a heavy load like yours.
