Hello fellow OpenBSD'ers.
I would just like to share some information with the list about our new
firewall/bridge and perhaps get some input as to where I might be able to look
to squeeze some additional performance improvements. I must say though, I am
very impressed with the performance improvements of networking/PF in the
snapshots.
Parameters:
===========
bridge: OpenBSD 4.8-BETA (snapshot Aug 5, 2010)
server: CentOS 5.5 w/Updates as of Aug 5, 2010 - head3)
client: Ubuntu 10.04 w/Updates as of today - buckeye)
iperf options on server/client
==============================
server:
-------
iperf -s
client:
-------
for count in 1 2 3 4 5; do iperf -i 1 -t 60 -c head3 && sleep 15; done
Transfer indicates the amount of data transferred throughout the duration of
the test. Bandwidth indicates the average bandwidth consumed for the test.
[ ID] Interval Transfer Bandwidth
[ 1] 0.0-60.0 sec 5.28 GBytes 756 Mbits/sec
[ 2] 0.0-60.0 sec 5.20 GBytes 744 Mbits/sec
[ 3] 0.0-60.0 sec 5.12 GBytes 733 Mbits/sec
[ 4] 0.0-60.0 sec 5.30 GBytes 759 Mbits/sec
[ 5] 0.0-60.0 sec 5.08 GBytes 727 Mbits/sec
So as can be seen here we are seeing data transfer rates of between 85 and
90MBps. Pretty impressive for an first pass, untweaked configuration.
However, there are some "unfortunates". During these tests the system was
running at between 80 and 95% interrupt, with the inverse being idle. This
means that either there are some tweaks that I can add to counteract the
interrupts, perhaps a tweak for interrupt mitigation, or that the hardware
is currently not able to handle more than a single gigabit link running at
full capacity. In any case I would like to know what the developers see if
better hardware would help as well as any performance tweaks that may help.
These "unfortunates" are not really "bad news". The box is certainly up to
the task of dealing with our network traffic. Some tweaking may help and
for a first pass test it is a good baseline to work from and understand
where the bottlenecks are.
Obligitory Configuration Information:
===============================
# cat /etc/pf.conf
# See pf.conf(5) for syntax and examples.
# Remember to set net.inet.ip.forwarding=1 and/or net.inet6.ip6.forwarding=1
# in /etc/sysctl.conf if packets are to be forwarded between interfaces.
set skip on lo
# Table definitions
table <bad_hosts> persist
pass in on vlan300 # to establish keep-state
# block any host deemed for whatever reason to be bad
block quick from <bad_hosts>
pass out on vlan300
# if a host is found to be connecting more than 100 times within 10 minutes
# add them to bad_hosts table so they can be blocked
pass in proto tcp to any port ssh keep state \
(max-src-conn-rate 15/5, overload <bad_hosts> flush global)
# By default, do not permit remote connections to X11
#block in on ! lo0 proto tcp to port 6000:6010
# cat /etc/sysctl.conf
# $OpenBSD: sysctl.conf,v 1.47 2009/06/09 11:52:54 sthen Exp $
#
# This file contains a list of sysctl options the user wants set at
# boot time. See sysctl(3) and sysctl(8) for more information on
# the many available variables.
#
net.inet.ip.forwarding=1 # 1=Permit forwarding (routing) of IPv4 packets
#net.inet.ip.mforwarding=1 # 1=Permit forwarding (routing) of IPv4
multicast packets
#net.inet.ip.multipath=1 # 1=Enable IP multipath routing
#net.inet.icmp.rediraccept=1 # 1=Accept ICMP redirects
#net.inet6.icmp6.rediraccept=0 # 0=Don't accept IPv6 ICMP redirects
#net.inet6.ip6.forwarding=1 # 1=Permit forwarding (routing) of IPv6 packets
#net.inet6.ip6.mforwarding=1 # 1=Permit forwarding (routing) of IPv6
multicast packets
#net.inet6.ip6.multipath=1 # 1=Enable IPv6 multipath routing
#net.inet6.ip6.accept_rtadv=1 # 1=Permit IPv6 autoconf (forwarding must be 0)
#net.inet.tcp.rfc1323=0 # 0=Disable TCP RFC1323 extensions (for if tcp
is slow)
#net.inet.tcp.rfc3390=0 # 0=Disable RFC3390 for TCP window increasing
#net.inet.esp.enable=0 # 0=Disable the ESP IPsec protocol
#net.inet.ah.enable=0 # 0=Disable the AH IPsec protocol
#net.inet.esp.udpencap=0 # 0=Disable ESP-in-UDP encapsulation
#net.inet.ipcomp.enable=1 # 1=Enable the IPCOMP protocol
#net.inet.etherip.allow=1 # 1=Enable the Ethernet-over-IP protocol
#net.inet.tcp.ecn=1 # 1=Enable the TCP ECN extension
#net.inet.carp.preempt=1 # 1=Enable carp(4) preemption
#net.inet.carp.log=1 # 1=Enable logging of carp(4) packets
#ddb.panic=0 # 0=Do not drop into ddb on a kernel panic
#ddb.console=1 # 1=Permit entry of ddb from the console
#fs.posix.setuid=0 # 0=Traditional BSD chown() semantics
#vm.swapencrypt.enable=0 # 0=Do not encrypt pages that go to swap
#vfs.nfs.iothreads=4 # Number of nfsio kernel threads
#net.inet.ip.mtudisc=0 # 0=Disable tcp mtu discovery
#kern.usercrypto=0 # 0=Disable userland use of /dev/crypto
#kern.splassert=2 # 2=Enable with verbose error messages
#kern.nosuidcoredump=2 # 2=Put suid coredumps in /var/crash
#kern.watchdog.period=32 # >0=Enable hardware watchdog(4) timer if
available
#kern.watchdog.auto=0 # 0=Disable automatic watchdog(4) retriggering
#machdep.allowaperture=2 # See xf86(4)
#machdep.kbdreset=1 # permit console CTRL-ALT-DEL to do a nice halt
#pfctl -sa (without states)
INFO:
Status: Enabled for 5 days 03:51:55 Debug: err
State Table Total Rate
current entries 6292
searches 4258348201 9549.7/s
inserts 45197611 101.4/s
removals 45191319 101.3/s
Counters
match 2397196364 5375.9/s
bad-offset 0 0.0/s
fragment 441 0.0/s
short 111 0.0/s
normalize 370 0.0/s
memory 14103 0.0/s
bad-timestamp 0 0.0/s
congestion 681 0.0/s
ip-option 447562 1.0/s
proto-cksum 0 0.0/s
state-mismatch 7560 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 5 0.0/s
synproxy 0 0.0/s
TIMEOUTS:
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90s
tcp.tsdiff 30s
udp.first 60s
udp.single 30s
udp.multiple 60s
icmp.first 20s
icmp.error 10s
other.first 60s
other.single 30s
other.multiple 60s
frag 30s
interval 10s
adaptive.start 6000 states
adaptive.end 12000 states
src.track 0s
LIMITS:
states hard limit 10000
src-nodes hard limit 10000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 200000
TABLES:
bad_hosts
#dmesg
OpenBSD 4.8-beta (GENERIC.MP) #259: Tue Aug 3 09:06:37 MDT 2010
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 3487244288 (3325MB)
avail mem = 3380592640 (3223MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xcff9c000 (46 entries)
bios0: vendor Dell Inc. version "1.4.3" date 05/15/2009
bios0: Dell Inc. PowerEdge R200
acpi0 at bios0: rev 2
acpi0: tables DSDT FACP APIC SPCR HPET MCFG WDAT SLIC ERST HEST BERT EINJ SSDT
SSDT SSDT
acpi0: wakeup devices PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM)2 Duo CPU E7400 @ 2.80GHz, 2800.50 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,NXE,LONG
cpu0: 3MB 64b/line 8-way L2 cache
cpu0: apic clock running at 266MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Core(TM)2 Duo CPU E7400 @ 2.80GHz, 2800.10 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,NXE,LONG
cpu1: 3MB 64b/line 8-way L2 cache
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 2
ioapic1 at mainbus0: apid 3 pa 0xfec10000, version 20, 24 pins
ioapic1: misconfigured as apic 0, remapped to apid 3
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEX1)
acpiprt2 at acpi0: bus 2 (SBE0)
acpiprt3 at acpi0: bus 3 (PXHA)
acpiprt4 at acpi0: bus 4 (SBE4)
acpiprt5 at acpi0: bus 5 (SBE5)
acpiprt6 at acpi0: bus 6 (COMP)
acpicpu0 at acpi0: PSS
acpicpu1 at acpi0: PSS
ipmi at mainbus0 not configured
cpu0: Enhanced SpeedStep 2800 MHz: speeds: 2800, 2400, 2133, 1867, 1600 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 3200/3210 Host" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 3200/3210 PCIE" rev 0x01: apic 2 int 16
(irq 15)
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic 2 int
16 (irq 15), address 00:15:17:d6:18:d0
em1 at pci1 dev 0 function 1 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic 2 int
17 (irq 14), address 00:15:17:d6:18:d1
ppb1 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci3 at ppb2 bus 3
ppb3 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02
pci4 at ppb3 bus 4
bge0 at pci4 dev 0 function 0 "Broadcom BCM5721" rev 0x21, BCM5750 C1
(0x4201): apic 2 int 16 (irq 15), address 00:25:64:3c:c1:0a
brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
ppb4 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02
pci5 at ppb4 bus 5
bge1 at pci5 dev 0 function 0 "Broadcom BCM5721" rev 0x21, BCM5750 C1
(0x4201): apic 2 int 17 (irq 14), address 00:25:64:3c:c1:0b
brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
uhci0 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 2 int 21 (irq
11)
uhci1 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 2 int 20 (irq
10)
uhci2 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 2 int 21 (irq
11)
ehci0 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 2 int 21 (irq
11)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb5 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci6 at ppb5 bus 6
vga1 at pci6 dev 5 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
radeondrm0 at vga1: apic 2 int 19 (irq 5)
drm0 at radeondrm0
pcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02
pciide0 at pci0 dev 31 function 2 "Intel 82801I SATA" rev 0x02: DMA, channel 0
configured to native-PCI, channel 1 configured to native-PCI
pciide0: using apic 2 int 23 (irq 6) for native-PCI interrupt
wd0 at pciide0 channel 0 drive 0: <WDC WD1602ABKS-18N8A0>
wd0: 16-sector PIO, LBA48, 152587MB, 312500000 sectors
atapiscsi0 at pciide0 channel 0 drive 1
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <TEAC, DVD-ROM DV28SV, D.0L> ATAPI 5/cdrom
removable
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6
cd0(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 5
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
mtrr: Pentium Pro MTRR support
uhub4 at uhub0 port 5 "Cypress Semiconductor USB2 Hub" rev 2.00/0.0b addr 2
uhidev0 at uhub2 port 2 configuration 1 interface 0 "Avocent Dell 03R874" rev
1.10/1.00 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes, country code 33
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub2 port 2 configuration 1 interface 1 "Avocent Dell 03R874" rev
1.10/1.00 addr 2
uhidev1: iclass 3/1, 3 report ids
ums0 at uhidev1 reportid 1: 5 buttons, Z dir
wsmouse0 at ums0 mux 0
uhid0 at uhidev1 reportid 2: input=2, output=0, feature=0
uhid1 at uhidev1 reportid 3: input=1, output=0, feature=0
softraid0 at root
root on wd0a swap on wd0b dump on wd0b
---
James A. Peltier [email protected]
