Ronnie Garcia a icrit :
I recently switched one of our firewalls from Linux to oBSD 4.0.
Its handling approx 8-9 kpps (in+out) on both interfaces. It has a
D-Link DFE-570TX quad ports NIC (dc driver), two ports are used.
On Linux, the CPU was loaded at approx 20% when, and on oBSD, its
actually loaded at ~30%. No big deal, but on Linux we had queueing
(shaping) with TC/HTB, whereas ALTQ is not (yet) enabled on oBSD.
The CPU usage is almost only "interrupt", as you can see on this top
output :
[The rest of the message is left bellow for the record.]
I can now tell that i have the exact same behaviour with bsd.mp.
I'm graphing a lot of kernel/pf variables with cacti, and i'm clearly
seeing the box maxing at 15k interrupts/s.
I'm raising 15k interrupts/s when the box is routing approx 13k pps and
then the CPU is at 50-55%.
When i disable pf (pfctl -d), the CPU downs to ~40% but the interrupts
rate does not decrease. This means that the high interrupts rate is due
to network activity, and not to pf.
The interrupts rate is higher than the packets rate !
I might try with an Intel Pro/1000MT quad instead of the D-Link
DFE-570TX quad to see if my problem is the NIC or the PCI bus/chipset.
Again, my dmesg :
syncing disks... 00000
OpenBSD 4.0 (GENERIC) #1107: Sat Sep 16 19:15:58 MDT 2006
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 2.40GHz ("GenuineIntel" 686-class) 2.41 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID
real mem = 528035840 (515660K)
avail mem = 473710592 (462608K)
using 4256 buffers containing 26505216 bytes (25884K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(fc) BIOS, date 06/11/03, BIOS32 rev. 0 @
0xf10a0, SMBIOS rev. 2.3 @ 0xf2d10 (44 entries)
bios0: ASUSTeK Computer INC. P4S533MX
apm0 at bios0: Power Management spec V1.2 (BIOS mgmt disabled)
apm0: APM power management enable: unrecognized device ID (9)
apm0: APM engage (device 1): power management disabled (1)
apm0: AC on, battery charge unknown
apm0: flags b0102 dobusy 0 doidle 1
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x16d2
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf1640/144 (7 entries)
pcibios0: PCI Interrupt Router at 000:02:0 ("SiS 962 ISA" rev 0x00)
pcibios0: PCI bus #2 is the last bus
bios0: ROM list: 0xc0000/0xc000
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "SiS 651 PCI" rev 0x02
ppb0 at pci0 dev 1 function 0 "SiS 86C201 AGP" rev 0x00
pci1 at ppb0 bus 1
vga1 at pci1 dev 0 function 0 "SiS 650 VGA" rev 0x00: aperture at
0xf0000000, size 0x400000
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 2 function 0 "SiS 962 ISA" rev 0x25
pciide0 at pci0 dev 2 function 5 "SiS 5513 EIDE" rev 0x00: 651: DMA,
channel 0 configured to compatibility, channel 1 configured to compatibility
wd0 at pciide0 channel 0 drive 0: <ST3802110A>
wd0: 16-sector PIO, LBA48, 76319MB, 156301488 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
atapiscsi0 at pciide0 channel 1 drive 1
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <, 54X CD-ROM, 6.53> SCSI0 5/cdrom removable
cd0(pciide0:1:1): using PIO mode 4, Ultra-DMA mode 2
auich0 at pci0 dev 2 function 7 "SiS 7012 AC97" rev 0xa0: irq 12,
SiS7012 AC97
ac97: codec id 0x41445370 (Analog Devices AD1980)
ac97: codec features headphone, 20 bit DAC, No 3D Stereo
audio0 at auich0
ohci0 at pci0 dev 3 function 0 "SiS 5597/5598 USB" rev 0x0f: irq 5,
version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0
uhub0: SiS OHCI root hub, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1 at pci0 dev 3 function 1 "SiS 5597/5598 USB" rev 0x0f: irq 9,
version 1.0, legacy support
usb1 at ohci1: USB revision 1.0
uhub1 at usb1
uhub1: SiS OHCI root hub, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
ehci0 at pci0 dev 3 function 3 "SiS 7002 USB" rev 0x00: irq 9
usb2 at ehci0: USB revision 2.0
uhub2 at usb2
uhub2: SiS EHCI root hub, rev 2.00/1.00, addr 1
uhub2: 6 ports with 6 removable, self powered
sis0 at pci0 dev 4 function 0 "SiS 900 10/100BaseTX" rev 0x91: irq 3,
address 00:0c:6e:d8:4a:59
rlphy0 at sis0 phy 1: RTL8201L 10/100 PHY, rev. 1
ppb1 at pci0 dev 14 function 0 "Intel S21152BB PCI-PCI" rev 0x00
pci2 at ppb1 bus 2
dc0 at pci2 dev 4 function 0 "DEC 21142/3" rev 0x41: irq 10, address
00:80:c8:cd:c8:21
nsphyter0 at dc0 phy 1: DP83843 10/100 PHY, rev. 0
dc1 at pci2 dev 5 function 0 "DEC 21142/3" rev 0x41: irq 12, address
00:80:c8:cd:c8:22
nsphyter1 at dc1 phy 1: DP83843 10/100 PHY, rev. 0
dc2 at pci2 dev 6 function 0 "DEC 21142/3" rev 0x41: irq 3, address
00:80:c8:cd:c8:23
nsphyter2 at dc2 phy 1: DP83843 10/100 PHY, rev. 0
dc3 at pci2 dev 7 function 0 "DEC 21142/3" rev 0x41: irq 11, address
00:80:c8:cd:c8:24
nsphyter3 at dc3 phy 1: DP83843 10/100 PHY, rev. 0
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
it0 at isa0 port 0x290/8: IT87
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask e365 netmask ff6d ttymask ffef
pctr: user-level cycle counter enabled
dkcsum: wd0 matches BIOS drive 0x80
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302
# top
load averages: 0.09, 0.10, 0.08
19 processes: 18 idle, 1 on processor
CPU states: 0.0% user, 0.0% nice, 0.0% system, 30.4% interrupt, 69.6%
idle
Memory: Real: 9768K/76M act/tot Free: 413M Swap: 0K/2048M used/tot
Note : %CPU interrupt goes from ~15 to ~35%
I tryed to disable PF by loading a minimal conf (pass in all, pass out
all), but the %interrupt did not decrease.
I'm not trying to compare Linux to oBSD but I'm wondering if this could
be because of a bad PCI bus, a bad NIC, or a "bad" driver
implementation. I might change the NIC if its the culprit.
What do you think ?
Another oBSD 4.0 box, which is a router in front of the firewall (thus
handling the same traffic), is only loaded at ~10-15% interrupt. This
one has an Intel PRO/1000MT quad ports card (em driver). (It has other
hardware differences).
Other usefull infos about the firewall :
# uname -a
OpenBSD XX 4.0 GENERIC#1107 i386
# sysctl net.inet.ip.ifq
net.inet.ip.ifq.len=0
net.inet.ip.ifq.maxlen=512
net.inet.ip.ifq.drops=13183292
Note : since i set ifq.maxlen to 512 (was 50), the ifq.drops stopped
growing.
# dmesg |grep cpu
cpu0: Intel(R) Pentium(R) 4 CPU 2.40GHz ("GenuineIntel" 686-class) 2.41 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID
cpu0 at mainbus0
# dmesg |grep dc
dc0 at pci2 dev 4 function 0 "DEC 21142/3" rev 0x41: irq 10, address
00:80:c8:cd:c8:21
nsphyter0 at dc0 phy 1: DP83843 10/100 PHY, rev. 0
dc1 at pci2 dev 5 function 0 "DEC 21142/3" rev 0x41: irq 12, address
00:80:c8:cd:c8:22
nsphyter1 at dc1 phy 1: DP83843 10/100 PHY, rev. 0
dc2 at pci2 dev 6 function 0 "DEC 21142/3" rev 0x41: irq 3, address
00:80:c8:cd:c8:23
nsphyter2 at dc2 phy 1: DP83843 10/100 PHY, rev. 0
dc3 at pci2 dev 7 function 0 "DEC 21142/3" rev 0x41: irq 11, address
00:80:c8:cd:c8:24
nsphyter3 at dc3 phy 1: DP83843 10/100 PHY, rev. 0
# dmesg |grep pci
pcib0 at pci0 dev 2 function 0 "SiS 962 ISA" rev 0x25
pciide0 at pci0 dev 2 function 5 "SiS 5513 EIDE" rev 0x00: 651: DMA,
channel 0 configured to compatibility, channel 1 configured to
compatibility
wd0 at pciide0 channel 0 drive 0: <ST3802110A>
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
atapiscsi0 at pciide0 channel 1 drive 1
cd0(pciide0:1:1): using PIO mode 4, Ultra-DMA mode 2
auich0 at pci0 dev 2 function 7 "SiS 7012 AC97" rev 0xa0: irq 12,
SiS7012 AC97
ohci0 at pci0 dev 3 function 0 "SiS 5597/5598 USB" rev 0x0f: irq 5,
version 1.0, legacy support
ohci1 at pci0 dev 3 function 1 "SiS 5597/5598 USB" rev 0x0f: irq 9,
version 1.0, legacy support
ehci0 at pci0 dev 3 function 3 "SiS 7002 USB" rev 0x00: irq 9
sis0 at pci0 dev 4 function 0 "SiS 900 10/100BaseTX" rev 0x91: irq 3,
address 00:0c:6e:d8:4a:59
ppb1 at pci0 dev 14 function 0 "Intel S21152BB PCI-PCI" rev 0x00
pci2 at ppb1 bus 2
dc0 at pci2 dev 4 function 0 "DEC 21142/3" rev 0x41: irq 10, address
00:80:c8:cd:c8:21
dc1 at pci2 dev 5 function 0 "DEC 21142/3" rev 0x41: irq 12, address
00:80:c8:cd:c8:22
dc2 at pci2 dev 6 function 0 "DEC 21142/3" rev 0x41: irq 3, address
00:80:c8:cd:c8:23
dc3 at pci2 dev 7 function 0 "DEC 21142/3" rev 0x41: irq 11, address
00:80:c8:cd:c8:24
isa0 at pcib0
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x16d2
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf1640/144 (7 entries)
pcibios0: PCI Interrupt Router at 000:02:0 ("SiS 962 ISA" rev 0x00)
pcibios0: PCI bus #2 is the last bus
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "SiS 651 PCI" rev 0x02
ppb0 at pci0 dev 1 function 0 "SiS 86C201 AGP" rev 0x00
pci1 at ppb0 bus 1
vga1 at pci1 dev 0 function 0 "SiS 650 VGA" rev 0x00: aperture at
0xf0000000, size 0x400000
Kind regards,
--
Ronnie Garcia <r.garcia at ovea dot com>
Directeur
oooo ovea
Til : +33 4 67670000
Gsm : +33 6 29500295
http://www.ovea.com
