On Nov 16, 2007 2:57 PM, Calomel <[EMAIL PROTECTED]> wrote:
>
> Doing a quick google search I see other people have also reported problems
> with the on board Broadcom BCM5708 on the dells. Can you try another
> network card like the Intel Pro/1000 MT (OpenBSD interface name: em0) ?
>
> My place of business transfers an average of 450Mbit with OpenBSD 4.1/4.2
> with ALTQ (HFSC) without issue. CPU usage for the interrupts are around 33%
> on a amd64 2.2GHz.
Hi Calomel,
It took me a little longer than I expected, bit I got a dual Intel
Pro/1000 to try. Unfortunately the results weren't very different:
interrupts were at 80% when I put 90Mbps of traffic over the bridge.
I wonder what are the major differences between our servers. Could you
send me a dmesg of your bridge ?
fmbraga:130$ sudo pfctl -si
Status: Enabled for 0 days 00:14:19 Debug: Urgent
State Table Total Rate
current entries 291977
searches 28283453 32926.0/s
inserts 1779088 2071.1/s
removals 1487111 1731.2/s
Counters
match 2024938 2357.3/s
bad-offset 0 0.0/s
fragment 8 0.0/s
short 176 0.2/s
normalize 81 0.1/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 28 0.0/s
proto-cksum 0 0.0/s
state-mismatch 5530 6.4/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
fmbraga:132$
load averages: 0.72, 0.83, 0.55 06:26:40
33 processes: 32 idle, 1 on processor
CPU0 states: 3.4% user, 0.0% nice, 0.4% system, 83.4% interrupt, 12.8% idle
CPU1 states: 2.2% user, 0.0% nice, 0.4% system, 0.0% interrupt, 97.4% idle
CPU2 states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU3 states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Memory: Real: 20M/271M act/tot Free: 2963M Swap: 0K/2055M used/tot
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
17166 fmbraga -6 0 636K 1052K sleep/0 piperd 0:23 8.94% sed
26875 fmbraga -6 0 456K 1028K sleep/3 piperd 0:31 3.91% sed
29981 fmbraga 2 0 3248K 2200K sleep/3 select 0:00 0.00% sshd
28812 fmbraga 2 0 3396K 2208K sleep/0 select 0:00 0.00% sshd
12557 fmbraga 2 0 3416K 2152K sleep/0 select 0:00 0.00% sshd
10314 root 2 0 3276K 2864K idle netio 0:00 0.00% sshd
29572 _pflogd 4 0 588K 476K sleep/0 bpf 0:00 0.00% pflogd
10264 fmbraga 2 0 924K 2072K sleep/0 poll 0:00 0.00% top
19219 root 10 0 636K 492K sleep/0 nanosle 0:00 0.00% pfctl
11936 root 10 0 664K 500K sleep/0 nanosle 0:00 0.00% pfctl
30336 root 2 0 640K 1380K idle select 0:00 0.00% sshd
19057 root 2 0 3440K 2920K idle netio 0:00 0.00% sshd
25636 root 2 0 3668K 6248K sleep/0 select 0:00 0.00% snmpd
29407 fmbraga 28 0 752K 1952K onproc/0 - 0:00 0.00% top
16659 root 2 0 1320K 1752K sleep/0 select 0:00 0.00% sendmail
24774 root 2 0 3420K 2952K idle netio 0:00 0.00% sshd
18059 _syslogd 2 0 504K 828K sleep/0 poll 0:00 0.00% syslogd
30791 fmbraga 18 0 624K 752K idle pause 0:00 0.00% ksh
4597 fmbraga 18 0 776K 744K idle pause 0:00 0.00% ksh
31505 fmbraga 2 0 3336K 2172K sleep/0 select 0:00 0.00% sshd
4308 root 2 0 3364K 2924K idle netio 0:00 0.00% sshd
17683 root 2 0 452K 812K idle select 0:00 0.00% inetd
32085 fmbraga 18 0 568K 756K idle pause 0:00 0.00% ksh
22181 fmbraga 18 0 668K 736K idle pause 0:00 0.00% ksh
1 root 10 0 448K 384K idle wait 0:00 0.00% init
29714 root 3 0 416K 856K idle ttyin 0:00 0.00% getty
6541 root 3 0 284K 860K idle ttyin 0:00 0.00% getty
new dmesg follows:
fmbraga:134$ dmesg
OpenBSD 4.2 (GENERIC.MP) #1378: Tue Aug 28 10:48:58 MDT 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 3488907264 (3327MB)
avail mem = 3373899776 (3217MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xcffbc000 (62 entries)
bios0: vendor Dell Inc. version "1.3.7" date 03/26/2007
bios0: Dell Inc. PowerEdge 1950
acpi at mainbus0 not configured
ipmi0 at mainbus0: version 2.0 interface KCS iobase 0xca8/8 spacing 4
mainbus0: Intel MP Specification (Version 1.4)
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.87 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG
cpu0: 4MB 64b/line 16-way L2 cache
cpu0: apic clock running at 332MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.53 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG
cpu1: 4MB 64b/line 16-way L2 cache
cpu2 at mainbus0: apid 3 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.53 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG
cpu2: 4MB 64b/line 16-way L2 cache
cpu3 at mainbus0: apid 1 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.53 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG
cpu3: 4MB 64b/line 16-way L2 cache
mpbios: bus 0 is type PCI
mpbios: bus 1 is type PCI
mpbios: bus 2 is type PCI
mpbios: bus 3 is type PCI
mpbios: bus 4 is type PCI
mpbios: bus 5 is type PCI
mpbios: bus 6 is type PCI
mpbios: bus 7 is type PCI
mpbios: bus 8 is type PCI
mpbios: bus 9 is type PCI
mpbios: bus 10 is type PCI
mpbios: bus 11 is type PCI
mpbios: bus 12 is type PCI
mpbios: bus 13 is type PCI
mpbios: bus 14 is type PCI
mpbios: bus 15 is type PCI
mpbios: bus 16 is type PCI
mpbios: bus 17 is type ISA
ioapic0 at mainbus0 apid 4 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 4
ioapic1 at mainbus0 apid 5 pa 0xfec81000, version 20, 24 pins
ioapic1: misconfigured as apic 0, remapped to apid 5
pci0 at mainbus0 bus 0: configuration mode 1
pchb0 at pci0 dev 0 function 0 "Intel 5000X Host" rev 0x12
ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE" rev 0x12
pci1 at ppb0 bus 6
ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci2 at ppb1 bus 7
ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci3 at ppb2 bus 8
ppb3 at pci3 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
pci4 at ppb3 bus 9
bnx0 at pci4 dev 0 function 0 "Broadcom BCM5708" rev 0x12: apic 4 int 16 (irq 5)
ppb4 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01
pci5 at ppb4 bus 10
ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
pci6 at ppb5 bus 11
ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0x12
pci7 at ppb6 bus 1
ppb7 at pci7 dev 0 function 0 "Intel IOP333 PCIE-PCIX" rev 0x00
pci8 at ppb7 bus 2
mfi0 at pci8 dev 14 function 0 "Dell PERC 5" rev 0x00: apic 5 int 14 (irq 6)
mfi0: logical drives 1, version 5.1.1-0040, 256MB RAM
scsibus0 at mfi0: 1 targets
sd0 at scsibus0 targ 0 lun 0: <DELL, PERC 5/i, 1.03> SCSI3 0/direct fixed
sd0: 152064MB, 19385 cyl, 255 head, 63 sec, 512 bytes/sec, 311427072 sec total
ppb8 at pci7 dev 0 function 2 "Intel IOP333 PCIE-PCIX" rev 0x00
pci9 at ppb8 bus 3
ppb9 at pci0 dev 4 function 0 "Intel 5000 PCIE" rev 0x12
pci10 at ppb9 bus 12
em0 at pci10 dev 0 function 0 "Intel PRO/1000 PT (82571EB)" rev 0x06:
apic 4 int 16 (irq 5), address 00:15:17:3a:65:10
em1 at pci10 dev 0 function 1 "Intel PRO/1000 PT (82571EB)" rev 0x06:
apic 4 int 17 (irq 11), address 00:15:17:3a:65:11
ppb10 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0x12
pci11 at ppb10 bus 13
ppb11 at pci0 dev 6 function 0 "Intel 5000 PCIE" rev 0x12
pci12 at ppb11 bus 14
ppb12 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0x12
pci13 at ppb12 bus 15
pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0x12
pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0x12
pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0x12
pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0x12
pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0x12
pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0x12
pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0x12
ppb13 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09
pci14 at ppb13 bus 4
ppb14 at pci14 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
pci15 at ppb14 bus 5
bnx1 at pci15 dev 0 function 0 "Broadcom BCM5708" rev 0x12: apic 4 int
16 (irq 5)
uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: apic 4
int 21 (irq 11)
uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: apic 4
int 20 (irq 10)
uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: apic 4
int 21 (irq 11)
ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: apic 4
int 21 (irq 11)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0: Intel EHCI root hub, rev 2.00/1.00, addr 1
ppb15 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xd9
pci16 at ppb15 bus 16
vga1 at pci16 dev 13 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09
pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA,
channel 0 configured to compatibility, channel 1 configured to
compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <HL-DT-ST, CDRW/DVD GCC4244, B101> SCSI0
5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
usb1 at uhci0: USB revision 1.0
uhub1 at usb1: Intel UHCI root hub, rev 1.00/1.00, addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2: Intel UHCI root hub, rev 1.00/1.00, addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3: Intel UHCI root hub, rev 1.00/1.00, addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
uhub4 at uhub0 port 1: Dell product 0xa001, rev 2.00/0.00, addr 2
uhidev0 at uhub4 port 1 configuration 1 interface 0
uhidev0: Dell DRAC5, rev 1.10/0.00, addr 3, iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub4 port 1 configuration 1 interface 1
uhidev1: Dell DRAC5, rev 1.10/0.00, addr 3, iclass 3/1
ums0 at uhidev1
ums0: X report 0x0002 not supported
umass0 at uhub4 port 2 configuration 1 interface 0
umass0: DELL INC. DRAC5 VIRTUAL MEDIA, rev 2.00/0.00, addr 4
umass0: using SCSI over Bulk-Only
scsibus2 at umass0: 2 targets
cd1 at scsibus2 targ 1 lun 0: <Dell, Virtual CDROM, 123> SCSI0 5/cdrom removable
umass1 at uhub4 port 2 configuration 1 interface 1
umass1: DELL INC. DRAC5 VIRTUAL MEDIA, rev 2.00/0.00, addr 4
umass1: using SCSI over Bulk-Only
scsibus3 at umass1: 2 targets
sd1 at scsibus3 targ 1 lun 0: <Dell, Virtual Floppy, 123> SCSI0
0/direct removable
sd1: drive offline
uhub5 at uhub0 port 5: Cypress Semiconductor USB2 Hub, rev 2.00/0.0b, addr 5
uhidev2 at uhub2 port 1 configuration 1 interface 0
uhidev2: vendor 0x04d9 product 0x1400, rev 1.10/1.43, addr 2, iclass 3/1
ukbd1 at uhidev2: 8 modifier keys, 6 key codes
wskbd2 at ukbd1 mux 1
wskbd2: connecting to wsdisplay0
uhidev3 at uhub2 port 1 configuration 1 interface 1
uhidev3: vendor 0x04d9 product 0x1400, rev 1.10/1.43, addr 2, iclass 3/1
uhidev3: 3 report ids
ums1 at uhidev3 reportid 1: 5 buttons and Z dir.
wsmouse0 at ums1 mux 0
uhid0 at uhidev3 reportid 2: input=3, output=0, feature=0
uhid1 at uhidev3 reportid 3: input=1, output=0, feature=0
dkcsum: sd0 matches BIOS drive 0x80
root on sd0a swap on sd0b dump on sd0b
bnx1: address 00:19:b9:eb:75:d1
brgphy0 at bnx1 phy 1: BCM5708C 10/100/1000baseT PHY, rev. 6
bnx0: address 00:19:b9:eb:75:d3
brgphy1 at bnx0 phy 1: BCM5708C 10/100/1000baseT PHY, rev. 6
fmbraga:135$
Same pf.conf was applied:
> On Mon, Nov 12, 2007 at 02:05:54PM -0300, Fernando Braga wrote:
> >
> >Is there a way to minimize interrupts usage in this configuration ?
> >
> >My pf.conf is:
> >
> ># $OpenBSD: pf.conf,v 1.34 2007/02/24 19:30:59 millert Exp $
> >
> >ext_if="bnx0"
> >int_if="bnx1"
> >
> >table <network-int> persist { 200.254.131.128/25 }
> >table <redes-biz> persist file "/etc/pf.biz"
> >
> >set skip on lo
> >set limit states 750000
> >
> >scrub in
> >
> >altq on $int_if bandwidth 1000Mb hfsc queue { local, embratel }
> >altq on $ext_if bandwidth 1000Mb hfsc queue { local, embratel }
> >
> >queue local bandwidth 100Mb hfsc
> >
> >queue embratel bandwidth 200Mb hfsc(ecn realtime 200Mb upperlimit
> >200Mb ) \
> > { Q-pri Q-icmp Q-vpn Q-biz Q-mail Q-http Q-ftp Q-def }
> >
> >queue Q-pri bandwidth 25Mb hfsc(realtime 15Mb linkshare 25Mb
> >upperlimit 180Mb) priority 7
> >queue Q-icmp bandwidth 25Mb hfsc(realtime 15Mb linkshare 25Mb
> >upperlimit 30Mb) priority 7
> >queue Q-vpn bandwidth 25Mb hfsc(realtime 15Mb linkshare 25Mb
> >upperlimit 180Mb) priority 6
> >queue Q-biz bandwidth 25Mb hfsc(realtime 15Mb linkshare 25Mb
> >upperlimit 180Mb) priority 6
> >queue Q-mail bandwidth 25Mb hfsc(realtime 15Mb linkshare 25Mb
> >upperlimit 180Mb) priority 4
> >queue Q-http bandwidth 25Mb hfsc(realtime 15Mb linkshare 25Mb
> >upperlimit 180Mb) priority 4
> >queue Q-ftp bandwidth 25Mb hfsc(realtime 15Mb linkshare 25Mb
> >upperlimit 180Mb) priority 4
> >queue Q-def bandwidth 25Mb hfsc(default linkshare 25Mb
> >upperlimit 128Mb) priority 0
> >
> >block in log
> >pass out
> >
> >antispoof quick for { lo $ext_if }
> >pass in quick on lo0 no state
> >
> >## Regras de merovingio -- bridge
> >pass in on $ext_if proto tcp to ($ext_if) port ssh queue(local)
> >
> >pass in on $int_if
> >queue(Q-def Q-pri) label "int-DEFA"
> >pass in on $int_if proto icmp
> >queue(Q-icmp ) label "int-ICMP"
> >pass in on $int_if proto gre
> >queue(Q-vpn ) label "int-VPN-gre"
> >pass in on $int_if proto esp
> >queue(Q-vpn ) label "int-VPN-esp"
> >pass in on $int_if proto ah
> >queue(Q-vpn ) label "int-VPN-ah"
> >pass in on $int_if proto l2tp
> >queue(Q-vpn ) label "int-VPN-l2tp"
> >pass in on $int_if proto { tcp udp } to port { 500 4500 }
> >queue(Q-vpn ) label "int-VPN-ipsec"
> >pass in on $int_if proto { tcp udp } to port { pptp }
> >queue(Q-vpn ) label "int-VPN-pptp"
> >pass in on $int_if proto tcp to port { snmp }
> >queue(Q-icmp Q-pri) label "int-ICMP-snmp"
> >pass in on $int_if proto tcp to port { www }
> >queue(Q-http Q-pri) label "int-HTTP"
> >pass in on $int_if proto tcp to port { https }
> >queue(Q-biz Q-pri) label "int-BIZZ-https"
> >pass in on $int_if proto { tcp udp } to port { domain ntp } queue(
> > Q-pri) label "int-PRII"
> >
> >pass in on $int_if to <redes-biz>
> >queue(Q-biz Q-pri) label "int-BIZZ-redes"
> >pass in on $int_if proto tcp to port { smtp pop3 imap 465 995 }
> >queue(Q-mail q-pri) label "int-MAIL"
> >
> >pass in on $ext_if
> >queue(Q-def Q-pri) label "ext-DEFA"
> >pass in on $ext_if proto icmp
> >queue(Q-icmp ) label "ext-ICMP"
> >pass in on $ext_if proto gre
> >queue(Q-vpn ) label "ext-VPN-gre"
> >pass in on $ext_if proto esp
> >queue(Q-vpn ) label "ext-VPN-esp"
> >pass in on $ext_if proto ah
> >queue(Q-vpn ) label "ext-VPN-ah"
> >pass in on $ext_if proto l2tp
> >queue(Q-vpn ) label "ext-VPN-l2tp"
> >pass in on $ext_if proto { tcp udp } to port { 500 4500 }
> >queue(Q-vpn ) label "ext-VPN-ipsec"
> >pass in on $ext_if proto { tcp udp } to port { pptp }
> >queue(Q-vpn ) label "ext-VPN-pptp"
> >pass in on $ext_if proto tcp to port { snmp }
> >queue(Q-icmp Q-pri) label "ext-ICMP-snmp"
> >pass in on $ext_if proto tcp to port { www }
> >queue(Q-http Q-pri) label "ext-HTTP"
> >pass in on $ext_if proto tcp to port { https }
> >queue(Q-biz Q-pri) label "ext-BIZZ-https"
> >pass in on $ext_if proto { tcp udp } to port { domain ntp } queue(
> > Q-pri) label "ext-PRII"
> >
> >pass in on $ext_if from <redes-biz>
> >queue(Q-biz Q-pri) label "ext-BIZZ-https"
> >pass in on $ext_if proto tcp to port { smtp pop3 imap 465 995 }
> >queue(Q-mail q-pri) label "ext-MAIL"
> >
Kind regards,
--
Fernando M. Braga
+55 82 9985-4579
