2007/7/13, Adriaan <[EMAIL PROTECTED]>:
On 7/13/07, TuxR <[EMAIL PROTECTED]> wrote:
> Hello.
>
> I trying to use OpenBSD under high load and have problems with PF.
>
> When there is very many connections to server in some point other
> connections just failes.
>
> I try to use simple test application that creates 1000 connections to
> server for 1000 iteration. Maximum number I have observed with pf was
> '12' but with 'pfctl -d' all cycle successfully works ('1000').
>
> I try to use following simple test application:
>
> Also I have looked the same when testing 'ab' from apache2
> distribution. 'ab -c 100 -n 100' : maximum 9 iteration with pf enabled
> and 100 without.
>
> There is instant connection closing if "keep state" is enabled. When
> "keep state" is disabled there is following behaviour: in some moment
> the program is waiting for reply but do not get it and connection also
> close because timeout.
>
> I have looked no problems in tcpdump reports. Also no blocked packets
> was in pflog0 interface ('block log all' rule)
>
> I am sure that states limit is not exceed. Now I have
>
> set limit states 500000
> set limit src-nodes 50000
> set limit frags 32000
>
> And `pfctl -si` have normal values.
>
> 'antispoof' and 'scrub' options are not affected. 'set optimization'
> make more bad.
>
> I looked the same behaviour in real use: when there is many
> connection, in some point they just closed.
>
> Any help will be appropriated. Many thanks.
>
> P.S. Sorry for my bad english.
>
Study the execellent 3 part series of OpenBSD developer at
http://undeadly.org/cgi?action=article&sid=20060927091645&mode=expanded
If after following his advice, your firewall still does not perform
adequately come back here with a posting of:
1) dmesg to see what kind of hardware you are using
2) vmstat -i output to show the interrupt rate of the NICs
Using 'systat vmstat" will give you a 'live' view of the interrupt
rate and other resources
3) netstat -m output to see the mbuf stats
4) your pf.conf
Others may have additional suggestions of course ;)
=Adriaan=
Adriaan, thank you for reply.
I believe, this is not hardware problem. The system is not under high
CPU-Load during tests.
Hmmm... Of cource, I have read excelent Daniel Hartmeier's articles.
It runnings on FujitsuSiemens SX200 1U Server, 1 Gb RAM, 2x Intel Xeon
3000 (but for now we using non-SMP kernel).
# dmesg
OpenBSD 4.1 (GENERIC) #1435: Sat Mar 10 19:07:45 MST 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Xeon(TM) CPU 3.00GHz ("GenuineIntel" 686-class) 3 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,
SSE3,MWAIT,DS-CPL,VMX,EST,CNXT-ID,CX16,xTPR
real mem = 1072652288 (1047512K)
avail mem = 971362304 (948596K)
using 4278 buffers containing 53764096 bytes (52504K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+ BIOS, date 10/20/06, BIOS32 rev. 0 @
0xfd66a, SMBIOS rev. 2.34 @ 0x3fee8000 (67 entries)
bios0: FUJITSU SIEMENS PRIMERGY RX200 S3
pcibios0 at bios0: rev 2.1 @ 0xfd590/0xa70
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfde30/432 (25 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82371FB ISA" rev 0x00)
pcibios0: PCI bus #12 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x5800 0xe2800/0x1400!
acpi at mainbus0 not configured
ipmi0 at mainbus0: version 1.5 interface KCS iobase 0xca2/2 spacing 1
cpu0 at mainbus0
cpu0: Enhanced SpeedStep disabled by BIOS
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 vendor "Intel", unknown product 0x25d8 rev 0x92
ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE" rev 0x92
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci3 at ppb2 bus 3
ppb3 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01
pci4 at ppb3 bus 4
ppb4 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
pci5 at ppb4 bus 5
mpi0 at pci5 dev 5 function 0 "Symbios Logic SAS1068" rev 0x01: irq 11
scsibus0 at mpi0: 63 targets
sd0 at scsibus0 targ 0 lun 0: <ATA, HITACHI HDS7225S, A6DA> SCSI3 0/direct fixed
sd0: 238471MB, 238472 cyl, 16 head, 127 sec, 512 bytes/sec, 488390625 sec total
sd1 at scsibus0 targ 1 lun 0: <ATA, HITACHI HDS7225S, A6DA> SCSI3 0/direct fixed
sd1: 238471MB, 238472 cyl, 16 head, 127 sec, 512 bytes/sec, 488390625 sec total
ppb5 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0x92
pci6 at ppb5 bus 6
ppb6 at pci0 dev 4 function 0 "Intel 5000 PCIE" rev 0x92
ppb7 at pci7 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xb5
pci8 at ppb7 bus 8
bge0 at pci8 dev 4 function 0 "Broadcom BCM5715" rev 0xa3, BCM5715 A3
(0x9003): irq 11, address 00:0a:e4:82:11:60
brgphy0 at bge0 phy 1: BCM5714 10/100/1000baseT PHY, rev. 0
bge1 at pci8 dev 4 function 1 "Broadcom BCM5715" rev 0xa3, BCM5715 A3
(0x9003): irq 9, address 00:0a:e4:82:11:61
brgphy1 at bge1 phy 1: BCM5714 10/100/1000baseT PHY, rev. 0
ppb8 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0x92
pci9 at ppb8 bus 9
ppb9 at pci0 dev 6 function 0 "Intel 5000 PCIE" rev 0x92
pci10 at ppb9 bus 10
ppb10 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0x92
pci11 at ppb10 bus 11
pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0x92
pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0x92
pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0x92
pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0x92
pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0x92
pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0x92
pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0x92
uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: irq 11
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: irq 10
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: irq 5
usb2 at uhci2: USB revision 1.0
uhub2 at usb2
uhub2: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3 at pci0 dev 29 function 3 "Intel 6321ESB USB" rev 0x09: irq 9
usb3 at uhci3: USB revision 1.0
uhub3 at usb3
uhub3: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: irq 11
ehci0: timed out waiting for BIOS
usb4 at ehci0: USB revision 2.0
uhub3 at usb3
uhub3: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: irq 11
ehci0: timed out waiting for BIOS
usb4 at ehci0: USB revision 2.0
uhub4 at usb4
uhub4: Intel EHCI root hub, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
ppb11 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xd9
pci12 at ppb11 bus 12
vga1 at pci12 dev 5 function 0 vendor "Matrox", unknown product 0x0522 rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09: PM disabled
pciide0 at pci0 dev 31 function 2 "Intel 6321ESB SATA" rev 0x09: DMA,
channel 0 wired to compatibility, channel 1 wired to co
mpatibility
atapiscsi0 at pciide0 channel 1 drive 1
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <HL-DT-ST, RW/DVD GCC-4244N, 1.00> SCSI0
5/cdrom removable
cd0(pciide0:1:1): using PIO mode 4, Ultra-DMA mode 2
ichiic0 at pci0 dev 31 function 3 "Intel 6321ESB SMBus" rev 0x09: irq 9
iic0 at ichiic0: disabled to avoid ipmi0 interactions
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
biomask ffed netmask ffed ttymask ffef
pctr: user-level cycle counter enabled
dkcsum: sd0 matches BIOS drive 0x80
dkcsum: sd1 matches BIOS drive 0x81
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
________________________________________
# vmstat -i
interrupt total rate
irq11/mpi0 45783943 12
irq11/bge0 11521151 3
irq9/bge1 16868267 4
irq15/pciide0 644 0
irq1/pckbc0 3219 0
irq0/clock 380070563 100
irq8/rtc 486442730 128
Total 940690517 247
______________________________________
I have tried to hard tweaking timeouts and it was just a little
helpfull. (Note that any 'set optimisation' option was harmfull - in
agressive sometimes even first iteration was not complete).
# cat /etc/pf.conf
ext_if="bge1"
int_if="bge0"
me="{10.10.10.101}"
set block-policy drop # pOLITIKA PO-UMOL^ANI@
set limit states 500000
set limit src-nodes 50000
set limit frags 32000
set state-policy floating
set timeout tcp.first 120
set timeout tcp.opening 30
set timeout tcp.established 86400
set timeout tcp.closing 900
set timeout tcp.finwait 60
set timeout tcp.closed 0
set timeout tcp.tsdiff 30
set timeout udp.first 60
set timeout udp.single 30
set timeout udp.multiple 60
set timeout icmp.first 20
set timeout icmp.error 10
set timeout other.first 60
set timeout other.single 30
set timeout other.multiple 60
set timeout frag 100
set timeout interval 1
set timeout adaptive.start 300000
set timeout adaptive.end 600000
set timeout src.track 0
pass log quick on $int_if proto tcp from $me to 10.10.10.10 port 80
# (In 4.1 there is keep state by default)
_____________________________________________
# netstat -m
646 mbufs in use:
560 mbufs allocated to data
20 mbufs allocated to packet headers
66 mbufs allocated to socket names and addresses
560/682/12288 mbuf clusters in use (current/peak/max)
1616 Kbytes allocated to network (79% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
___________________________________________
Thank you.
P.S. Sorry for my bad english.
