.. I have seen some similiar messages where people on the list where talking about encrypted VPN's over a BSD firewall, especially those out of WINs boxes ... perhaps this helps
Volker Chris Cameron <[EMAIL PROTECTED]> Sent by: [EMAIL PROTECTED] 11/01/2005 16:23 To [email protected] cc Subject State searches sky rocket / Firewall dies I've sent a very similar message to misc@ on the 5th, and didn't find any help. Today my client asked me to unplug my OpenBSD firewall because of the following problem. Any help, ideas, thoughts, etc. on this will be most appreciated as currently neither OpenBSD or I are looking very good in the eyes of this client. Prior to this problem a OBSD 3.5 firewall using the exact same ruleset (on slower hardware) was in, and didn't have any such problems. The message I sent to misc@: Have a 3.6 firewall/bridge that every once in a while (3 times so far today) will drop numerous packets for about 5 to 10 minutes, only to come back fine again. I turned PF debugging to 'misc', and what I saw in the logs just prior to one of these episodes was: Dec 15 13:02:32 baracus /bsd: pf: State failure on: | Dec 15 13:02:32 baracus /bsd: pf: BAD state: TCP 216.194.85.40:80 216.194.85.40:80 216.79.119.184:1998 [lo=1975532775 high=1975549336 win=65535 modulator=0] [lo=1392693791 high=1392759326 win=16560 modulator=0] 4:2 RA seq=1975532775 ack=1392693791 len=0 ackskew=0 pkts=1:2 dir=in,fwd Dec 15 13:02:32 baracus /bsd: pf: State failure on: | Dec 15 13:02:32 baracus /bsd: pf: BAD state: TCP 216.194.85.40:80 216.194.85.40:80 216.79.119.184:2013 [lo=2974235582 high=2974252143 win=65535 modulator=0] [lo=151789620 high=151855155 win=16560 modulator=0] 4:2 RA seq=2974235582 ack=151789620 len=0 ackskew=0 pkts=1:2 dir=in,fwd Dec 15 13:02:32 baracus /bsd: pf: State failure on: | Dec 15 13:02:32 baracus /bsd: pf: BAD state: TCP 216.194.85.40:80 216.194.85.40:80 216.79.119.184:2011 [lo=2230903065 high=2230919626 win=65535 modulator=0] [lo=1017748228 high=1017813763 win=16560 modulator=0] 4:2 RA seq=2230903065 ack=1017748228 len=0 ackskew=0 pkts=1:2 dir=in,fwd Which is just a very very small piece of all the errors that happened at 13:02, give or take a couple of seconds. The other thing I noticed was the state table stayed relatively empty, however number of searches/second reported skyrocketed. It may just be that I wasn't able to catch it at the right time. The two things I tried were 'set limit states 40000' and 'set optimization aggressive'. Which for all I know may have put off this happening, but certainly didn't prevent it. Any direction on this would be appreciated. This server is 100% a firewall, with nothing else other than sshd running. dmesg (from messages, sorry): Dec 10 14:47:38 baracus /bsd: OpenBSD 3.6 (GENERIC) #59: Fri Sep 17 12:32:57 MDT 2004 Dec 10 14:47:38 baracus /bsd: [EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC Dec 10 14:47:38 baracus /bsd: cpu0: Intel Pentium III ("GenuineIntel" 686-class, 128KB L2 cache) 952 MHz Dec 10 14:47:38 baracus /bsd: cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE Dec 10 14:47:38 baracus /bsd: real mem = 132685824 (129576K) Dec 10 14:47:38 baracus /bsd: avail mem = 114442240 (111760K) Dec 10 14:47:38 baracus /bsd: using 1645 buffers containing 6737920 bytes (6580K) of memory Dec 10 14:47:38 baracus /bsd: mainbus0 (root) Dec 10 14:47:38 baracus /bsd: bios0 at mainbus0: AT/286+(00) BIOS, date 10/09/01, BIOS32 rev. 0 @ 0xfdb70 Dec 10 14:47:38 baracus /bsd: pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000 Dec 10 14:47:38 baracus /bsd: pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf6a20/80 (3 entries) Dec 10 14:47:38 baracus /bsd: pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82801AA LPC" rev 0x00) Dec 10 14:47:38 baracus /bsd: pcibios0: PCI bus #2 is the last bus Dec 10 14:47:38 baracus /bsd: bios0: ROM list: 0xc0000/0x8000 Dec 10 14:47:38 baracus /bsd: cpu0 at mainbus0 Dec 10 14:47:38 baracus /bsd: pci0 at mainbus0 bus 0: configuration mode 1 (no bios) Dec 10 14:47:38 baracus /bsd: pchb0 at pci0 dev 0 function 0 "Intel 82810" rev 0x03: rng active, 9Kb/sec Dec 10 14:47:38 baracus /bsd: vga1 at pci0 dev 1 function 0 "Intel 82810 Graphics" rev 0x03: aperture at 0xdc000000, size 0x4000000 Dec 10 14:47:38 baracus /bsd: wsdisplay0 at vga1: console (80x25, vt100 emulation) Dec 10 14:47:38 baracus /bsd: wsdisplay0: screen 1-5 added (80x25, vt100 emulation) Dec 10 14:47:38 baracus /bsd: ppb0 at pci0 dev 30 function 0 "Intel 82801AA Hub-to-PCI" rev 0x02 Dec 10 14:47:38 baracus /bsd: pci1 at ppb0 bus 1 Dec 10 14:47:38 baracus /bsd: ppb1 at pci1 dev 3 function 0 "DEC 21152 PCI-PCI" rev 0x03 Dec 10 14:47:38 baracus /bsd: pci2 at ppb1 bus 2 Dec 10 14:47:38 baracus /bsd: fxp0 at pci2 dev 4 function 0 "Intel 82557" rev 0x05: irq 11, address 00:03:47:08:e2:61 Dec 10 14:47:38 baracus /bsd: inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 0 Dec 10 14:47:38 baracus /bsd: fxp1 at pci2 dev 5 function 0 "Intel 82557" rev 0x05: irq 10, address 00:03:47:08:e2:62 Dec 10 14:47:38 baracus /bsd: inphy1 at fxp1 phy 1: i82555 10/100 media interface, rev. 0 Dec 10 14:47:38 baracus /bsd: ichpcib0 at pci0 dev 31 function 0 "Intel 82801AA LPC" rev 0x02 Dec 10 14:47:38 baracus /bsd: pciide0 at pci0 dev 31 function 1 "Intel 82801AA IDE" rev 0x02: DMA, channel 0 wired to compatibility, channel 1 wired to comp atibility Dec 10 14:47:38 baracus /bsd: wd0 at pciide0 channel 0 drive 0: <Maxtor 2B010H1> Dec 10 14:47:38 baracus /bsd: wd0: 16-sector PIO, LBA, 9771MB, 20012832 sectors Dec 10 14:47:38 baracus /bsd: wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4 Dec 10 14:47:38 baracus /bsd: pciide0: channel 1 disabled (no drives) Dec 10 14:47:38 baracus /bsd: uhci0 at pci0 dev 31 function 2 "Intel 82801AA USB" rev 0x02: irq 12 Dec 10 14:47:38 baracus /bsd: usb0 at uhci0: USB revision 1.0 Dec 10 14:47:38 baracus /bsd: uhub0 at usb0 Dec 10 14:47:38 baracus /bsd: uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 Dec 10 14:47:38 baracus /bsd: uhub0: 2 ports with 2 removable, self powered Dec 10 14:47:38 baracus /bsd: "Intel 82801AA SMBus" rev 0x02 at pci0 dev 31 function 3 not configured Dec 10 14:47:38 baracus /bsd: auich0 at pci0 dev 31 function 5 "Intel 82801AA AC97" rev 0x02: irq 10, ICH AC97 Dec 10 14:47:38 baracus /bsd: ac97: codec id 0x83847609 (SigmaTel STAC9721/23) Dec 10 14:47:38 baracus /bsd: ac97: codec features 18 bit DAC, 18 bit ADC, SigmaTel 3D Dec 10 14:47:38 baracus /bsd: audio0 at auich0 Dec 10 14:47:38 baracus /bsd: isa0 at ichpcib0 Dec 10 14:47:38 baracus /bsd: isadma0 at isa0 Dec 10 14:47:38 baracus /bsd: pckbc0 at isa0 port 0x60/5 Dec 10 14:47:38 baracus /bsd: pckbd0 at pckbc0 (kbd slot) Dec 10 14:47:38 baracus /bsd: pckbc0: using irq 1 for kbd slot Dec 10 14:47:38 baracus /bsd: wskbd0 at pckbd0: console keyboard, using wsdisplay0 Dec 10 14:47:38 baracus /bsd: pcppi0 at isa0 port 0x61 Dec 10 14:47:38 baracus /bsd: midi0 at pcppi0: <PC speaker> Dec 10 14:47:38 baracus /bsd: sysbeep0 at pcppi0 Dec 10 14:47:38 baracus /bsd: lpt0 at isa0 port 0x378/4 irq 7 Dec 10 14:47:38 baracus /bsd: lm0 at isa0 port 0x290/8: W83627HF Dec 10 14:47:38 baracus /bsd: npx0 at isa0 port 0xf0/16: using exception 16 Dec 10 14:47:38 baracus /bsd: pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo Dec 10 14:47:38 baracus /bsd: fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 Dec 10 14:47:38 baracus /bsd: fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec Dec 10 14:47:38 baracus /bsd: biomask f36d netmask ff6d ttymask ffef Dec 10 14:47:38 baracus /bsd: pctr: 686-class user-level performance counters enabled Dec 10 14:47:38 baracus /bsd: mtrr: Pentium Pro MTRR support Dec 10 14:47:38 baracus /bsd: dkcsum: wd0 matched BIOS disk 80 Dec 10 14:47:38 baracus /bsd: root on wd0a Dec 10 14:47:38 baracus /bsd: rootdev=0x0 rrootdev=0x300 rawdev=0x302 Dec 10 14:47:38 baracus savecore: no core dump pf.conf, slightly sanitized: $ sudo cat /etc/pf.conf ################################################### ## MACROS ################################################### ## Settings ########### set limit states 40000 set optimization aggressive set debug misc ## Interfaces ############# # External to internet (bridge0) ext_if = "fxp0" # Internal to lan (bridge0) int_if = "fxp1" # Loopback interface lpb_if = "lo0" ## Servers ########## # baracus # OpenBSD 3.6 baracus = "216.194.85.46" # mightythor.advantcomp.com mightythor1 = "216.194.85.43" mightythor2 = "216.194.85.44" mightythor = "{" $mightythor1 $mightythor2 "}" # Windows 2000 Server netdisciple1 = "216.194.85.34" netdisciple2 = "216.194.85.35" netdisciple = "{" $netdisciple1 $netdisciple2 "}" # Windows 2003 Server bizminerdb = "216.194.85.45" # FreeBSD 4.8 boudica1 = "216.194.85.48" boudica2 = "216.194.85.50" boudica = "{" $boudica1 $boudica2 "}" # Windows 2003 Server ardvark = "216.194.85.40" # Redhat Linux 9.0 locnet1 = "216.194.85.37" locnet2 = "216.194.85.39" # OpenBSD 3.5 gak1 = "216.194.85.41" gak2 = "216.194.85.42" # Crap devadv = "216.194.85.51" ## Private IPs ############## private = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }" ################################################### ## OPTIONS ################################################### set loginterface fxp0 set optimization normal set block-policy drop ################################################### ## NORMALIZATION ################################################### # scrub incoming packets scrub in on $ext_if all ################################################### ## TRANSLATION ################################################### # redirect connections to boudica from any on port 26 to # port 25 on the server rdr on $ext_if proto tcp from any to $boudica1 port 26 -> $boudica1 port 25 ################################################### ## FILTER RULES ################################################### ## Pass in/out on loopback interface #################################### pass in quick on $lpb_if all pass out quick on $lpb_if all ## Filter on external interface ############################### # default deny in and log block in log on $ext_if all #pass in quick on $ext_if all # stop spoofing attempts and log them block in quick log on $ext_if from $private to any block out quick log on $ext_if from any to $private # let everything out that isn't spoofed pass out quick on $ext_if from any to any keep state # baracus pass in quick on $ext_if proto tcp from any to $baracus port 22 keep state label BARACUS # pass in quick on $ext_if proto tcp from any to $netdisciple port { 21 25 80 110 366 443 993 995 3389 9999 } flags S/SA keep state label NETDISCIPLE # pass in quick on $ext_if proto tcp from any to $mightythor port { 21 80 443 3389 } flags S/SA keep state label MIGHTYTHOR # # no external access except through authpf rules block in quick on $ext_if from 64.41.168.243 to any pass in quick on $ext_if proto tcp from any to $bizminerdb port 3389 flags S/SA keep state # pass in quick on $ext_if proto tcp from any to $boudica port { 21 22 25 69 80 110 143 993 } flags S/SA keep state label BOUDICA pass in quick on $ext_if proto udp from any to $boudica port 69 keep state label BOUDICA # pass in quick on $ext_if proto tcp from any to $ardvark port { 21 80 1433 3389 } flags S/SA keep state label ARDVARK # pass in quick on $ext_if proto tcp from any to $locnet2 port { 21 22 80 } flags S/SA keep state label LOCNET pass in quick on $ext_if proto udp from any to $locnet2 port 53 keep state label LOCNET # # - Explicitly block people to Gak block in quick on $ext_if from 208.166.208.214 to $gak1 label GAK # - Allow standard services pass in quick on $ext_if proto tcp from any to $gak1 port { 21 22 25 80 110 143 443 993 } flags S/SA keep state label GAK pass in quick on $ext_if proto tcp from any to $gak1 port 49152 >< 65535 flags S/SA keep state pass in quick on $ext_if proto udp from any to $gak1 port 53 keep state label GAK pass in quick on $ext_if proto tcp from any to $gak2 port { 80 443 } flags S/SA keep state label GAK # - Special People allowed in pass in quick on $ext_if proto tcp from xx.xx.xx.xx to $gak1 port 5432 flags S/SA keep state label GAK pass in quick on $ext_if proto tcp from xx.xx.xx.xx to $gak1 port 5432 flags S/SA keep state label GAK pass in quick on $ext_if proto tcp from xx.xx.xx.xx to $gak1 port 5432 flags S/SA keep state label GAK pass in quick on $ext_if proto tcp from xx.xx.xx.xx to $gak1 port 3306 flags S/SA keep state label GAK # pass in quick on $ext_if proto tcp from any to $devadv port { 21 80 443 1433 3389 } flags S/SA keep state label DEVADV # allow useful ICMP packets for ping pass in quick on $ext_if inet proto icmp all icmp-type echoreq keep state # anchor for authpf to allow certain users special access anchor authpf in on $ext_if block in log on $ext_if all ## Pass in/out on internal interface #################################### pass in quick on $int_if all pass out quick on $int_if all General PF stats: INFO: Status: Enabled for 0 days 00:00:11 Debug: Misc Hostid: 0xd40ec4ab Interface Stats for fxp0 IPv4 IPv6 Bytes In 1779014344 0 Bytes Out 12435348156 352 Packets In Passed 12134747 0 Blocked 464176 0 Packets Out Passed 16979477 0 Blocked 297329 5 State Table Total Rate current entries 162 searches 59072275 57800.7/s inserts 2416059 2364.0/s removals 2415897 2363.9/s Counters match 32351974 31655.6/s bad-offset 0 0.0/s fragment 126 0.1/s short 23 0.0/s normalize 196 0.2/s memory 295524 289.2/s bad-timestamp 0 0.0/s TIMEOUTS: tcp.first 120s tcp.opening 30s tcp.established 86400s tcp.closing 900s tcp.finwait 45s tcp.closed 90s tcp.tsdiff 30s udp.first 60s udp.single 30s udp.multiple 60s icmp.first 20s icmp.error 10s other.first 60s other.single 30s other.multiple 60s frag 30s interval 10s adaptive.start 0 states adaptive.end 0 states src.track 0s LIMITS: states hard limit 40000 src-nodes hard limit 10000 frags hard limit 5000
