Hi All, We saw a strange issue today with two of our CARP'ed firewalls. At two different points in the afternoon, the state table suddenly jumped from it's normal level of around 30,000 entries to the limit of 200,000 entries. As expected, no new states could be created. We drove to our datacenter, logged into to the primary machine and flushed the state table. Strangely, the output from pfctl was something like "Cleared 26 state entries" or some similarly small number. At this point, the state count creeped back up to its normal level of around 30k entries. Our network went back to normal and we drove back to the office. I bumped the state limit up on our CARP master to 600000 and then attempted to copy the pf.conf to the CARP backup server. The scp timed out. I haven't yet made it back to the datacenter but my guess is that the state table is still full on that machine. This is really strange. Wouldn't pfsync clear out the state tables on the backup host when the primary host was cleared with 'pfctl -F state'? Has anybody experienced sudden surges of state entries like this? Denial of service attack perhaps?
Also, I just noticed some strange entries in /var/log/messages: Jan 15 15:57:15 fw-01 /bsd: carp666: ip_output failed: 65 Jan 15 15:57:15 fw-01 /bsd: carp667: ip_output failed: 65 Jan 15 15:57:15 fw-01 /bsd: carp668: ip_output failed: 65 Jan 15 15:57:15 fw-01 /bsd: carp669: ip_output failed: 65 Googling didn't yield anything interesting (beyond CARP source code) for this error. Does anybody have any ideas? Chris OpenBSD 4.0-current (GENERIC) #744: Fri Nov 10 16:16:08 MST 2006 [EMAIL PROTECTED]:/usr/src/sys/arch/amd64/compile/GENERIC real mem = 3757633536 (3669564K) avail mem = 3223883776 (3148324K) using 22937 buffers containing 375971840 bytes (367160K) of memory mainbus0 (root) bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfcfe0 (52 entries) bios0: Sun Microsystems Sun Fire X2200 M2 ipmi0 at mainbus0: version 1.5 interface KCS iobase 0xca2/2 spacing 1 cpu0 at mainbus0: (uniprocessor) cpu0: Dual-Core AMD Opteron(tm) Processor 2214, 2211.65 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB 64b/line 16-way L2 cache cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative pci0 at mainbus0 bus 0: configuration mode 1 "NVIDIA MCP55 Memory" rev 0xa2 at pci0 dev 0 function 0 not configured pcib0 at pci0 dev 1 function 0 "NVIDIA MCP55 ISA" rev 0xa3 nviic0 at pci0 dev 1 function 1 "NVIDIA MCP55 SMBus" rev 0xa3 iic0 at nviic0: disabled to avoid ipmi0 interactions iic1 at nviic0: disabled to avoid ipmi0 interactions ohci0 at pci0 dev 2 function 0 "NVIDIA MCP55 USB" rev 0xa1: irq 15, version 1.0, legacy support usb0 at ohci0: USB revision 1.0 uhub0 at usb0 uhub0: NVIDIA OHCI root hub, rev 1.00/1.00, addr 1 uhub0: 10 ports with 10 removable, self powered ehci0 at pci0 dev 2 function 1 "NVIDIA MCP55 USB" rev 0xa2: irq 7 usb1 at ehci0: USB revision 2.0 uhub1 at usb1 uhub1: NVIDIA EHCI root hub, rev 2.00/1.00, addr 1 uhub1: 10 ports with 10 removable, self powered pciide0 at pci0 dev 4 function 0 "NVIDIA MCP55 IDE" rev 0xa1: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility pciide0: channel 0 disabled (no drives) pciide0: channel 1 ignored (disabled) pciide1 at pci0 dev 5 function 0 "NVIDIA MCP55 SATA" rev 0xa3: DMA pciide1: using irq 10 for native-PCI interrupt wd0 at pciide1 channel 0 drive 0: <HDT722525DLA380> wd0: 16-sector PIO, LBA48, 238475MB, 488397168 sectors wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5 wd1 at pciide1 channel 1 drive 0: <HDT722525DLA380> wd1: 16-sector PIO, LBA48, 238475MB, 488397168 sectors wd1(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 5 ppb0 at pci0 dev 6 function 0 "NVIDIA MCP55 PCI-PCI" rev 0xa2 pci1 at ppb0 bus 1 vga1 at pci1 dev 5 function 0 "ASPEED Technology AST2000" rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) nfe0 at pci0 dev 8 function 0 "NVIDIA MCP55 LAN" rev 0xa3: irq 11, address 00:16:36:76:43:f7 eephy0 at nfe0 phy 2: Marvell 88E1149 Gigabit PHY, rev. 1 nfe1 at pci0 dev 9 function 0 "NVIDIA MCP55 LAN" rev 0xa3: irq 5, address 00:16:36:76:43:f8 eephy1 at nfe1 phy 3: Marvell 88E1149 Gigabit PHY, rev. 1 ppb1 at pci0 dev 10 function 0 "NVIDIA MCP55 PCIE" rev 0xa3 pci2 at ppb1 bus 2 ppb2 at pci0 dev 11 function 0 "NVIDIA MCP55 PCIE" rev 0xa3 pci3 at ppb2 bus 3 ppb3 at pci0 dev 12 function 0 "NVIDIA MCP55 PCIE" rev 0xa3 pci4 at ppb3 bus 4 ppb4 at pci0 dev 13 function 0 "NVIDIA MCP55 PCIE" rev 0xa3 pci5 at ppb4 bus 5 ppb5 at pci5 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xb5 pci6 at ppb5 bus 6 bge0 at pci6 dev 4 function 0 "Broadcom BCM5715" rev 0xa3, BCM5715 A3 (0x9003): irq 15, address 00:16:36:76:43:f5 brgphy0 at bge0 phy 1: BCM5714 10/100/1000baseT PHY, rev. 0 bge1 at pci6 dev 4 function 1 "Broadcom BCM5715" rev 0xa3, BCM5715 A3 (0x9003): irq 10, address 00:16:36:76:43:f6 brgphy1 at bge1 phy 1: BCM5714 10/100/1000baseT PHY, rev. 0 ppb6 at pci0 dev 15 function 0 "NVIDIA MCP55 PCIE" rev 0xa3 pci7 at ppb6 bus 7 pchb0 at pci0 dev 24 function 0 "AMD AMD64 HyperTransport" rev 0x00 pchb1 at pci0 dev 24 function 1 "AMD AMD64 Address Map" rev 0x00 pchb2 at pci0 dev 24 function 2 "AMD AMD64 DRAM Cfg" rev 0x00 pchb3 at pci0 dev 24 function 3 "AMD AMD64 Misc Cfg" rev 0x00 pchb4 at pci0 dev 25 function 0 "AMD AMD64 HyperTransport" rev 0x00 pchb5 at pci0 dev 25 function 1 "AMD AMD64 Address Map" rev 0x00 pchb6 at pci0 dev 25 function 2 "AMD AMD64 DRAM Cfg" rev 0x00 pchb7 at pci0 dev 25 function 3 "AMD AMD64 Misc Cfg" rev 0x00 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 midi0 at pcppi0: <PC speaker> spkr0 at pcppi0 uhidev0 at uhub0 port 2 configuration 1 interface 0 uhidev0: Virtual USB MULTIMEDIA KEYBOARD, rev 1.10/0.01, addr 2, iclass 3/1 ukbd0 at uhidev0: 8 modifier keys, 6 key codes, country code 33 wskbd1 at ukbd0 mux 1 wskbd1: connecting to wsdisplay0 uhidev1 at uhub0 port 2 configuration 1 interface 1 uhidev1: Virtual USB MULTIMEDIA KEYBOARD, rev 1.10/0.01, addr 2, iclass 3/1 ums0 at uhidev1: 3 buttons and Z dir. wsmouse0 at ums0 mux 0 dkcsum: wd0 matches BIOS drive 0x80 dkcsum: wd1 matches BIOS drive 0x81 root on wd0a rootdev=0x0 rrootdev=0x300 rawdev=0x302 carp: pfsync0 demoted group carp to 1 carp: carp666 demoted group carp to 2 carp: carp667 demoted group carp to 3 carp: carp668 demoted group carp to 4 carp: carp669 demoted group carp to 5 carp: carp666 demoted group carp to 4 carp: carp666 demoted group egress to 0 carp: carp669 demoted group carp to 3 carp: pfsync0 demoted group carp to 2 uhidev2 at uhub0 port 9 configuration 1 interface 0 uhidev2: Logitech Logitech USB Keyboard, rev 1.10/15.00, addr 3, iclass 3/1 ukbd1 at uhidev2: 8 modifier keys, 6 key codes wskbd2 at ukbd1 mux 1 wskbd2: connecting to wsdisplay0 uhidev3 at uhub0 port 9 configuration 1 interface 1 uhidev3: Logitech Logitech USB Keyboard, rev 1.10/15.00, addr 3, iclass 3/0 uhidev3: 3 report ids uhid0 at uhidev3 reportid 1: input=2, output=0, feature=0 uhid1 at uhidev3 reportid 2: input=1, output=0, feature=0 ums1 at uhidev3 reportid 3: 0 buttons and Z dir. wsmouse1 at ums1 mux 0 uhidev2: at uhub0 port 9 (addr 3) disconnected wskbd2: disconnecting from wsdisplay0 wskbd2 detached ukbd1 detached uhidev2 detached uhidev3: at uhub0 port 9 (addr 3) disconnected uhid0 detached uhid1 detached wsmouse1 detached ums1 detached uhidev3 detached carp: carp667 demoted group carp to 1 carp: carp668 demoted group carp to 0 uhidev2 at uhub0 port 8 configuration 1 interface 0 uhidev2: Logitech Logitech USB Keyboard, rev 1.10/15.00, addr 3, iclass 3/1 ukbd1 at uhidev2: 8 modifier keys, 6 key codes wskbd2 at ukbd1 mux 1 wskbd2: connecting to wsdisplay0 uhidev3 at uhub0 port 8 configuration 1 interface 1 uhidev3: Logitech Logitech USB Keyboard, rev 1.10/15.00, addr 3, iclass 3/0 uhidev3: 3 report ids uhid0 at uhidev3 reportid 1: input=2, output=0, feature=0 uhid1 at uhidev3 reportid 2: input=1, output=0, feature=0 ums1 at uhidev3 reportid 3: 0 buttons and Z dir. wsmouse1 at ums1 mux 0 uhidev2: at uhub0 port 8 (addr 3) disconnected wskbd2: disconnecting from wsdisplay0 wskbd2 detached ukbd1 detached uhidev2 detached uhidev3: at uhub0 port 8 (addr 3) disconnected uhid0 detached uhid1 detached wsmouse1 detached uhidev2: at uhub0 port 8 (addr 3) disconnected wskbd2: disconnecting from wsdisplay0 wskbd2 detached ukbd1 detached uhidev2 detached uhidev3: at uhub0 port 8 (addr 3) disconnected uhid0 detached uhid1 detached wsmouse1 detached ums1 detached uhidev3 detached uplcom0 at uhub0 port 7 uplcom0: Prolific Technology Inc. USB-Serial Controller, rev 1.10/3.00, addr 3 ucom0 at uplcom0 uplcom0: at uhub0 port 7 (addr 3) disconnected ucom0 detached uplcom0 detached arplookup: unable to enter address for 10.0.0.24 arplookup: unable to enter address for 10.0.0.28 arplookup: unable to enter address for 10.0.0.23 arplookup: unable to enter address for 10.0.0.22 arplookup: unable to enter address for 10.0.0.25 arplookup: unable to enter address for 10.0.0.21 arplookup: unable to enter address for 10.0.0.30 arplookup: unable to enter address for 10.0.0.29 arplookup: unable to enter address for 10.0.0.21 arplookup: unable to enter address for 10.0.0.30 arplookup: unable to enter address for 10.0.0.21 arplookup: unable to enter address for 10.0.0.22 arplookup: unable to enter address for 10.0.0.30 arplookup: unable to enter address for 10.0.0.28 arplookup: unable to enter address for 10.0.0.21 uhidev2 at uhub0 port 6 configuration 1 interface 0 uhidev2: Logitech Logitech USB Keyboard, rev 1.10/15.00, addr 3, iclass 3/1 ukbd1 at uhidev2: 8 modifier keys, 6 key codes wskbd2 at ukbd1 mux 1 wskbd2: connecting to wsdisplay0 uhidev3 at uhub0 port 6 configuration 1 interface 1 uhidev3: Logitech Logitech USB Keyboard, rev 1.10/15.00, addr 3, iclass 3/0 uhidev3: 3 report ids uhid0 at uhidev3 reportid 1: input=2, output=0, feature=0 uhid1 at uhidev3 reportid 2: input=1, output=0, feature=0 ums1 at uhidev3 reportid 3: 0 buttons and Z dir. wsmouse1 at ums1 mux 0 carp666: ip_output failed: 65 carp667: ip_output failed: 65 carp668: ip_output failed: 65 carp669: ip_output failed: 65 uhidev2: at uhub0 port 6 (addr 3) disconnected wskbd2: disconnecting from wsdisplay0 wskbd2 detached ukbd1 detached uhidev2 detached uhidev3: at uhub0 port 6 (addr 3) disconnected uhid0 detached uhid1 detached wsmouse1 detached ums1 detached uhidev3 detached uplcom0 at uhub0 port 6 uplcom0: Prolific Technology Inc. USB-Serial Controller, rev 1.10/3.00, addr 3 ucom0 at uplcom0
