Re: CARP strangeness after 5.0 upgrade
On 2-2-2012 17:34, Matt Hamilton wrote: > Camiel Dobbelaar sentia.nl> writes: > >> Can you post the output of "netstat -m" and a dmesg? > > # netstat -m > 94 mbufs in use: > 88 mbufs allocated to data > 3 mbufs allocated to packet headers > 3 mbufs allocated to socket names and addresses > 87/938/8192 mbuf 2048 byte clusters in use (current/peak/max) > 0/8/8192 mbuf 4096 byte clusters in use (current/peak/max) > 0/8/8192 mbuf 8192 byte clusters in use (current/peak/max) > 0/8/8192 mbuf 9216 byte clusters in use (current/peak/max) > 0/8/8192 mbuf 12288 byte clusters in use (current/peak/max) > 0/8/8192 mbuf 16384 byte clusters in use (current/peak/max) > 0/8/8192 mbuf 65536 byte clusters in use (current/peak/max) > 2308 Kbytes allocated to network (8% in use) > 0 requests for memory denied > 0 requests for memory delayed > 0 calls to protocol drain routines That looks ok. What is the carpdev of carp1 ? -- Cam
Re: CARP strangeness after 5.0 upgrade
Camiel Dobbelaar sentia.nl> writes:
> Can you post the output of "netstat -m" and a dmesg?
# netstat -m
94 mbufs in use:
88 mbufs allocated to data
3 mbufs allocated to packet headers
3 mbufs allocated to socket names and addresses
87/938/8192 mbuf 2048 byte clusters in use (current/peak/max)
0/8/8192 mbuf 4096 byte clusters in use (current/peak/max)
0/8/8192 mbuf 8192 byte clusters in use (current/peak/max)
0/8/8192 mbuf 9216 byte clusters in use (current/peak/max)
0/8/8192 mbuf 12288 byte clusters in use (current/peak/max)
0/8/8192 mbuf 16384 byte clusters in use (current/peak/max)
0/8/8192 mbuf 65536 byte clusters in use (current/peak/max)
2308 Kbytes allocated to network (8% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
And dmesg.boot:
OpenBSD 5.0 (GENERIC) #43: Wed Aug 17 10:10:52 MDT 2011
dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 3.00GHz ("GenuineIntel" 686-class) 3
GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,
CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,F
XSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-I
D,xTPR
real mem = 535818240 (510MB)
avail mem = 517001216 (493MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 03/29/05, BIOS32 rev. 0 @
0xfd770, SMBIOS rev. 2.33 @ 0xd8010 (37 entries)
bios0: vendor Phoenix Technologies LTD version "6.00" date 03/29/2005
bios0: Intel Corporation Canterwood CRB Board
acpi0 at bios0: rev 0
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP ASF! APIC BOOT SSDT
acpi0: wakeup devices CSA_(S5) LAN_(S5) PCIB(S5) COMA(S1) COMB(S1)
AC97(S5) USB1(S3) USB2(S3) USB3(S3) USB4(S3) EUSB(S3)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 199MHz
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 2 (CSA_)
acpiprt2 at acpi0: bus 1 (AGP_)
acpiprt3 at acpi0: bus 3 (PCIB)
acpicpu0 at acpi0: C3
acpibtn0 at acpi0: PWRB
bios0: ROM list: 0xc/0x8000 0xc8000/0x1000 0xc9000/0x1000
0xca000/0x800 0xd8000/0x4000! 0xdc000/0x4000!
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 82875P Host" rev 0x02
intelagp0 at pchb0
agp0 at intelagp0: aperture at 0x0, size 0x800
ppb0 at pci0 dev 1 function 0 "Intel 82875P AGP" rev 0x02
pci1 at ppb0 bus 1
ppb1 at pci0 dev 3 function 0 "Intel 82875P CSA" rev 0x02
pci2 at ppb1 bus 2
em0 at pci2 dev 1 function 0 "Intel PRO/1000CT (82547GI)" rev 0x00:
apic 2 int 18, address 00:40:d0:43:bb:e4
uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: apic
2 int 16
uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: apic
2 int 19
uhci2 at pci0 dev 29 function 2 "Intel 82801EB/ER USB" rev 0x02: apic
2 int 18
uhci3 at pci0 dev 29 function 3 "Intel 82801EB/ER USB" rev 0x02: apic
2 int 16
ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB2" rev 0x02: apic
2 int 23
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb2 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xc2
pci3 at ppb2 bus 3
vga1 at pci3 dev 0 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
xl0 at pci3 dev 1 function 0 "3Com 3c905C 100Base-TX" rev 0x78: apic 2
int 17, address 00:0a:5e:57:3f:27
exphy0 at xl0 phy 24: 3Com internal media interface
em1 at pci3 dev 2 function 0 "Intel PRO/1000MT (82541GI)" rev 0x00:
apic 2 int 18, address 00:40:d0:43:bb:e5
ichpcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
pciide0 at pci0 dev 31 function 1 "Intel 82801EB/ER IDE" rev 0x02:
DMA, channel 0 configured to compatibility, channel 1 configured
to compatibility
wd0 at pciide0 channel 0 drive 0:
wd0: 16-sector PIO, LBA, 76319MB, 156301488 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
pciide0: channel 1 disabled (no drives)
ichiic0 at pci0 dev 31 function 3 "Intel 82801EB/ER SMBus" rev 0x02:
apic 2 int 17
iic0 at ichiic0
adt0 at iic0 addr 0x2e: adm1027 rev 0x6a
spdmem0 at iic0 addr 0x52: 256MB DDR SDRAM ECC PC3200CL3.0
spdmem1 at iic0 addr 0x53: 256MB DDR SDRAM ECC PC3200CL3.0
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci3: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: usin
Re: CARP strangeness after 5.0 upgrade
On 2-2-2012 16:38, Matt Hamilton wrote: > Camiel Dobbelaar sentia.nl> writes: > >> Can you show the output of: >> - ifconfig carp >> - ifconfig -g carp >> - netstat -s -p carp >> - sysctl net.inet.carp > > Ahhh... actually, I noticed mbuf memory error with one of these: > > # netstat -s -p carp > carp: > 3112793 packets received (IPv4) > 0 packets received (IPv6) > 0 packets discarded for bad interface > 0 packets discarded for wrong TTL > 0 packets shorter than header > 0 discarded for bad checksums > 0 discarded packets with a bad version > 1347685 discarded because packet too short > 0 discarded for bad authentication > 0 discarded for unknown vhid > 0 discarded because of a bad address list > 4512672 packets sent (IPv4) > 0 packets sent (IPv6) > 8589 send failed due to mbuf memory error > 391 transitions to master > > And also increasing the carp logging I now see: > > Feb 1 13:50:02 fw1 /bsd: carp: carp1 demoted group carp by -1 to 0 (< > snderrors) > Feb 1 13:50:04 fw1 /bsd: carp0: state transition: BACKUP -> MASTER > Feb 1 13:56:48 fw1 /bsd: carp: carp1 demoted group carp by 1 to 1 (> > snderrors) > Feb 1 13:56:48 fw1 /bsd: carp0: state transition: MASTER -> BACKUP > > So how do I go about debugging this? Can you post the output of "netstat -m" and a dmesg?
Re: CARP strangeness after 5.0 upgrade
Camiel Dobbelaar sentia.nl> writes: > Can you show the output of: > - ifconfig carp > - ifconfig -g carp > - netstat -s -p carp > - sysctl net.inet.carp Ahhh... actually, I noticed mbuf memory error with one of these: # netstat -s -p carp carp: 3112793 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for wrong TTL 0 packets shorter than header 0 discarded for bad checksums 0 discarded packets with a bad version 1347685 discarded because packet too short 0 discarded for bad authentication 0 discarded for unknown vhid 0 discarded because of a bad address list 4512672 packets sent (IPv4) 0 packets sent (IPv6) 8589 send failed due to mbuf memory error 391 transitions to master And also increasing the carp logging I now see: Feb 1 13:50:02 fw1 /bsd: carp: carp1 demoted group carp by -1 to 0 (< snderrors) Feb 1 13:50:04 fw1 /bsd: carp0: state transition: BACKUP -> MASTER Feb 1 13:56:48 fw1 /bsd: carp: carp1 demoted group carp by 1 to 1 (> snderrors) Feb 1 13:56:48 fw1 /bsd: carp0: state transition: MASTER -> BACKUP So how do I go about debugging this? > Do you use pfsync? If yes, can you try adding "keep state (no-sync)" to > the carp rules? I tried adding this, no effect. I also tried removing IPv6 from the interface as someone suggested, but that didn't help either it seems. -Matt
Re: CARP strangeness after 5.0 upgrade
On 01/25/12 18:23, Matt Hamilton wrote: > > pass in quick on $ext_if proto carp from $fw_ext_ips to 224.0.0.18 > queue carp_out > pass in quick on $int_if proto carp from $fw_int_ips to 224.0.0.18 > queue carp_in > pass out quick on $ext_if proto carp from $fw_ext_ips to 224.0.0.18 > queue carp_out > pass out quick on $int_if proto carp from $fw_ext_ips to 224.0.0.18 > queue carp_in And $fw_ext_ips/$fw_int_ips do really contain the ip addresses of BOTH boxes? > > I don't understand why the master is the one with the highest > advskew. This is the same on the inside carp interface too. You said you saw carp advertisments on the net. Who is sending those? Can you set sysctl net.inet.carp.log=7 and see if any carp-related errors appear in the syslog? /m
Re: CARP strangeness after 5.0 upgrade
On 25-1-2012 18:23, Matt Hamilton wrote: > I'm also getting strange weirdnesses with carp on 5.0. I too upgraded > from quite an old 4.x version (4.6 IIRC). > > The main thing I'm seeing is my master and backup switching back and > forth quite a few times. This is a pair of firewalls with carp > running on both the inside and outside firewall interfaces. > > According to tcpdump I can see advertisements from the master being > broadcast, but I never see any broadcast from the backup (I can't > work out if that is correct behaviour or not). > > My PF rules allow the CARP packets through: > > pass in quick on $ext_if proto carp from $fw_ext_ips to 224.0.0.18 > queue carp_out > pass in quick on $int_if proto carp from $fw_int_ips to 224.0.0.18 > queue carp_in > pass out quick on $ext_if proto carp from $fw_ext_ips to 224.0.0.18 > queue carp_out > pass out quick on $int_if proto carp from $fw_ext_ips to 224.0.0.18 > queue carp_in > > And according to pfctl -sr -vv I can see that those rules are indeed > matching packets. > > The very odd thing is that on FW1: > > carp: BACKUP carpdev em0 vhid 1 advbase 1 advskew 10 > > and on FW2: > > carp: MASTER carpdev em1 vhid 2 advbase 1 advskew 200 > > I don't understand why the master is the one with the highest > advskew. This is the same on the inside carp interface too. Can you show the output of: - ifconfig carp - ifconfig -g carp - netstat -s -p carp - sysctl net.inet.carp Do you use pfsync? If yes, can you try adding "keep state (no-sync)" to the carp rules? -- Cam
Re: CARP strangeness after 5.0 upgrade
I'm also getting strange weirdnesses with carp on 5.0. I too upgraded from quite an old 4.x version (4.6 IIRC). The main thing I'm seeing is my master and backup switching back and forth quite a few times. This is a pair of firewalls with carp running on both the inside and outside firewall interfaces. According to tcpdump I can see advertisements from the master being broadcast, but I never see any broadcast from the backup (I can't work out if that is correct behaviour or not). My PF rules allow the CARP packets through: pass in quick on $ext_if proto carp from $fw_ext_ips to 224.0.0.18 queue carp_out pass in quick on $int_if proto carp from $fw_int_ips to 224.0.0.18 queue carp_in pass out quick on $ext_if proto carp from $fw_ext_ips to 224.0.0.18 queue carp_out pass out quick on $int_if proto carp from $fw_ext_ips to 224.0.0.18 queue carp_in And according to pfctl -sr -vv I can see that those rules are indeed matching packets. The very odd thing is that on FW1: carp: BACKUP carpdev em0 vhid 1 advbase 1 advskew 10 and on FW2: carp: MASTER carpdev em1 vhid 2 advbase 1 advskew 200 I don't understand why the master is the one with the highest advskew. This is the same on the inside carp interface too. Any ideas? -Matt
Re: CARP strangeness after 5.0 upgrade
On 01/12/12 00:05, Markus Wernig wrote: > If I set net.inet.carp.log=7, I get lots of the following on both fws, > only for carp1 and carp2, never for carp0 and carp3: > carp2: ip_output failed: 65 > carp1: ip_output failed: 65 > carp2: ip_output failed: 65 > carp1: ip_output failed: 65 > carp2: ip_output failed: 65 > carp1: ip_output failed: 65 Hi all After another round of reboots (no config changed) this has now shifted to carp2 and carp3: Jan 12 08:33:17 fw1 /bsd: carp2: ip_output failed: 65 Jan 12 08:33:17 fw1 /bsd: carp3: ip_output failed: 65 Jan 12 08:33:18 fw1 /bsd: carp2: ip_output failed: 65 Jan 12 08:33:18 fw1 /bsd: carp3: ip_output failed: 65 And consequently tcpdump shows outgoing carp traffic on em0 and em1 only. Does anybody have an idea where to search further? krgds /markus
CARP strangeness after 5.0 upgrade
Hello all
I have recently upgraded a pair of CARPed firewalls from 4.6 to 5.0
(late, I know ...) after almost 2 years of absolutely flawless operation
(ipv4 interfaces only).
I have changed all the nat/rdr rules in pf.conf to the new syntax, not
changed any other fw/nw setting (at least to my knowledge - I used
sysmerge in the process, carefully, and haven't noticed any fw/nw
related changes in any file. The boxes are rather straight forwardly
configured "plain" firewalls and very close to the default settings).
They have 4 interfaces each, the external (egress, carp0 on em0) one
being connected to the provider's switches (professional gear, Cisco or
the like), the dmz (internal, carp1-3 on em1-3) ones being connected to
a pair of levelone gsw-1641 ("web smart switch", the cheap stuff).
The two fw (fw1=master, and fw2=backup) and switches have been rebooted
multiple times by now.
The problem now is that the CARP master selection leads to weird
results. After rebooting both, I get the following picture:
fw1 (master, advbase 1 advskew 1):
carp0: BACKUP
carp1: MASTER
carp2: MASTER
carp3: BACKUP
ifconfig -g carp
carp: carp demote count 3
fw2 (backup, advbase 1 advskew 10)
carp0: MASTER
carp1: MASTER
carp2: MASTER
carp3: MASTER
ifconfig -g carp
carp: carp demote count 2
I get the following in dmesg on fw1:
carp: carp0 demoted group carp by 1 to 129 (carpdev)
carp: carp1 demoted group carp by 1 to 130 (carpdev)
carp: carp2 demoted group carp by 1 to 131 (carpdev)
carp: carp3 demoted group carp by 1 to 132 (carpdev)
carp: carp2 demoted group carp by -1 to 131 (carpdev)
carp: carp2 demoted group xfer by -1 to 0 (carpdev)
carp: carp0 demoted group carp by -1 to 130 (carpdev)
carp: pfsync0 demoted group carp by 1 to 131 (pfsync bulk start)
carp: pfsync0 demoted group pfsync by 1 to 1 (pfsync bulk start)
carp: carp3 demoted group carp by -1 to 130 (carpdev)
carp: carp3 demoted group mgmt by -1 to 0 (carpdev)
carp: carp1 demoted group carp by -1 to 129 (carpdev)
carp: carp1 demoted group coca by -1 to 0 (carpdev)
carp2: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp: pfsync0 demoted group carp by -1 to 128 (pfsync bulk done)
carp: pfsync0 demoted group pfsync by -1 to 0 (pfsync bulk done)
carp: carp2 demoted group carp by 1 to 129 (> snderrors)
carp: carp1 demoted group carp by 1 to 130 (> snderrors)
carp: carp1 demoted group coca by 1 to 1 (> snderrors)
carp: carp2 demoted group xfer by 1 to 1 (> snderrors)
carp0: state transition: BACKUP -> MASTER
carp3: state transition: BACKUP -> MASTER
carp: carp3 demoted group carp by 1 to 3 (> snderrors)
carp: carp3 demoted group mgmt by 1 to 1 (> snderrors)
carp0: state transition: MASTER -> BACKUP
nd6_na_input: duplicate IP6 address fe80:0008::0200:5eff:fe00:01c8
carp3: state transition: MASTER -> BACKUP
dmesg on fw2 gives this:
carp: carp0 demoted group carp by 1 to 129 (carpdev)
carp: carp1 demoted group carp by 1 to 130 (carpdev)
carp: carp2 demoted group carp by 1 to 131 (carpdev)
carp: carp3 demoted group carp by 1 to 132 (carpdev)
carp: pfsync0 demoted group carp by 1 to 133 (pfsync bulk start)
carp: pfsync0 demoted group pfsync by 1 to 1 (pfsync bulk start)
carp: carp2 demoted group carp by -1 to 132 (carpdev)
carp: carp2 demoted group xfer by -1 to 0 (carpdev)
carp: carp1 demoted group carp by -1 to 131 (carpdev)
carp: carp1 demoted group coca by -1 to 0 (carpdev)
carp: carp0 demoted group carp by -1 to 130 (carpdev)
carp: carp3 demoted group carp by -1 to 129 (carpdev)
carp: carp3 demoted group mgmt by -1 to 0 (carpdev)
carp: pfsync0 demoted group carp by -1 to 128 (pfsync bulk done)
carp: pfsync0 demoted group pfsync by -1 to 0 (pfsync bulk done)
carp2: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp: carp2 demoted group carp by 1 to 129 (> snderrors)
carp: carp1 demoted group carp by 1 to 130 (> snderrors)
carp: carp1 demoted group coca by 1 to 1 (> snderrors)
carp: carp2 demoted group xfer by 1 to 1 (> snderrors)
carp0: state transition: BACKUP -> MASTER
carp3: state transition: BACKUP -> MASTER
carp: carp3 demoted group carp by 1 to 3 (> snderrors)
carp: carp3 demoted group mgmt by 1 to 1 (> snderrors)
carp0: state transition: MASTER -> BACKUP
nd6_na_input: duplicate IP6 address fe80:0008::0200:5eff:fe00:01c8
arp info overwritten for 10.10.10.100 by 00:1e:68:9a:e4:4f on em2
nd6_na_input: duplicate IP6 address fe80:0009::0200:5eff:fe00:01c9
carp3: state transition: MASTER -> BACKUP
nd6_na_input: duplicate IP6 address fe80:000b::0200:5eff:fe00:01ff
nd6_na_input: duplicate IP6 address fe80:000a::0200:5eff:fe00:01d2
carp0: state transition: BACKUP -> MASTER
carp3: state transition: BACKUP -> MASTER
carp: carp3 demoted group carp by -1 to 2 (< snderrors)
carp: carp3 demoted group mgmt by -1 to 0 (< snderrors)
nd6_na_input: duplicate IP6 address fe80:000a::0200:5eff:fe00:01d2
nd6_na_input: duplicate IP6 address fe80:0009::0200:5eff:fe00:01c9
carp0: state transition: MASTER -> BACKUP
nd6_na
