On 3/29/15 12:38 PM, mxb wrote: > Probably your PF rules. > put in ‘pass quick proto icmp’.
No joy. This did not improve on the existing ICMP rule in pf.conf. I think the root problem is that on both firewalls the physical and CARP interface addresses are bound to lo0 instead of vic1. Here both .98 (CARP) and .99 (physical) should be bound to vic1 instead of lo0: netstat -nr -f inet | grep 12.220.174 default 12.220.174.97 UGS 0 4 - 8 vic1 12.220.174.96/27 link#2 UC 2 0 - 4 vic1 12.220.174.98 00:00:5e:00:01:dd HLl 0 0 - 1 lo0 12.220.174.99 00:50:56:b2:33:0e UHLl 0 8 - 1 lo0 hostname.vic1: inet 12.220.174.99 255.255.255.224 12.220.174.127 up hostname.carp221: inet 12.220.174.98 255.255.255.224 12.20.174.127 vhid 221 carpdev vic1 advskew 1 pass ****** CARP is up and MASTER/BACKUP state changes work between boxes, but neither firewall can ping other hosts or vice-versa via the CARP interface. How to get those interfaces to bind to vic1 instead of lo0? Thanks! dn > > >> On 28 mar 2015, at 00:59, David Newman <[email protected]> wrote: >> >> Greetings. In preparation for upgrading two CARP+pfsync boxes to >> 5.6/i386, I put together a lab network to test new firewall rules. >> >> Topology is pretty simple: >> >> outside box (vic0) <-> (vic1) two carp boxes (vic0) <-> inside box >> >> with a third interface on each firewall for pfsync traffic. I'm focused >> here on the outside box pinging the carp box's outside CARP interface. >> >> In the lab network everyone can ping everyone else, except for the CARP >> interfaces -- these are not pingable. Hosts on either side of the >> firewall can ping the underlying interfaces that the CARP interfaces are >> bound to. >> >> Also, 'netstat -f inet -nr' shows that CARP interfaces are bound to lo0. >> On the production boxes these systems model, carp interfaces are bound >> to the underlying physical interfaces. >> >> tcpdump on the physical interface of the master firewall says the >> outside box ARPs for the CARP interface, and the firewall sends an ARP >> response with the CARP interface's IP and MAC addresses. >> >> Thanks in advance for troubleshooting clues -- this is almost certainly >> a misconfiguration but I'm not sure where. >> >> dn >> >> Outside box's hostname.vic0: >> inet 12.220.174.101 255.255.255.224 12.220.174.127 >> >> FW1 hostname.vic1: >> inet 12.220.174.99 255.255.255.224 12.220.174.127 >> >> FW1 hostname.carp221: >> inet 12.220.174.98 255.255.255.224 12.220.174.127 vhid 221 advskew 1 >> pass ***** carpdev vic1 carppeer 12.220.174.100 >> >> FW1 ifconfig vic1: >> vic1: >> flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6> >> mtu 1500 >> lladdr 00:50:56:b2:33:0e >> priority: 0 >> groups: egress >> media: Ethernet autoselect >> status: active >> inet 12.220.174.99 netmask 0xffffffe0 broadcast 12.220.174.127 >> >> FW1 ifconfig carp221: >> net 12.220.174.98 255.255.255.224 12.220.174.127 vhid 221 advskew 1 pass >> w00h00 carpdev vic1 carppeer 12.220.174.100 >> # ifconfig carp221 >> carp221: flags=28843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,NOINET6> mtu >> 1500 >> lladdr 00:00:5e:00:01:dd >> priority: 0 >> carp: MASTER carpdev vic1 vhid 221 advbase 1 advskew 1 carppeer >> 12.220.174.100 >> groups: carp >> status: master >> inet 12.220.174.98 netmask 0xffffffe0 broadcast 12.220.174.127 >> >> FW1 netstat -f inet -nr: >> # netstat -f inet -nr >> Routing tables >> >> Internet: >> Destination Gateway Flags Refs Use Mtu Prio >> Iface >> default 12.220.174.97 UGS 0 38 - 8 vic1 >> 12.220.174.96/27 link#2 UC 2 0 - 4 vic1 >> 12.220.174.98 00:00:5e:00:01:dd HLl 0 0 - 1 >> lo0 # <-- NOTE lo0 BINDING >> 12.220.174.99 00:50:56:b2:33:0e UHLl 0 0 - 1 lo0 >> 12.220.174.100 00:50:56:b2:32:94 UHLc 0 274 - 4 vic1 >> 12.220.174.101 00:50:56:b2:5e:b5 UHLc 0 5 - 4 vic1 >> 127/8 127.0.0.1 UGRS 0 0 32768 8 lo0 >> 127.0.0.1 127.0.0.1 UH 1 4 32768 4 lo0 >> >> >> FW2 hostname.vic1: >> inet 12.220.174.100 255.255.255.224 12.220.174.127 >> >> FW2 hostname.carp221: >> inet 12.220.174.98 255.255.255.224 12.220.174.127 vhid 221 advskew 128 >> pass ***** carpdev vic1 carppeer 12.220.174.99 >> >> FW2 ifconfig carp221: >> carp221: flags=28843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,NOINET6> mtu >> 1500 >> lladdr 00:00:5e:00:01:dd >> priority: 0 >> carp: BACKUP carpdev vic1 vhid 221 advbase 1 advskew 128 >> carppeer 12.220.174.99 >> groups: carp >> status: backup >> inet 12.220.174.98 netmask 0xffffffe0 broadcast 12.220.174.127 >> >> pf.conf on both boxes: >> >> # interfaces >> pfsync0_if = "vic2" >> carp_dev = "{ vic0, vic1 }" >> >> set skip on lo >> >> ################## >> # Packet filtering >> ################## >> >> block return # block stateless traffic >> #pass # establish keep-state >> >> # By default, do not permit remote connections to X11 >> block return in on ! lo0 proto tcp to port 6000:6010 >> >> # icmp handling -- FIX THIS to specify ICMP types >> pass log inet proto icmp all >> >> # carp and pfsync >> pass on { $pfsync0_if } proto pfsync >> pass on $carp_dev proto carp >> >> FW1 dmesg: >> >> OpenBSD 5.6 (GENERIC.MP) #299: Fri Aug 8 00:10:33 MDT 2014 >> [email protected]:/usr/src/sys/arch/i386/compile/GENERIC.MP >> cpu0: Intel(R) Xeon(R) CPU E5649 @ 2.53GHz ("GenuineIntel" 686-class) >> 2.54 GHz >> cpu0: >> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,MMX,FXSR,SSE,SSE2,SS,NXE,LONG,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,LAHF,PERF,ITSC >> real mem = 536309760 (511MB) >> avail mem = 515063808 (491MB) >> mpath0 at root >> scsibus0 at mpath0: 256 targets >> mainbus0 at root >> bios0 at mainbus0: AT/286+ BIOS, date 04/14/14, BIOS32 rev. 0 @ 0xfd780, >> SMBIOS rev. 2.4 @ 0xe0010 (364 entries) >> bios0: vendor Phoenix Technologies LTD version "6.00" date 04/14/2014 >> bios0: VMware, Inc. VMware Virtual Platform >> acpi0 at bios0: rev 2 >> acpi0: sleep states S0 S1 S4 S5 >> acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET >> acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) >> S3F0(S3) S4F0(S3) S5F0(S3) S6F0(S3) S7F0(S3) S8F0(S3) S9F0(S3) S10F(S3) >> S11F(S3) S12F(S3) S13F(S3) [...] >> acpitimer0 at acpi0: 3579545 Hz, 24 bits >> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat >> cpu0 at mainbus0: apid 0 (boot processor) >> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges >> cpu0: apic clock running at 65MHz >> cpu1 at mainbus0: apid 1 (application processor) >> cpu1: Intel(R) Xeon(R) CPU E5649 @ 2.53GHz ("GenuineIntel" 686-class) >> 2.54 GHz >> cpu1: >> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,MMX,FXSR,SSE,SSE2,SS,NXE,LONG,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,LAHF,PERF,ITSC >> ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 24 pins >> acpimcfg0 at acpi0 addr 0xf0000000, bus 0-127 >> acpihpet0 at acpi0: 14318179 Hz >> acpiprt0 at acpi0: bus 0 (PCI0) >> acpicpu0 at acpi0 >> acpicpu1 at acpi0 >> acpibat0 at acpi0: BAT1 not present >> acpibat1 at acpi0: BAT2 not present >> acpiac0 at acpi0: AC unit online >> acpibtn0 at acpi0: SLPB >> acpibtn1 at acpi0: LID_ >> bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 0xc9000/0x1000 >> 0xca000/0x1000 0xcb000/0x1000 0xcc000/0x1e00! 0xdc000/0x4000! >> 0xe0000/0x8000! >> vmt0 at mainbus0 >> pci0 at mainbus0 bus 0: configuration mode 1 (bios) >> pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01 >> ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01 >> pci1 at ppb0 bus 1 >> piixpcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08 >> pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, >> channel 0 configured to compatibility, channel 1 configured to compatibility >> pciide0: channel 0 disabled (no drives) >> atapiscsi0 at pciide0 channel 1 drive 0 >> scsibus1 at atapiscsi0: 2 targets >> cd0 at scsibus1 targ 0 lun 0: <NECVMWar, VMware IDE CDR10, 1.00> ATAPI >> 5/cdrom removable >> cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2 >> piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus >> disabled >> "VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured >> vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00 >> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) >> wsdisplay0: screen 1-5 added (80x25, vt100 emulation) >> ppb1 at pci0 dev 17 function 0 "VMware PCI" rev 0x02 >> pci2 at ppb1 bus 2 >> vic0 at pci2 dev 0 function 0 "AMD 79c970 PCnet-PCI" rev 0x10: apic 2 >> int 18, address 00:50:56:b2:06:d6 >> vic1 at pci2 dev 2 function 0 "AMD 79c970 PCnet-PCI" rev 0x10: apic 2 >> int 16, address 00:50:56:b2:33:0e >> vic2 at pci2 dev 3 function 0 "AMD 79c970 PCnet-PCI" rev 0x10: apic 2 >> int 17, address 00:50:56:b2:4e:98 >> vic3 at pci2 dev 4 function 0 "AMD 79c970 PCnet-PCI" rev 0x10: apic 2 >> int 18, address 00:50:56:b2:40:1a >> ppb2 at pci0 dev 21 function 0 "VMware PCIE" rev 0x01 >> pci3 at ppb2 bus 3 >> mpi0 at pci3 dev 0 function 0 "Symbios Logic SAS1068" rev 0x01: apic 2 >> int 18 >> mpi0: SAS3444, firmware 1.3.41.32 >> scsibus2 at mpi0: 256 targets, initiator 16 >> sd0 at scsibus2 targ 0 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct >> fixed >> sd0: 8192MB, 512 bytes/sector, 16777216 sectors >> ppb3 at pci0 dev 21 function 1 "VMware PCIE" rev 0x01 >> pci4 at ppb3 bus 4 >> ppb4 at pci0 dev 21 function 2 "VMware PCIE" rev 0x01 >> pci5 at ppb4 bus 5 >> ppb5 at pci0 dev 21 function 3 "VMware PCIE" rev 0x01 >> pci6 at ppb5 bus 6 >> ppb6 at pci0 dev 21 function 4 "VMware PCIE" rev 0x01 >> pci7 at ppb6 bus 7 >> ppb7 at pci0 dev 21 function 5 "VMware PCIE" rev 0x01 >> pci8 at ppb7 bus 8 >> ppb8 at pci0 dev 21 function 6 "VMware PCIE" rev 0x01 >> pci9 at ppb8 bus 9 >> ppb9 at pci0 dev 21 function 7 "VMware PCIE" rev 0x01 >> pci10 at ppb9 bus 10 >> ppb10 at pci0 dev 22 function 0 "VMware PCIE" rev 0x01 >> pci11 at ppb10 bus 11 >> ppb11 at pci0 dev 22 function 1 "VMware PCIE" rev 0x01 >> pci12 at ppb11 bus 12 >> ppb12 at pci0 dev 22 function 2 "VMware PCIE" rev 0x01 >> pci13 at ppb12 bus 13 >> ppb13 at pci0 dev 22 function 3 "VMware PCIE" rev 0x01 >> pci14 at ppb13 bus 14 >> ppb14 at pci0 dev 22 function 4 "VMware PCIE" rev 0x01 >> pci15 at ppb14 bus 15 >> ppb15 at pci0 dev 22 function 5 "VMware PCIE" rev 0x01 >> pci16 at ppb15 bus 16 >> ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01 >> pci17 at ppb16 bus 17 >> ppb17 at pci0 dev 22 function 7 "VMware PCIE" rev 0x01 >> pci18 at ppb17 bus 18 >> ppb18 at pci0 dev 23 function 0 "VMware PCIE" rev 0x01 >> pci19 at ppb18 bus 19 >> ppb19 at pci0 dev 23 function 1 "VMware PCIE" rev 0x01 >> pci20 at ppb19 bus 20 >> ppb20 at pci0 dev 23 function 2 "VMware PCIE" rev 0x01 >> pci21 at ppb20 bus 21 >> ppb21 at pci0 dev 23 function 3 "VMware PCIE" rev 0x01 >> pci22 at ppb21 bus 22 >> ppb22 at pci0 dev 23 function 4 "VMware PCIE" rev 0x01 >> pci23 at ppb22 bus 23 >> ppb23 at pci0 dev 23 function 5 "VMware PCIE" rev 0x01 >> pci24 at ppb23 bus 24 >> ppb24 at pci0 dev 23 function 6 "VMware PCIE" rev 0x01 >> pci25 at ppb24 bus 25 >> ppb25 at pci0 dev 23 function 7 "VMware PCIE" rev 0x01 >> pci26 at ppb25 bus 26 >> ppb26 at pci0 dev 24 function 0 "VMware PCIE" rev 0x01 >> pci27 at ppb26 bus 27 >> ppb27 at pci0 dev 24 function 1 "VMware PCIE" rev 0x01 >> pci28 at ppb27 bus 28 >> ppb28 at pci0 dev 24 function 2 "VMware PCIE" rev 0x01 >> pci29 at ppb28 bus 29 >> ppb29 at pci0 dev 24 function 3 "VMware PCIE" rev 0x01 >> pci30 at ppb29 bus 30 >> ppb30 at pci0 dev 24 function 4 "VMware PCIE" rev 0x01 >> pci31 at ppb30 bus 31 >> ppb31 at pci0 dev 24 function 5 "VMware PCIE" rev 0x01 >> pci32 at ppb31 bus 32 >> ppb32 at pci0 dev 24 function 6 "VMware PCIE" rev 0x01 >> pci33 at ppb32 bus 33 >> ppb33 at pci0 dev 24 function 7 "VMware PCIE" rev 0x01 >> pci34 at ppb33 bus 34 >> isa0 at piixpcib0 >> isadma0 at isa0 >> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo >> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo >> pckbc0 at isa0 port 0x60/5 >> pckbd0 at pckbc0 (kbd slot) >> pckbc0: using irq 1 for kbd slot >> wskbd0 at pckbd0: console keyboard, using wsdisplay0 >> pms0 at pckbc0 (aux slot) >> pckbc0: using irq 12 for aux slot >> wsmouse0 at pms0 mux 0 >> pcppi0 at isa0 port 0x61 >> spkr0 at pcppi0 >> lpt0 at isa0 port 0x378/4 irq 7 >> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 >> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 >> fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec >> vscsi0 at root >> scsibus3 at vscsi0: 256 targets >> softraid0 at root >> scsibus4 at softraid0: 256 targets >> root on sd0a (1a644d282d120fac.a) swap on sd0b dump on sd0b >> carp211: state transition: BACKUP -> MASTER
