On Wed, Jun 3, 2009 at 11:18 AM, (private) HKS<[email protected]> wrote: > Yet another bizarre state problem that will probably turn out to be > being somehow braindead. > > office -> gw1 -> (INTERNET) -> gw2 -> datacenter > > My office and datacenter routers talk via IPSEC encrypted gif tunnels. > Most everything works. > > From any host on the office network, I can SSH to the internal > interfaces on gw2. I cannot, however, SSH to the external interfaces > (carp or physical). The traffic is routed properly, neatly traverses > the gif tunnel and is accepted by gw2. The reply takes the same path > but is blocked by gw1's default block policy. > > The state is created on gw1 as CLOSED:SYN_SENT: > # pfctl -vvss | grep -A 2 <host> | grep -A 2 <gw2> > all tcp <gw2>:8022 <- <host>:50831 CLOSED:SYN_SENT > [0 + 1] [1095549348 + 2] > age 00:00:02, expires in 00:01:58, 1:0 pkts, 60:0 bytes, rule 24 > > But the replies are rejected: > # tcpdump -eeni pflog0 'host <host>' > tcpdump: listening on pflog0, link-type PFLOG > 10:05:30.836901 rule 0/(match) block in on gif0: <gw2>.8022 > > <host>.50831: R 0:0(0) ack 1095549349 win 0 (DF) > 10:05:34.042631 rule 0/(match) block in on gif0: <gw2>.8022 > > <host>.50831: R 0:0(0) ack 1 win 0 (DF) > 10:05:37.243616 rule 0/(match) block in on gif0: <gw2>.8022 > > <host>.50831: R 0:0(0) ack 1 win 0 (DF) > 10:05:43.452693 rule 0/(match) block in on gif0: <gw2>.8022 > > <host>.50831: R 0:0(0) ack 1 win 0 (DF) > > To address any pf issues, I inserted a "pass quick from <host> to > <gw2>" at the top of my ruleset. Nothing. It works just fine to SSH > from gw1 to gw2's external interface. > > What am I overlooking here? > > dmesg of 4.5 machine follows. > > -HKS > > OpenBSD 4.5 (GENERIC) #1749: Sat Feb 28 14:51:18 MST 2009 > [email protected]:/usr/src/sys/arch/i386/compile/GENERIC > cpu0: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.80 GHz > cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,CX16,xTPR > real mem = 2146795520 (2047MB) > avail mem = 2067582976 (1971MB) > mainbus0 at root > bios0 at mainbus0: AT/286+ BIOS, date 04/25/08, BIOS32 rev. 0 @ > 0xffe90, SMBIOS rev. 2.3 @ 0xf9920 (87 entries) > bios0: vendor Dell Computer Corporation version "A07" date 04/25/2008 > bios0: Dell Computer Corporation PowerEdge 2850 > acpi0 at bios0: rev 0 > acpi0: tables DSDT FACP APIC SPCR HPET MCFG > acpi0: wakeup devices PCI0(S5) PALO(S5) PBLO(S5) VPR0(S5) PBHI(S5) > VPR1(S5) PICH(S5) > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: apic clock running at 199MHz > cpu at mainbus0: not configured > ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins > ioapic0: misconfigured as apic 0, remapped to apid 2 > ioapic1 at mainbus0: apid 3 pa 0xfec80000, version 20, 24 pins > ioapic1: misconfigured as apic 0, remapped to apid 3 > ioapic2 at mainbus0: apid 4 pa 0xfec83000, version 20, 24 pins > ioapic2: misconfigured as apic 0, remapped to apid 4 > ioapic3 at mainbus0: apid 5 pa 0xfec84000, version 20, 24 pins > ioapic3: misconfigured as apic 0, remapped to apid 5 > acpihpet0 at acpi0: 14318179 Hz > acpiprt0 at acpi0: bus 0 (PCI0) > acpiprt1 at acpi0: bus 1 (PALO) > acpiprt2 at acpi0: bus 2 (DOBA) > acpiprt3 at acpi0: bus 3 (DOBB) > acpiprt4 at acpi0: bus 4 (PBLO) > acpiprt5 at acpi0: bus 5 (PBHI) > acpiprt6 at acpi0: bus 6 (PXB1) > acpiprt7 at acpi0: bus 7 (PXB2) > acpiprt8 at acpi0: bus 8 (VPR1) > acpiprt9 at acpi0: bus 9 (PXC1) > acpiprt10 at acpi0: bus 10 (PXC2) > acpiprt11 at acpi0: bus 11 (PICH) > acpicpu0 at acpi0 > bios0: ROM list: 0xc0000/0xb000! 0xcb000/0x1000 0xcc000/0x1000 > 0xcd000/0x2200 0xec000/0x4000! > ipmi at mainbus0 not configured > pci0 at mainbus0 bus 0: configuration mode 1 (bios) > pchb0 at pci0 dev 0 function 0 "Intel E7520 Host" rev 0x09 > ppb0 at pci0 dev 2 function 0 "Intel E7520 PCIE" rev 0x09 > pci1 at ppb0 bus 1 > ppb1 at pci1 dev 0 function 0 "Intel IOP332 PCIE-PCIX" rev 0x06 > pci2 at ppb1 bus 2 > ami0 at pci2 dev 14 function 0 "Dell PERC 4e/Di" rev 0x06: apic 3 int 14 (irq 7) > ami0: Dell 16d, 32b, FW 513O, BIOS vH418, 256MB RAM > ami0: 2 channels, 0 FC loops, 1 logical drives > scsibus0 at ami0: 40 targets > sd0 at scsibus0 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed > sd0: 139900MB, 512 bytes/sec, 286515200 sec total > scsibus1 at ami0: 16 targets > safte0 at scsibus1 targ 6 lun 0: <PE/PV, 1x6 SCSI BP, 1.0> SCSI2 > 3/processor fixed > scsibus2 at ami0: 16 targets > ppb2 at pci1 dev 0 function 2 "Intel IOP332 PCIE-PCIX" rev 0x06 > pci3 at ppb2 bus 3 > ppb3 at pci0 dev 4 function 0 "Intel E7520 PCIE" rev 0x09 > pci4 at ppb3 bus 4 > ppb4 at pci0 dev 5 function 0 "Intel E7520 PCIE" rev 0x09 > pci5 at ppb4 bus 5 > ppb5 at pci5 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09 > pci6 at ppb5 bus 6 > em0 at pci6 dev 7 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05: > apic 4 int 0 (irq 11), address 00:11:43:d9:17:36 > ppb6 at pci5 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09 > pci7 at ppb6 bus 7 > em1 at pci7 dev 8 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05: > apic 4 int 1 (irq 3), address 00:11:43:d9:17:37 > ppb7 at pci0 dev 6 function 0 "Intel E7520 PCIE" rev 0x09 > pci8 at ppb7 bus 8 > ppb8 at pci8 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09 > pci9 at ppb8 bus 9 > ppb9 at pci8 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09 > pci10 at ppb9 bus 10 > em2 at pci10 dev 2 function 0 "Intel PRO/1000MT (82546GB)" rev 0x03: > apic 5 int 0 (irq 11), address 00:04:23:ad:04:04 > em3 at pci10 dev 2 function 1 "Intel PRO/1000MT (82546GB)" rev 0x03: > apic 5 int 1 (irq 3), address 00:04:23:ad:04:05 > uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: apic > 2 int 16 (irq 11) > uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: apic > 2 int 19 (irq 10) > uhci2 at pci0 dev 29 function 2 "Intel 82801EB/ER USB" rev 0x02: apic > 2 int 18 (irq 7) > ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB2" rev 0x02: apic > 2 int 23 (irq 5) > usb0 at ehci0: USB revision 2.0 > uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 > ppb10 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xc2 > pci11 at ppb10 bus 11 > vga1 at pci11 dev 13 function 0 "ATI Radeon VE" rev 0x00 > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) > wsdisplay0: screen 1-5 added (80x25, vt100 emulation) > radeondrm0 at vga1: apic 2 int 18 (irq 7) > drm0 at radeondrm0 > ichpcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02 > pciide0 at pci0 dev 31 function 1 "Intel 82801EB/ER IDE" rev 0x02: > DMA, channel 0 configured to compatibility, channel 1 configured to > compatibility > atapiscsi0 at pciide0 channel 0 drive 0 > scsibus3 at atapiscsi0: 2 targets > cd0 at scsibus3 targ 0 lun 0: <HL-DT-ST, DVD-ROM GDR8082N, 0106> ATAPI > 5/cdrom removable > cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 > pciide0: channel 1 disabled (no drives) > usb1 at uhci0: USB revision 1.0 > uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > usb2 at uhci1: USB revision 1.0 > uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > usb3 at uhci2: USB revision 1.0 > uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > isa0 at ichpcib0 > isadma0 at isa0 > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > pckbc0 at isa0 port 0x60/5 > pckbd0 at pckbc0 (kbd slot) > pckbc0: using irq 1 for kbd slot > wskbd0 at pckbd0: console keyboard, using wsdisplay0 > pcppi0 at isa0 port 0x61 > midi0 at pcppi0: <PC speaker> > spkr0 at pcppi0 > npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 > fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 > mtrr: Pentium Pro MTRR support > uhub4 at uhub0 port 3 "Dell product 0xa001" rev 2.00/0.00 addr 2 > uhidev0 at uhub4 port 1 configuration 1 interface 0 "Dell Dell USB > Keyboard" rev 1.10/3.01 addr 3 > uhidev0: iclass 3/1 > ukbd0 at uhidev0: 8 modifier keys, 6 key codes > wskbd1 at ukbd0 mux 1 > wskbd1: connecting to wsdisplay0 > softraid0 at root > root on sd0a swap on sd0b dump on sd0b >
Naturally, the moment I hit send I start putting pieces together in ways that make a bit more sense. It does seem to implicate the same bug I alluded to in my other recent email (IPSec and state). First, further tcpdumps found that the initial SYN was not being sent through the gif interface. This should have been obvious since the packet is destined for the same public IP that is the other end of the gif interface. Second, because of my IPSEC policy the initial SYN was still getting encrypted. This preempts NAT, so the encrypted packet still had the private source IP of my office host. gw2 got the IPSec packet, unwrapped it and saw that the source address was on my office network. So the reply was routed through the gif tunnel. So the routing mismatch led to a similar set of circumstances as my previous email. In my understanding, the floating-interface property of states should make this work, but either a) I'm wrong for some reason, b) my configuration is somehow fucking this up, or c) there's a bug in how asymmetric gif traffic with symmetric payloads are handled. Any insight is greatly appreciated. -HKS
