Hi!
I have following problem with my OpenBSD amd64 version firewall and
would be very thankful if you can help me with it.
Quite accidentally my collegue discovered that while he is accessing
content over http from behind natting firewall he doest get it every
time. And it happens seemengly randomly, say about ten times per 300
attempts (vise versa firewall is working all right and also with
routing). I tested it on living firewall and confirmed it and after that
i set up other computers dedicated to test this case more throughly.
This is my test setup
http server ---- em1 firewall bge0 --- mgm computer
server 10.0.5.2 --> 192.168.2.38
172.16.0.12 | em0
|
|
|
computer accessing http server (10.0.6.242)
firewall has following addresses
em0 - 10.0.6.248
em1 - 172.16.0.78
bge0 - 10.0.5.7
mgm computer actually is 192.168.2.38, a hop away.
I used 4.4 amd64 system with latest kernel patches (and userspace
patches between them) but i also tried original 4.4 kernel, results seem
to be the same. dmesg and full pfctl -sa are included in the end of this
letter.
rules on the firewall are no more no less like this
# pfctl -sn
nat on em1 inet all tagged ICMP_TEST -> 172.16.0.78
# pfctl -sr
block drop log all
pass in quick on bge0 inet from 192.168.2.0/24 to 10.0.5.7 flags S/SA
keep state (tcp.established 1064000)
pass in quick on bge0 inet from 10.0.5.0/24 to 10.0.5.7 flags S/SA keep
state (tcp.established 1064000)
pass in quick on em0 inet proto tcp from 10.0.6.242 to 172.16.0.12 port
= www flags S/SA keep state tag ICMP_TEST
pass out quick on em1 all flags S/SA keep state tagged ICMP_TEST
Here is my testing.
I access http in this manner (after fresh reboot)
$ for i in `seq 1 300`; do wget "http://172.16.0.12/README?count=$i"; -O
- 1>dhs.$i.log; done
and the results are like this, i.e. this time five responses are not
succeeding
$ find . -size 0
./dhs.251.log
./dhs.171.log
./dhs.179.log
./dhs.188.log
./dhs.149.log
while listening on firewall on em0 for icmp i get
# tcpdump -nettti em0 icmp
Jan 22 21:06:45.787661 00:04:23:09:14:30 70:10:00:00:62:42 0800 70:
10.0.6.248 > 10.0.6.242: icmp: host 172.16.0.12 unreachable
Jan 22 21:06:45.995783 00:04:23:09:14:30 70:10:00:00:62:42 0800 70:
10.0.6.248 > 10.0.6.242: icmp: host 172.16.0.12 unreachable
Jan 22 21:06:46.067863 00:04:23:09:14:30 70:10:00:00:62:42 0800 70:
10.0.6.248 > 10.0.6.242: icmp: host 172.16.0.12 unreachable
Jan 22 21:06:46.150686 00:04:23:09:14:30 70:10:00:00:62:42 0800 70:
10.0.6.248 > 10.0.6.242: icmp: host 172.16.0.12 unreachable
Jan 22 21:06:46.765440 00:04:23:09:14:30 70:10:00:00:62:42 0800 70:
10.0.6.248 > 10.0.6.242: icmp: host 172.16.0.12 unreachable
It may also be essential to say that there does not appear anything
relevant (on this network there are other traffic as well to be honest)
in pflog.
I also saved all traffic on both relevant firewall interfaces during
test and followed it and tcpdump shows that during connection failure
1. http client sends syn packet which do not get to the other side of
firewall
2. firewall answers this with icmp host unreachable message
3. wget saves zero result
4. client then sends out next syn which gets properly served
In the end i tested removing nat and it worked well i.e. without errors
(16k of queries).
I also tested the same thing with OpenBSD 4.3 and i did work for 24k
queries all right (didnt try longer).
If someone could please confirm whether this holds true generally on
amd64 and i386 (havent tried it yet) platform or it still is some kind
of specific combination of my computer and networking hardware, skills
and luck.
Best regards
Imre
OpenBSD 4.4 (GENERIC) #1562: Tue Aug 12 17:15:53 MDT 2008
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 1060478976 (1011MB)
avail mem = 1029427200 (981MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.3 @ 0xec000 (73 entries)
bios0: vendor HP version "P54" date 02/14/2006
bios0: HP ProLiant DL360 G4p
acpi0 at bios0: rev 2
acpi0: tables DSDT FACP SPCR MCFG APIC SSDT
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpiprt0 at acpi0: bus 1 (IP2P)
acpiprt1 at acpi0: bus 2 (ICHR)
acpiprt2 at acpi0: bus 7 (PCXA)
acpiprt3 at acpi0: bus 10 (PCXB)
acpiprt4 at acpi0: bus 6 (PTB0)
acpiprt5 at acpi0: bus 13 (PTA0)
acpiprt6 at acpi0: bus 3 (PTC0)
acpiprt7 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0
acpitz0 at acpi0: critical temperature 31 degC
cpu0 at mainbus0: (uniprocessor)
cpu0: Intel(R) Xeon(TM) CPU 3.60GHz, 3600.60 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,EST,T
M2,CNXT-ID,CX16,xTPR,LONG
cpu0: 2MB 64b/line 8-way L2 cache
pci0 at mainbus0 bus 0: configuration mode 1
pchb0 at pci0 dev 0 function 0 "Intel E7520 Host" rev 0x0c
ppb0 at pci0 dev 2 function 0 "Intel E7520 PCIE" rev 0x0c
pci1 at ppb0 bus 13
ppb1 at pci0 dev 4 function 0 "Intel E7520 PCIE" rev 0x0c
pci2 at ppb1 bus 6
ppb2 at pci2 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci3 at ppb2 bus 7
ppb3 at pci2 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
pci4 at ppb3 bus 10
ppb4 at pci4 dev 1 function 0 "IBM 133 PCIX-PCIX" rev 0x02
pci5 at ppb4 bus 11
em0 at pci5 dev 4 function 0 "Intel PRO/1000MT QP (82546EB)" rev 0x01:
irq 5, address 00:04:23:09:14:30
em1 at pci5 dev 4 function 1 "Intel PRO/1000MT QP (82546EB)" rev 0x01:
irq 5, address 00:04:23:09:14:31
em2 at pci5 dev 6 function 0 "Intel PRO/1000MT QP (82546EB)" rev 0x01:
irq 7, address 00:04:23:09:14:32
em3 at pci5 dev 6 function 1 "Intel PRO/1000MT QP (82546EB)" rev 0x01:
irq 5, address 00:04:23:09:14:33
ppb5 at pci0 dev 6 function 0 "Intel E7520 PCIE" rev 0x0c
pci6 at ppb5 bus 3
ppb6 at pci0 dev 28 function 0 "Intel 6300ESB PCIX" rev 0x02
pci7 at ppb6 bus 2
ciss0 at pci7 dev 1 function 0 "Compaq Smart Array 64xx" rev 0x01: irq 5
ciss0: 1 LD, HW rev 1, FW 2.58/2.58
scsibus0 at ciss0: 1 targets, initiator 1
sd0 at scsibus0 targ 0 lun 0: <HP, LOGICAL VOLUME, 2.58> SCSI0 0/direct
fixed
sd0: 69459MB, 8854 cyl, 255 head, 63 sec, 512 bytes/sec, 142253280 sec total
bge0 at pci7 dev 2 function 0 "Broadcom BCM5704C" rev 0x10, BCM5704 B0
(0x2100): irq 5, address 00:17:a4:3b:ab:0c
brgphy0 at bge0 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
bge1 at pci7 dev 2 function 1 "Broadcom BCM5704C" rev 0x10, BCM5704 B0
(0x2100): irq 5, address 00:17:a4:3b:ab:0b
brgphy1 at bge1 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
uhci0 at pci0 dev 29 function 0 "Intel 6300ESB USB" rev 0x02: irq 5
uhci1 at pci0 dev 29 function 1 "Intel 6300ESB USB" rev 0x02: irq 5
"Intel 6300ESB WDT" rev 0x02 at pci0 dev 29 function 4 not configured
"Intel 6300ESB APIC" rev 0x02 at pci0 dev 29 function 5 not configured
ehci0 at pci0 dev 29 function 7 "Intel 6300ESB USB" rev 0x02: irq 5
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb7 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x0a
pci8 at ppb7 bus 1
vga1 at pci8 dev 3 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
drm at vga1 unsupported
"Compaq iLO" rev 0x01 at pci8 dev 4 function 0 not configured
"Compaq iLO" rev 0x01 at pci8 dev 4 function 2 not configured
pcib0 at pci0 dev 31 function 0 "Intel 6300ESB LPC" rev 0x02
pciide0 at pci0 dev 31 function 1 "Intel 6300ESB IDE" rev 0x02: DMA,
channel 0 configured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets, initiator 7
cd0 at scsibus1 targ 0 lun 0: <HL-DT-ST, DVD-ROM GDR8084N, 3.00> ATAPI
5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com1: probed fifo depth: 0 bytes
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
mtrr: Pentium Pro MTRR support
softraid0 at root
root on sd0d swap on sd0b dump on sd0b
pfctl -sa when TIME_WAIT_2 are settled out
TRANSLATION RULES:
nat on em1 inet all tagged ICMP_TEST -> 172.16.0.78
FILTER RULES:
block drop log all
pass in quick on bge0 inet from 192.168.2.0/24 to 10.0.5.7 flags S/SA
keep state (tcp.established 1064000)
pass in quick on bge0 inet from 10.0.5.0/24 to 10.0.5.7 flags S/SA keep
state (tcp.established 1064000)
pass in quick on em0 inet proto tcp from 10.0.6.242 to 172.16.0.12 port
= www flags S/SA keep state tag ICMP_TEST
pass out quick on em1 all flags S/SA keep state tagged ICMP_TEST
No queue in use
STATES:
all tcp 10.0.5.7:22 <- 192.168.2.38:44896 ESTABLISHED:ESTABLISHED
all tcp 10.0.5.7:22 <- 192.168.2.38:47357 ESTABLISHED:ESTABLISHED
all tcp 10.0.5.7:22 <- 192.168.2.38:52714 ESTABLISHED:ESTABLISHED
INFO:
Status: Enabled for 0 days 00:57:33 Debug: Urgent
State Table Total Rate
current entries 8
searches 299539 86.7/s
inserts 39390 11.4/s
removals 39382 11.4/s
Counters
match 43300 12.5/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 0 0.0/s
state-insert 169 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
TIMEOUTS:
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90s
tcp.tsdiff 30s
udp.first 60s
udp.single 30s
udp.multiple 60s
icmp.first 20s
icmp.error 10s
other.first 60s
other.single 30s
other.multiple 60s
frag 30s
interval 10s
adaptive.start 40000 states
adaptive.end 50000 states
src.track 0s
LIMITS:
states hard limit 500000
src-nodes hard limit 10000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 200000
OS FINGERPRINTS:
696 fingerprints loaded
# sysctl net.inet.ip
net.inet.ip.forwarding=1
net.inet.ip.redirect=1
net.inet.ip.ttl=64
net.inet.ip.sourceroute=0
net.inet.ip.directed-broadcast=0
net.inet.ip.portfirst=1024
net.inet.ip.portlast=49151
net.inet.ip.porthifirst=49152
net.inet.ip.porthilast=65535
net.inet.ip.maxqueue=300
net.inet.ip.encdebug=0
net.inet.ip.ipsec-expire-acquire=30
net.inet.ip.ipsec-invalid-life=60
net.inet.ip.ipsec-pfs=1
net.inet.ip.ipsec-soft-allocs=0
net.inet.ip.ipsec-allocs=0
net.inet.ip.ipsec-soft-bytes=0
net.inet.ip.ipsec-bytes=0
net.inet.ip.ipsec-timeout=86400
net.inet.ip.ipsec-soft-timeout=80000
net.inet.ip.ipsec-soft-firstuse=3600
net.inet.ip.ipsec-firstuse=7200
net.inet.ip.ipsec-enc-alg=aes
net.inet.ip.ipsec-auth-alg=hmac-sha1
net.inet.ip.mtudisc=1
net.inet.ip.mtudisctimeout=600
net.inet.ip.ipsec-comp-alg=deflate
net.inet.ip.ifq.len=0
net.inet.ip.ifq.maxlen=256
net.inet.ip.ifq.drops=0
net.inet.ip.mforwarding=0
net.inet.ip.multipath=0
net.inet.ip.mrtproto=19
