Hi misc!
configuration is following:
Server
|
Router0
||
||ipsec tunnel
||
Router1
|
Client
ipsec tunnel between routers is using following parameters:
ike esp transport proto ipencap from R0_IP to R1_IP \
local R0_IP peer R1_IP quick auth hmac-sha1 enc 3des \
srcid R0_NAME dstid R1_NAME
Routers are directly connected(no other devices between them)
Both routers are running OpenBSD 4.7 amd64 MP kernel with
stable patches.
Both routers have a crypto accelerator:
ubsec0 at pci2 dev 1 function 0 "Sun Crypto 5821" rev 0x01:
3DES MD5 SHA1 RNG PK, apic 9 int 5 (irq 10)
Now the problem.
When using iperf, everything works perfectly(tried UDP and TCP
at various packet sizes). Same goes with ping.
The only way(so far), i've been able to reproduce the problem,
is using either mysql client to connect to mysql server or
doing telnet to the mysql servers 3306 port.
The connection fails on both cases.
Tcpdump in the server side routers ipsec tunnel gif shows
the following:
10:05:35.320881 192.168.8.46.45873 > 192.168.7.7.3306: S
3861220531:3861220531(0) win 5840 <mss 1460,sackOK,timestamp
2517252528 0,nop,wscale 6> [tos 0x10]
10:05:35.321063 192.168.7.7.3306 > 192.168.8.46.45873: S
25184535:25184535(0) ack 3861220532 win 5792 <mss 1460,sackOK,
timestamp 4053315240 2517252528,nop,wscale 7>
10:05:35.321951 192.168.8.46.45873 > 192.168.7.7.3306: . ack 1
win 92 <nop,nop,timestamp 2517252528 4053315240> [tos 0x10]
10:05:35.322402 192.168.7.7.3306 > 192.168.8.46.45873: P 1:75(74)
ack 1 win 46 <nop,nop,timestamp 4053315241 2517252528> [tos 0x8]
10:05:35.530663 192.168.7.7.3306 > 192.168.8.46.45873: P 1:75(74)
ack 1 win 46 <nop,nop,timestamp 4053315293 2517252528> [tos 0x8]
10:05:35.937570 192.168.7.7.3306 > 192.168.8.46.45873: P 1:75(74)
ack 1 win 46 <nop,nop,timestamp 4053315395 2517252528> [tos 0x8]
10:05:36.753450 192.168.7.7.3306 > 192.168.8.46.45873: P 1:75(74)
ack 1 win 46 <nop,nop,timestamp 4053315599 2517252528> [tos 0x8]
10:05:38.385517 192.168.7.7.3306 > 192.168.8.46.45873: P 1:75(74)
ack 1 win 46 <nop,nop,timestamp 4053316007 2517252528> [tos 0x8]
10:05:41.649605 192.168.7.7.3306 > 192.168.8.46.45873: P 1:75(74)
ack 1 win 46 <nop,nop,timestamp 4053316823 2517252528> [tos 0x8]
On the client side routers gif interface i can see only the first
three packets. "netstat -ssp esp" shows that, everytime i try this,
counter "packets that failed verification received" increases only
on the client side router.
The problem is symmetrical - if i reverse the direction, same things
happen, but all the beforementioned events have swithced places and
are on the other router now.
When using aes instead of 3des, the problem does not occur. My guess
is that it is so due to the lack of aes support of the crypto
accelerator. When crypo accelerator is not present in the system,
i haven't been able to reproduce the problem.
I have seen the same problem also in 4.8(amd64) and 4.9 current(amd64)
from 08.02.11. Using SP kernel doesn't solve the problem.
In the test environment i couldn't reproduce the problem with 4.7 i386
(with mysql client), but i saw the same symptoms when i tried it in
the live environment. Perhaps the problem didn't occur due to low
system load in the lab setup.
I have also tested it with IPSec tunnel mode. Problem doesn't occur
with mysql client, but in live environment some packets still fail
verification and there are problems with some services. It's nature
seems to me similar to some MTU problems but packet sizes do not
confirm that and there is no problem with aes. For that reason,
i believe, that we can exclude PF as the source of the problem.
Routers hardware is almost identical(processor speed and the amount of
memory are different). Both are running on HP Proliant 365 G1.
Dmesg of the server side router is following:
OpenBSD 4.7 (GENERIC.MP) #1: Mon Jan 17 16:12:03 EET 2011
root@router.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2144657408 (2045MB)
avail mem = 2078121984 (1981MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xee000 (64 entries)
bios0: vendor HP version "A10" date 03/27/2008
bios0: HP ProLiant DL365 G1
acpi0 at bios0: rev 2
acpi0: tables DSDT FACP SPCR MCFG HPET SPMI ERST APIC SRAT FFFF BERT HEST
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpihpet0 at acpi0: 14318180 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Dual-Core AMD Opteron(tm) Processor 2222, 3000.59 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT
,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,NXE,MMXX,FFXSR,LONG,3DNOW2,
3DNOW
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully
associative
cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully
associative
cpu0: apic clock running at 200MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Dual-Core AMD Opteron(tm) Processor 2222, 3000.11 MHz
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,
PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,NXE,MMXX,FFXSR,LONG,
3DNOW2,3DNOW
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache,
1MB 64b/line 16-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully
associative
cpu1: DTLB 32 4KB entries fully associative, 8 4MB entries fully
associative
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 11, 16 pins
ioapic1 at mainbus0: apid 9 pa 0xfec01000, version 11, 16 pins
acpiprt0 at acpi0: bus 2 (PPXB)
acpiprt1 at acpi0: bus 0 (PCI0)
acpiprt2 at acpi0: bus 76 (EXB0)
acpiprt3 at acpi0: bus 72 (EXB1)
acpiprt4 at acpi0: bus 66 (NB01)
acpiprt5 at acpi0: bus 68 (NB02)
acpiprt6 at acpi0: bus 69 (EXB4)
acpiprt7 at acpi0: bus 64 (PCI1)
acpicpu0 at acpi0: PSS
acpicpu1 at acpi0: PSS
ipmi at mainbus0 not configured
cpu0: PowerNow! K8 3000 MHz: speeds: 3000 2800 2600 2400 2200
2000 1800 1000 MHz
pci0 at mainbus0 bus 0
vga1 at pci0 dev 3 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
radeondrm0 at vga1: apic 9 int 11 (irq 10)
drm0 at radeondrm0
"Compaq iLO" rev 0x03 at pci0 dev 4 function 0 not configured
"Compaq iLO" rev 0x03 at pci0 dev 4 function 2 not configured
uhci0 at pci0 dev 4 function 4 "Hewlett-Packard USB" rev 0x00: apic
9 int 10 (irq 11)
"Hewlett-Packard IPMI" rev 0x00 at pci0 dev 4 function 6 not configured
ppb0 at pci0 dev 5 function 0 "ServerWorks HT-1000 PCI" rev 0x00
pci1 at ppb0 bus 1
ppb1 at pci1 dev 13 function 0 "ServerWorks HT-1000 PCIX" rev 0xc0
pci2 at ppb1 bus 2
ubsec0 at pci2 dev 1 function 0 "Sun Crypto 5821" rev 0x01: 3DES MD5
SHA1 RNG PK, apic 9 int 5 (irq 10)
piixpm0 at pci0 dev 6 function 0 "ServerWorks HT-1000" rev 0x00: polling
iic0 at piixpm0
pciide0 at pci0 dev 6 function 1 "ServerWorks HT-1000 IDE" rev 0x00: DMA
pcib0 at pci0 dev 6 function 2 "ServerWorks HT-1000 LPC" rev 0x00
ohci0 at pci0 dev 7 function 0 "ServerWorks HT-1000 USB" rev 0x01: apic
8 int 5 (irq 5), version 1.0, legacy support
ohci1 at pci0 dev 7 function 1 "ServerWorks HT-1000 USB" rev 0x01: apic
8 int 5 (irq 5), version 1.0, legacy support
ehci0 at pci0 dev 7 function 2 "ServerWorks HT-1000 USB" rev 0x01: apic
8 int 5 (irq 5)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "ServerWorks EHCI root hub" rev 2.00/1.00 addr 1
pchb0 at pci0 dev 24 function 0 "AMD AMD64 0Fh HyperTransport" rev 0x00
pci3 at pchb0 bus 64
ppb2 at pci3 dev 15 function 0 "ServerWorks HT-2100 PCIE" rev 0xa2
pci4 at ppb2 bus 76
ppb3 at pci3 dev 16 function 0 "ServerWorks HT-2100 PCIE" rev 0xa2
pci5 at ppb3 bus 72
ppb4 at pci5 dev 0 function 0 "IDT 89HPES12N3A" rev 0x0e
pci6 at ppb4 bus 73
ppb5 at pci6 dev 2 function 0 "IDT 89HPES12N3A" rev 0x0e
pci7 at ppb5 bus 74
em0 at pci7 dev 0 function 0 "Intel PRO/1000 QP (82571EB)" rev 0x06:
apic 9 int 12 (irq 11), address 00:1f:29:5e:76:79
em1 at pci7 dev 0 function 1 "Intel PRO/1000 QP (82571EB)" rev 0x06:
apic 9 int 1 (irq 11), address 00:1f:29:5e:76:78
ppb6 at pci6 dev 4 function 0 "IDT 89HPES12N3A" rev 0x0e
pci8 at ppb6 bus 75
em2 at pci8 dev 0 function 0 "Intel PRO/1000 QP (82571EB)" rev 0x06:
apic 9 int 0 (irq 11), address 00:1f:29:5e:76:7b
em3 at pci8 dev 0 function 1 "Intel PRO/1000 QP (82571EB)" rev 0x06:
apic 9 int 4 (irq 11), address 00:1f:29:5e:76:7a
ppb7 at pci3 dev 17 function 0 "ServerWorks HT-2100 PCIE" rev 0xa2
pci9 at ppb7 bus 65
ppb8 at pci9 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
pci10 at ppb8 bus 66
bnx0 at pci10 dev 0 function 0 "Broadcom BCM5708" rev 0x12: apic 9
int 3 (irq 11)
ppb9 at pci3 dev 18 function 0 "ServerWorks HT-2100 PCIE" rev 0xa2
pci11 at ppb9 bus 67
ppb10 at pci11 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
pci12 at ppb10 bus 68
bnx1 at pci12 dev 0 function 0 "Broadcom BCM5708" rev 0x12: apic 9
int 12 (irq 11)
ppb11 at pci3 dev 19 function 0 "ServerWorks HT-2100 PCIE" rev 0xa2
pci13 at ppb11 bus 69
ciss0 at pci13 dev 0 function 0 "Hewlett-Packard Smart Array" rev
0x01: apic 9 int 1 (irq 11)
ciss0: 1 LD, HW rev 1, FW 6.86/6.86, 64bit fifo
scsibus0 at ciss0: 1 targets
sd0 at scsibus0 targ 0 lun 0: <HP, LOGICAL VOLUME, 6.86> SCSI3
0/direct fixed
sd0: 69973MB, 512 bytes/sec, 143305920 sec total
pchb1 at pci0 dev 24 function 1 "AMD AMD64 0Fh Address Map" rev 0x00
pchb2 at pci0 dev 24 function 2 "AMD AMD64 0Fh DRAM Cfg" rev 0x00
kate0 at pci0 dev 24 function 3 "AMD AMD64 0Fh Misc Cfg" rev 0x00:
core rev JH-F3
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Hewlett-Packard UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com1: console
pckbc0 at isa0 port 0x60/5
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
usb2 at ohci0: USB revision 1.0
uhub2 at usb2 "ServerWorks OHCI root hub" rev 1.00/1.00 addr 1
usb3 at ohci1: USB revision 1.0
uhub3 at usb3 "ServerWorks OHCI root hub" rev 1.00/1.00 addr 1
mtrr: Pentium Pro MTRR support
uhub4 at uhub3 port 1 "Cypress Semiconductor USB2 Hub" rev 2.00/0.07
addr 2
uhidev0 at uhub1 port 1 configuration 1 interface 0 "HP Virtual
Keyboard" rev 1.10/0.02 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes, country code 33
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub1 port 1 configuration 1 interface 1 "HP Virtual
Keyboard" rev 1.10/0.02 addr 2
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons
wsmouse0 at ums0 mux 0
uhub5 at uhub1 port 2 "HP Virtual Hub" rev 1.10/0.01 addr 3
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
root on sd0a swap on sd0b dump on sd0b
bnx0: address 00:1b:78:ce:da:22
brgphy0 at bnx0 phy 1: BCM5708C 10/100/1000baseT PHY, rev. 6
bnx1: address 00:1b:78:ce:da:0c
brgphy1 at bnx1 phy 1: BCM5708C 10/100/1000baseT PHY, rev. 6
I have an understanding, that there is no configuration changes
needed when using crypto accelerators on OpenBSD.
For now, i have run out of ideas what to try, in order to
get it working. Am i doing something wrong?
I would be very grateful, for any hints and
suggestions for further debuging.
All the best,
Joosep
