Dear list,
Recently I built a new VPN hub and it seems I reached a limit in ospfd.
The configuration is the following :
2 central OpenBSD (4.7 on production, 4.8 and latest snapshot in our
lab). they both run ospfd on LAN side.
49 OpenBSD clients, running IPSEC + gif encapsulation over to each
central server. Each client is running ospfd too. Everyone is in
area 0.0.0.0.
On the 50th client, the central daemon stop to function normally and
emit a *LOT* of traffic to each client.
The only solution is to kill simultaneously ospfd on each central server
and restart the daemon after the packet storm ended.
I was able to reproduce the problem with 2 servers :
- the first one has a single ospfd daemon for all 50 gif,
- the second one has 50 rdomains and each rdomain contains one gif
and an ospfd daemon
- pf was configured with "pass all"
- no IPSEC
Note also that the problem only occurs if the ospf states are FULL/P2P,
We had to establish each 50 peering in order to reproduce the problem.
Nothing useful can be found on the log files, ("ospfd -vd")
Here is a sample of what is emitted continuously (look at the timestamp
to see how aggressive the flood is) :
17:49:46.220024 10.10.254.140 > 172.16.0.138: 192.168.200.153 > 224.0.0.6:
OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id
58146, len 48) [tos 0xc0] (ttl 64, id 34204, len 68)
17:49:46.220035 10.10.254.140 > 172.16.0.106: 192.168.200.25 > 192.168.200.26:
OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id
9168, len 48) [tos 0xc0] (ttl 64, id 53652, len 68)
17:49:46.220047 10.10.254.140 > 172.16.0.111: 192.168.200.45 > 224.0.0.6:
OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id
34368, len 48) [tos 0xc0] (ttl 64, id 57261, len 68)
17:49:46.220066 10.10.254.140 > 172.16.0.100: 192.168.200.1 > 192.168.200.2:
OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id
22329, len 48) [tos 0xc0] (ttl 64, id 44263, len 68)
17:49:46.220077 10.10.254.140 > 172.16.0.147: 192.168.200.189 >
192.168.200.190: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos
0xc0] [ttl 1] (id 39764, len 48) [tos 0xc0] (ttl 64, id 21228, len 68)
17:49:46.220093 10.10.254.140 > 172.16.0.115: 192.168.200.61 > 224.0.0.6:
OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id
65435, len 48) [tos 0xc0] (ttl 64, id 43562, len 68)
17:49:46.220105 10.10.254.140 > 172.16.0.102: 192.168.200.9 > 224.0.0.6:
OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id
21586, len 48) [tos 0xc0] (ttl 64, id 38683, len 68)
17:49:46.220118 10.10.254.140 > 172.16.0.144: 192.168.200.177 > 224.0.0.6:
OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id
8955, len 48) [tos 0xc0] (ttl 64, id 2926, len 68)
17:49:46.220135 10.10.254.140 > 172.16.0.126: 192.168.200.105 >
192.168.200.106: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos
0xc0] [ttl 1] (id 52430, len 48) [tos 0xc0] (ttl 64, id 27209, len 68)
17:49:46.220146 10.10.254.140 > 172.16.0.134: 192.168.200.137 >
192.168.200.138: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos
0xc0] [ttl 1] (id 18572, len 48) [tos 0xc0] (ttl 64, id 46924, len 68)
17:49:46.220157 10.10.254.140 > 172.16.0.102: 192.168.200.9 > 192.168.200.10:
OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id
26438, len 48) [tos 0xc0] (ttl 64, id 51262, len 68)
17:49:46.220168 10.10.254.140 > 172.16.0.129: 192.168.200.117 >
192.168.200.118: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos
0xc0] [ttl 1] (id 8920, len 48) [tos 0xc0] (ttl 64, id 38270, len 68)
17:49:46.220187 10.10.254.140 > 172.16.0.124: 192.168.200.97 > 192.168.200.98:
OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id
61062, len 48) [tos 0xc0] (ttl 64, id 50506, len 68)
17:49:46.220198 10.10.254.140 > 172.16.0.120: 192.168.200.81 > 192.168.200.82:
OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id
38498, len 48) [tos 0xc0] (ttl 64, id 50045, len 68)
17:49:46.220213 10.10.254.140 > 172.16.0.143: 192.168.200.173 >
192.168.200.174: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos
0xc0] [ttl 1] (id 28513, len 48) [tos 0xc0] (ttl 64, id 45727, len 68)
17:49:46.220226 10.10.254.140 > 172.16.0.117: 192.168.200.69 > 224.0.0.6:
OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id
53678, len 48) [tos 0xc0] (ttl 64, id 49607, len 68)
17:49:46.220237 10.10.254.140 > 172.16.0.133: 192.168.200.133 >
192.168.200.134: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos
0xc0] [ttl 1] (id 30748, len 48) [tos 0xc0] (ttl 64, id 15801, len 68)
Sample master ospfd.conf :
password="XXXXXX"
router-id 10.8.2.53
auth-key $password
auth-type simple
no redistribute 10.8.2.48/30
redistribute 10.0.0.0/8
redistribute 10.10.250.128/25
# areas
area 0.0.0.0 {
interface em1 { metric 1 }
interface gif100
interface gif101
interface gif102
interface gif103
interface gif104
interface gif105
interface gif106
interface gif107
interface gif108
interface gif109
...
interface gif148
interface gif149
}
Sample remote ospfd.conf :
# cat /etc/rdom/ospfd_100.conf
rdomain 100
router-id 172.16.0.100
auth-key XXXXXX
auth-type simple
area 0.0.0.0 {
interface gif100 { metric 40 }
}
Dmesg follows
OpenBSD 4.9-beta (GENERIC) #644: Fri Jan 28 16:09:13 MST 2011
t...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Xeon(R) CPU X3430 @ 2.40GHz ("GenuineIntel" 686-class) 2.40 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,T
M2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT
real mem = 2137194496 (2038MB)
avail mem = 2092097536 (1995MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 10/15/10, BIOS32 rev. 0 @ 0xf9ea0, SMBIOS
rev. 2.6 @ 0x7f79c000 (66 entries)
bios0: vendor Dell Inc. version "1.5.2" date 10/15/2010
bios0: Dell Inc. PowerEdge R310
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC SPCR HPET DM__ MCFG WD__ SLIC ERST HEST BERT EINJ
TCPA SSDT
acpi0: wakeup devices PCI0(S5) USBA(S0) USBB(S0)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 133MHz
cpu at mainbus0: not configured
cpu at mainbus0: not configured
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
acpihpet0 at acpi0: 14318179 Hz
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (LYD0)
acpiprt2 at acpi0: bus 8 (LYD2)
acpiprt3 at acpi0: bus -1 (HVD0)
acpiprt4 at acpi0: bus -1 (HVD2)
acpiprt5 at acpi0: bus 3 (PEX0)
acpiprt6 at acpi0: bus -1 (PEX2)
acpiprt7 at acpi0: bus -1 (PEX3)
acpiprt8 at acpi0: bus 2 (PEX4)
acpiprt9 at acpi0: bus 1 (COMP)
acpicpu0 at acpi0: C3, C1
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 0xc9000/0x2e00 0xec000/0x4000!
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel Core DMI" rev 0x11
ppb0 at pci0 dev 3 function 0 "Intel Core PCIE" rev 0x11: apic 0 int 16 (irq 0)
pci1 at ppb0 bus 4
ppb1 at pci1 dev 0 function 0 "IDT 89HPES12N3A" rev 0x0e
pci2 at ppb1 bus 5
ppb2 at pci2 dev 2 function 0 "IDT 89HPES12N3A" rev 0x0e
pci3 at ppb2 bus 6
em0 at pci3 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int
18 (irq 11), address 00:1b:21:6a:33:28
em1 at pci3 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int
19 (irq 11), address 00:1b:21:6a:33:29
ppb3 at pci2 dev 4 function 0 "IDT 89HPES12N3A" rev 0x0e
pci4 at ppb3 bus 7
em2 at pci4 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int
16 (irq 15), address 00:1b:21:6a:33:2c
em3 at pci4 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int
17 (irq 10), address 00:1b:21:6a:33:2d
ppb4 at pci0 dev 5 function 0 "Intel Core PCIE" rev 0x11: apic 0 int 16 (irq 0)
pci5 at ppb4 bus 8
ppb5 at pci5 dev 0 function 0 "IDT 89HPES12N3A" rev 0x0c
pci6 at ppb5 bus 9
ppb6 at pci6 dev 2 function 0 "IDT 89HPES12N3A" rev 0x0c
pci7 at ppb6 bus 10
em4 at pci7 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int
18 (irq 11), address 00:1b:21:7d:5f:2c
em5 at pci7 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int
19 (irq 11), address 00:1b:21:7d:5f:2d
ppb7 at pci6 dev 4 function 0 "IDT 89HPES12N3A" rev 0x0c
pci8 at ppb7 bus 11
em6 at pci8 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int
16 (irq 15), address 00:1b:21:7d:5f:2e
em7 at pci8 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int
17 (irq 10), address 00:1b:21:7d:5f:2f
"Intel Core Management" rev 0x11 at pci0 dev 8 function 0 not configured
"Intel Core Scratch" rev 0x11 at pci0 dev 8 function 1 not configured
"Intel Core Control" rev 0x11 at pci0 dev 8 function 2 not configured
"Intel Core Misc" rev 0x11 at pci0 dev 8 function 3 not configured
"Intel Core QPI Link" rev 0x11 at pci0 dev 16 function 0 not configured
"Intel Core QPI Routing" rev 0x11 at pci0 dev 16 function 1 not configured
ehci0 at pci0 dev 26 function 0 "Intel 3400 USB" rev 0x05: apic 0 int 22 (irq
14)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb8 at pci0 dev 28 function 0 "Intel 3400 PCIE" rev 0x05
pci9 at ppb8 bus 3
ppb9 at pci0 dev 28 function 4 "Intel 3400 PCIE" rev 0x05
pci10 at ppb9 bus 2
bnx0 at pci10 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic 0 int 16 (irq
15)
bnx1 at pci10 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic 0 int 17 (irq
10)
ehci1 at pci0 dev 29 function 0 "Intel 3400 USB" rev 0x05: apic 0 int 22 (irq
14)
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb10 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa5
pci11 at ppb10 bus 1
vga1 at pci11 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel 3420 LPC" rev 0x05
pciide0 at pci0 dev 31 function 2 "Intel 3400 SATA" rev 0x05: DMA, channel 0
configured to native-PCI, channel 1 configured to native-PCI
pciide0: using apic 0 int 20 (irq 11) for native-PCI interrupt
wd0 at pciide0 channel 0 drive 0: <WDC WD1602ABKS-18N8A0>
wd0: 16-sector PIO, LBA48, 152587MB, 312500000 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6
wd1 at pciide0 channel 1 drive 0: <WDC WD1602ABKS-18N8A0>
wd1: 16-sector PIO, LBA48, 152587MB, 312500000 sectors
wd1(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 6
pciide1 at pci0 dev 31 function 5 "Intel 3400 SATA" rev 0x05: DMA, channel 0
wired to native-PCI, channel 1 wired to native-PCI
pciide1: using apic 0 int 21 (irq 10) for native-PCI interrupt
atapiscsi0 at pciide1 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <TSSTcorp, DVD+-RW TS-L633C, D250> ATAPI 5/cdrom
removable
cd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
kbc: cmd word write error
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
mtrr: Pentium Pro MTRR support
uhub2 at uhub0 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
uhub3 at uhub2 port 1 "Standard Microsystems product 0x2514" rev 2.00/0.00 addr
3
uhidev0 at uhub3 port 1 configuration 1 interface 0 "Dell Dell USB Keyboard"
rev 1.10/1.00 addr 4
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes
wskbd0 at ukbd0 mux 1
wskbd0: connecting to wsdisplay0
uhub4 at uhub1 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
root on wd0a swap on wd0b dump on wd0b
bnx0: address 84:2b:2b:71:65:c4
brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
bnx1: address 84:2b:2b:71:65:c5
brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
