Hi Michał,
I'm having same issue without 100 ipsec tunnels and dedicated hardware.
Unfortunately it's a production environment so I can't really
troubleshooting this issue to track down the culprit.
Anyway maybe it's not related to your issue.
Regards,
Alexis.
On 28/05/2017 14:31, Michał Koc wrote:
Hi all,
I'm running 6.0/amd64 inside KVM/Quemu with over 100 ipsec tunnels.
Everything was running just fine when the number of tunnels was lower.
But as we have been setting up more and more tunnels we suddenly run
on problems.
The isakmpd deaemon keeps dying quietly. Probably I'm running out of
something, but I need some help to find out what it is and how to
monitor it and tweak.
Thank You in advance.
Best Regards
M.K.
root@vgate0:/root# netstat -m
215 mbufs in use:
163 mbufs allocated to data
46 mbufs allocated to packet headers
6 mbufs allocated to socket names and addresses
160/920/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/14/6146 mbuf 9216 byte clusters in use (current/peak/max)
0/10/6150 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
2760 Kbytes allocated to network (13% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Sample tail of the log:
When I run "isakmpd -K -d -DA=10":
142043.246192 Sdep 10 pf_key_v2_set_spi: satype 2 dst xxx.xxx.xxx.xxx
SPI 0x42f03e5d
142043.246209 Timr 10 timer_add_event: event
sa_soft_expire(0x1fb9d0bdf400) added before
sa_soft_expire(0x1fb9c8f05400), expiration in 25056s
142043.246223 Timr 10 timer_add_event: event
sa_hard_expire(0x1fb9d0bdf400) added before
sa_soft_expire(0x1fb9dd458200), expiration in 28800s
142043.246326 Sdep 10 pf_key_v2_set_spi: satype 2 dst xxx.xxx.xxx.xxx
SPI 0x3ffa5955
142043.268229 Default responder_recv_HASH_SA_NONCE: KEY_EXCH payload
without a group desc. attribute
142043.268250 Default dropped message from xxx.xxx.xxx.xxx port 500
due to notification type NO_PROPOSAL_CHOSEN
142043.268281 Timr 10 timer_add_event: event
exchange_free_aux(0x1fb9a5336400) added before
sa_soft_expire(0x1fba0d6a2a00), expiration in 120s
142043.268289 Exch 10 exchange_establish_p2: 0x1fb9a5336400 <unnamed>
<no policy> policy initiator phase 2 doi 1 exchange 5 step 0
142043.268295 Exch 10 exchange_establish_p2: icookie 8c58f4e7f8269ed3
rcookie 0fe2d7657125a339
142043.268301 Exch 10 exchange_establish_p2: msgid de2c5cc3 sa_list
142043.269079 Timr 10 timer_add_event: event
message_send_expire(0x1fb994136900) added before
connection_checker(0x1fb9b2646280), expiration in 7s
142043.269614 Exch 10 exchange_finalize: 0x1fb9a5336400 <unnamed> <no
policy> policy initiator phase 2 doi 1 exchange 5 step 1
142043.269630 Exch 10 exchange_finalize: icookie 8c58f4e7f8269ed3
rcookie 0fe2d7657125a339
142043.269637 Exch 10 exchange_finalize: msgid de2c5cc3 sa_list
142043.269653 Timr 10 timer_remove_event: removing event
exchange_free_aux(0x1fb9a5336400)
142043.289465 Timr 10 timer_remove_event: removing event
message_send_expire(0x1fb994136900)
142043.289513 Exch 10 exchange_finalize: 0x1fb972b59400
from-xxx.xxx.xxx.xxx/24-to-xxx.xxx.xxx.xxx/24 <no policy> policy
responder phase 2 doi 1 exchange 32 step 2
142043.289521 Exch 10 exchange_finalize: icookie 8c58f4e7f8269ed3
rcookie 0fe2d7657125a339
142043.289528 Exch 10 exchange_finalize: msgid de079ef6 sa_list
0x1fb9dd458800 0x1fb985d09e00
142043.289578 Sdep 10 pf_key_v2_set_spi: satype 2 dst xxx.xxx.xxx.xxx
SPI 0xe5d04953
142043.289594 Timr 10 timer_add_event: event
sa_soft_expire(0x1fb9dd458800) added before
sa_soft_expire(0x1fba1d81de00), expiration in 3279s
142043.289608 Timr 10 timer_add_event: event
sa_hard_expire(0x1fb9dd458800) added before
sa_soft_expire(0x1fba2c980800), expiration in 3600s
142043.289710 Sdep 10 pf_key_v2_set_spi: satype 2 dst xxx.xxx.xxx.xxx
SPI 0x4d895568
root@vgate0:/root#
OpenBSD 6.0-stable (GENERIC.MP) #0: Sat Feb 4 21:55:17 CET 2017
root@amd64.vcomp:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 1056956416 (1007MB)
avail mem = 1020506112 (973MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf1dc0 (11 entries)
bios0: vendor Bochs version "Bochs" date 01/01/2011
bios0: Bochs Bochs
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: QEMU Virtual CPU version 2.1.2, 3492.32 MHz
cpu0:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,HV,NXE,LONG,LAHF,ABM
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: QEMU Virtual CPU version 2.1.2, 3491.95 MHz
cpu1:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,HV,NXE,LONG,LAHF,ABM
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: smt 0, core 0, package 1
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
"PNP0303" at acpi0 not configured
"PNP0F13" at acpi0 not configured
"PNP0700" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
pvbus0 at mainbus0: KVM
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0
int 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic
0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 00:50:56:00:0e:94
virtio0: msix shared
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Network" rev 0x00
vio1 at virtio1: address 52:54:00:05:43:8a
virtio1: msix shared
virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio2
scsibus1 at vioblk0: 2 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct
fixed
sd0: 4096MB, 512 bytes/sector, 8388608 sectors
virtio2: msix shared
virtio3 at pci0 dev 6 function 0 "Qumranet Virtio Memory" rev 0x00
viomb0 at virtio3
virtio3: apic 0 int 10
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 1: density unknown
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
nvram: invalid checksum
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (51b2d1225ee6d760.a) swap on sd0b dump on sd0b
--
M.K.
