>Synopsis: Assertion failure when adding point-to-point routes to
>interfaces in rdomain with deleted loopback
>Category: Reliability
>Environment:
System : OpenBSD 6.3
Details : OpenBSD 6.3 (GENERIC) #3: Thu May 17 23:54:13 CEST 2018
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
Architecture: OpenBSD.amd64
Machine : amd64 (see Description)
>Description:
Adding a route to a point-to-point interface such as gre(4) or tun(4) where the
interface is in a
non-default rdomain and the loopback device for the given rdomain has been
destroyed will trigger a
kernel assertion failure, causing a system crash.
This issue has been observed and reproduced on both an amd64 system (virtual
machine on a Debian 9
host) and a macppc system (iBook G4).
>How-To-Repeat:
1) Create a new loopback device in a non-default rdomain. Example:
# ifconfig lo2 rdomain 2
2) The following two steps can be performed in any order.
2a) Create a point-to-point interface. The following example creates a new
tun(4) interface,
though this has also been reproduced with a gre(4) interface.
# ifconfig tun0 rdomain 2
2b) Delete the loopback device associated with the rdomain.
# ifconfig lo2 -rdomain destroy
3) Add a route to the point-to-point interface, e.g.
# ifconfig tun0 inet 192.168.200.1 192.168.200.2
The system will crash and drop to a ddb(4) prompt.
An example session is shown below:
bsd00# ifconfig lo2 rdomain 2
bsd00# ifconfig tun0 rdomain 2
bsd00# ifconfig lo2 -rdomain destroy
bsd00# ifconfig tun0 inet 192.168.200.1 192.168.200.2
panic: kernel diagnostic assertion "lo0ifp != NULL" failed: file
"/usr/src/sys/net/if.c", line 1483
Stopped at db_enter+0x5: popq %rbp
TID PID UID PRFLAGS PFLAGS CPU COMMAND
*411802 54483 0 0x3 0 0 ifconfig
db_enter() at db_enter+0x5
panic() at panic+0x129
__assert(ffffffff8138ada4,ffff800014c333a0,ffff80000049b800,ffffff003ef6be78) a
t __assert+0x24
p2p_rtrequest(ffff800014c334b8,ffff800000025710,2) at p2p_rtrequest+0x118
rtrequest(2,ffff80000049b800,ffff80000049c700,ffff80000049c758,200004) at rtreq
uest+0x61e
rt_ifa_add(0,ffff80000049c700,0) at rt_ifa_add+0x188
rt_ifa_addlocal(1c8a8c0) at rt_ifa_addlocal+0x92
in_ifinit(ffff80000049c700,ffff80000049b800,2d,1) at in_ifinit+0x109
in_ioctl(ffff800014c33780,ffff80000049b800,8040691a,6) at in_ioctl+0x5b2
ifioctl(ffff800014c33870,ffff800014bbdaa8,ffffff0025bec320,40) at ifioctl+0x2a2
sys_ioctl(360,ffff800014bbdaa8,36) at sys_ioctl+0x346
syscall() at syscall+0x206
--- syscall (number 54) ---
end of kernel
end trace frame: 0x7f7fffff07f0, count: 3
0x1bf363c1f57a:
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb> ps
PID TID PPID UID S FLAGS WAIT COMMAND
*54483 411802 67804 0 7 0x3 ifconfig
67804 406719 1 0 3 0x10008b pause ksh
33752 349240 1 0 3 0x100083 ttyin getty
69344 49464 1 0 3 0x100083 ttyin getty
86419 261149 1 0 3 0x100083 ttyin getty
16084 314847 1 0 3 0x100083 ttyin getty
14693 130667 1 0 3 0x100083 ttyin getty
18804 62490 1 0 3 0x100098 poll cron
93656 99210 1 0 3 0x80 select sshd
94598 493734 50765 83 3 0x100092 poll ntpd
50765 460758 22801 83 3 0x100092 poll ntpd
22801 14785 1 0 3 0x100080 poll ntpd
44177 492405 42746 73 3 0x100090 kqread syslogd
42746 141789 1 0 3 0x100082 netio syslogd
55388 108269 1 77 3 0x100090 poll dhclient
7242 504291 1 0 3 0x80 poll dhclient
92439 192126 0 0 2 0x14200 zerothread
14480 23135 0 0 3 0x14200 aiodoned aiodoned
81670 149974 0 0 3 0x14200 syncer update
40688 504393 0 0 3 0x14200 cleaner cleaner
28036 8918 0 0 3 0x14200 reaper reaper
69539 195139 0 0 3 0x14200 pgdaemon pagedaemon
4531 504194 0 0 3 0x14200 bored crynlk
48613 430891 0 0 3 0x14200 bored crypto
10293 303442 0 0 3 0x40014200 acpi0 acpi0
79193 510300 0 0 3 0x14200 bored softnet
77561 116378 0 0 3 0x14200 bored systqmp
41595 509656 0 0 3 0x14200 bored systq
36723 125478 0 0 3 0x40014200 bored softclock
79066 497396 0 0 3 0x40014200 idle0
1 472223 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb>
>Fix:
Inspecting the source code in "/usr/src/sys/net/if.c" places the failing assert
in the
"p2p_rtrequest()" function. The line immediately above the "KASSERT()" macro
which fails attempts to
obtain a pointer to the loopback interface associated with the given rdomain.
However, the loopback
interface associated with the rdomain has been destroyed, and so a null pointer
is returned,
triggering the assert.
I am not familiar with the kernel code, though I have suggestions for possible
fixes:
- In the case where step 2a is performed first, prevent the deletion of a
loopback interface if
other devices exist in the same rdomain.
- In the case where step 2b is performed first, ensure that an appropriate
loopback device is
created when any other interface is created within a non-default rdomain
where no loopback device
already exists. This already happens when an rdomain is created for the
first time, however it
does not happen when the rdomain already exists but the loopback has been
deleted.
The dmesg shown below is from the amd64 VM used for testing.
OpenBSD 6.3 (GENERIC) #3: Thu May 17 23:54:13 CEST 2018
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 1056833536 (1007MB)
avail mem = 1017876480 (970MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf6920 (9 entries)
bios0: vendor SeaBIOS version "1.10.2-1" date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: QEMU Virtual CPU version 2.5+, 3492.35 MHz
cpu0:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,MELTDOWN
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line
16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 999MHz
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
pvbus0 at mainbus0: KVM
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0
wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <QEMU HARDDISK>
wd0: 16-sector PIO, LBA48, 10240MB, 20971520 sectors
wd0(pciide0:0:0): using PIO mode 4, DMA mode 2
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> ATAPI 5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
em0 at pci0 dev 3 function 0 "Intel 82540EM" rev 0x03: apic 0 int 11, address
52:54:00:12:34:00
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on wd0a (137f00ae4f00d839.a) swap on wd0b dump on wd0b
WARNING: / was not properly unmounted
fd0 at fdc0 drive 1: density unknown
