Re: panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_LOCAL)" failed
On 2020-04-21 10:38, Martin Pieuchot wrote:
> On 20/04/20(Mon) 14:27, Julian Brost wrote:
>> On 2020-04-20 12:14, Martin Pieuchot wrote:
>>>> login: panic: kernel diagnostic assertion "!ISSET(rt->rt_flags,
>>>> RTF_LOCAL)" failed: file "/usr/src/sys/netinet6/nd6.c", line 727
>>>
>>> That means some part of the ND code is incorrectly setting an `expire'
>>> value to an entry that is local, and therefor should never expire.
>>>
>>> Could you try to reproduce the issue with the diff below? It should
>>> also panic but points us to the place where the bug is.
>>>
>>> [...]
>> With the diff applied, this is the panic message:
>>
>> starting network
>> vio0: DAD detected duplicate IPv6 address fe80:1::1: NS in/out=0/1, NA in=1
>> vio0: DAD complete for fe80:1::1 - duplicate found
>> vio0: manual intervention required
>> reordering libraries:ndp info overwritten for fe80:1::1 by
>> 76:fa:d3:57:ec:56 on vio0
>> panic: kernel diagnostic assertion "!ISSET(ln->ln_rt->rt_flags,
>> RTF_LOCAL)" failed: file "/usr/src/sys/netinet6/nd6.c", line 309
>> Stopped at db_enter+0x10: popq%rbp
>>
>> TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
>> *457148 43436 0 0x14000 0x2000 softnet
>> db_enter() at db_enter+0x10
>> panic() at panic+0x128
>> __assert(81c8d6ea,81c94c17,135,81c9fb69) at
>> __assert+0x
>> 2b
>>
>> nd6_llinfo_settimer(fd803ec6ff00,15180) at nd6_llinfo_settimer+0xdf
>> nd6_cache_lladdr(800972a8,800014a496a0,fd803714f874,8,86,0)
>> at n
>> d6_cache_lladdr+0x2be
>>
>> nd6_rtr_cache(fd8036ee3000,28,38,86) at nd6_rtr_cache+0x31e
>> icmp6_input(800014a499d8,800014a499e4,3a,18) at icmp6_input+0x33d
>> ip_deliver(800014a499d8,800014a499e4,3a,18) at ip_deliver+0x1b3
>> ip6_input_if(800014a499d8,800014a499e4,29,0,800972a8) at
>
> Thanks, diff below fixes nd6_rtr_cache(). It was already skipping
> static entries the same should be done for local entries.
>
> I left the panic in there in case there's another place where the bug
> can be triggered.
>
> Index: netinet6/nd6.c
> ===
> RCS file: /cvs/src/sys/netinet6/nd6.c,v
> retrieving revision 1.229
> diff -u -p -r1.229 nd6.c
> --- netinet6/nd6.c29 Nov 2019 16:41:01 - 1.229
> +++ netinet6/nd6.c21 Apr 2020 08:36:01 -
> @@ -306,6 +306,7 @@ nd6_llinfo_settimer(struct llinfo_nd6 *l
> time_t expire = time_uptime + secs;
>
> NET_ASSERT_LOCKED();
> + KASSERT(!ISSET(ln->ln_rt->rt_flags, RTF_LOCAL));
>
> ln->ln_rt->rt_expire = expire;
> if (!timeout_pending(_timer_to) || expire < nd6_timer_next) {
> @@ -,17 +1112,11 @@ nd6_cache_lladdr(struct ifnet *ifp, stru
>
> rt = nd6_lookup(from, 0, ifp, ifp->if_rdomain);
> if (rt == NULL) {
> -#if 0
> - /* nothing must be done if there's no lladdr */
> - if (!lladdr || !lladdrlen)
> - return NULL;
> -#endif
> -
> rt = nd6_lookup(from, 1, ifp, ifp->if_rdomain);
> is_newentry = 1;
> } else {
> - /* do nothing if static ndp is set */
> - if (rt->rt_flags & RTF_STATIC) {
> + /* do not overwrite local or static entry */
> + if (ISSET(rt->rt_flags, RTF_STATIC|RTF_LOCAL)) {
> rtfree(rt);
> return;
> }
>
Looks good to me. With this applied, I can no longer reproduce the
panic. Thanks for looking into this!
Julian
Re: panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_LOCAL)" failed
On 2020-04-20 12:14, Martin Pieuchot wrote: >> login: panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, >> RTF_LOCAL)" failed: file "/usr/src/sys/netinet6/nd6.c", line 727 > > That means some part of the ND code is incorrectly setting an `expire' > value to an entry that is local, and therefor should never expire. > > Could you try to reproduce the issue with the diff below? It should > also panic but points us to the place where the bug is. > > [...] With the diff applied, this is the panic message: starting network vio0: DAD detected duplicate IPv6 address fe80:1::1: NS in/out=0/1, NA in=1 vio0: DAD complete for fe80:1::1 - duplicate found vio0: manual intervention required reordering libraries:ndp info overwritten for fe80:1::1 by 76:fa:d3:57:ec:56 on vio0 panic: kernel diagnostic assertion "!ISSET(ln->ln_rt->rt_flags, RTF_LOCAL)" failed: file "/usr/src/sys/netinet6/nd6.c", line 309 Stopped at db_enter+0x10: popq%rbp TIDPIDUID PRFLAGS PFLAGS CPU COMMAND *457148 43436 0 0x14000 0x2000 softnet db_enter() at db_enter+0x10 panic() at panic+0x128 __assert(81c8d6ea,81c94c17,135,81c9fb69) at __assert+0x 2b nd6_llinfo_settimer(fd803ec6ff00,15180) at nd6_llinfo_settimer+0xdf nd6_cache_lladdr(800972a8,800014a496a0,fd803714f874,8,86,0) at n d6_cache_lladdr+0x2be nd6_rtr_cache(fd8036ee3000,28,38,86) at nd6_rtr_cache+0x31e icmp6_input(800014a499d8,800014a499e4,3a,18) at icmp6_input+0x33d ip_deliver(800014a499d8,800014a499e4,3a,18) at ip_deliver+0x1b3 ip6_input_if(800014a499d8,800014a499e4,29,0,800972a8) at ip6_in put_if+0x8a7 ipv6_input(800972a8,fd8036c8c200) at ipv6_input+0x39 ether_input(800972a8,fd8036c8c200,0) at ether_input+0x1d3 if_input_process(800972a8,800014a49ac8) at if_input_process+0x8c ifiq_process(80097648) at ifiq_process+0x69 taskq_thread(80022040) at taskq_thread+0x3d end trace frame: 0x0, count: 1 https://www.openbsd.org/ddb.html describes the minimum info required in bug reports. Insufficient info makes it difficult to find and fix bugs. ddb> ps PID TID PPIDUID S FLAGS WAIT COMMAND 52319 220335 48472 0 20x13ar 48472 160852 1385 0 30x100089 pause sh 4518 212129 53183115 30x100092 kqreadslaacd 18367 312990 53183115 30x100092 kqreadslaacd 53183 69949 1 0 30x100080 kqreadslaacd 1385 473193 1 0 30x10008b pause sh 35971 370200 0 0 3 0x14200 pgzerozerothread 99205 77666 0 0 3 0x14200 aiodoned aiodoned 74221 215874 0 0 3 0x14200 syncerupdate 70709 427908 0 0 3 0x14200 cleaner cleaner 64788 144629 0 0 3 0x14200 reaperreaper 12214 515539 0 0 3 0x14200 pgdaemon pagedaemon 92505 317381 0 0 3 0x14200 bored crynlk 76094 225660 0 0 3 0x14200 bored crypto 12517 102463 0 0 3 0x14200 fdprobe fdcattach 56081 234408 0 0 3 0x14200 bored viomb 37692 155909 0 0 3 0x14200 usbtskusbtask 94142 69832 0 0 3 0x14200 usbatsk usbatsk 4210 166502 0 0 3 0x40014200 acpi0 acpi0 *43436 457148 0 0 7 0x14200softnet 1013 500947 0 0 3 0x14200 bored systqmp 75108 420942 0 0 3 0x14200 bored systq 51573 324335 0 0 3 0x40014200 bored softclock 2368 335789 0 0 3 0x40014200idle0 28301 80641 0 0 3 0x14200 bored smr 1 451792 0 0 30x82 wait init 0 0 -1 0 3 0x10200 scheduler swapper
panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_LOCAL)" failed
Hi,
I encountered a reproducible kernel panic during an accidental IPv6
misconfiguration. In order to reproduce, the OpenBSD machine must be in
the same subnet as a router that has fe80::1/64 configured and sends
IPv6 route advertisements, for example with radvd using this config:
interface eth0 {
AdvSendAdvert on;
MinRtrAdvInterval 10;
MaxRtrAdvInterval 30;
prefix 2001:db8::/64 {
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
};
With this setup, I was able to to reliably trigger the assertion using
the following steps:
- Install Openbsd using 6.6/amd64 install66.iso
- IPv4: none
- IPv6: autoconf
- Reboot into system, log in
- echo inet6 alias fe80::1 64 >> /etc/hostname.vio0
# The file now contains the following:
# inet6 autoconf
# inet6 alias fe80::1 64
- Reboot and log in again
- ping6 2001::
# The exact address doesn't seem to matter, it also doesn't have to
# respond or anything. Sometimes this step isn't even necessary as the
# panic occurs by itself after the login prompt.
- Wait a bit (less than a minute in my case) and observe the panic
These steps ignore installing updates, however, this can also be
reproduced after using syspatch. The boot and panic log below was
created using the latest kernel available on 6.6. In case it's relevant,
I observed this in a QEMU VM running on Debian sid using libvirt.
Julian
>> OpenBSD/amd64 BOOT 3.45
boot>
booting hd0a:/bsd: 12670280+2937872+333136+0+704512
[992525+128+1010520+739210]=0x127fc78
entry point at 0x81001000
[ using 2743416 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2019 OpenBSD. All rights reserved.
https://www.OpenBSD.org
OpenBSD 6.6 (GENERIC) #8: Fri Apr 17 13:49:18 MDT 2020
r...@syspatch-66-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 1056800768 (1007MB)
avail mem = 1012199424 (965MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5ad0 (9 entries)
bios0: vendor SeaBIOS version "1.13.0-1" date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S5
acpi0: tables DSDT FACP APIC
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS), 2295.05 MHz, 06-3a-09
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,VMX,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,XSAVE,HV,NXE,RDTSCP,LONG,LAHF,FSGSBASE,TSC_ADJUST,SMEP,ERMS,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 999MHz
ioapic0 at mainbus0: apid 0 pa 0xfec0, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: removable
cd0(pciide0:0:0): using PIO mode 4, DMA mode 2
pciide0: channel 1 disabled (no drives)
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0
int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Red Hat QXL Video" rev 0x04
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 52:54:00:f6:d7:f8
virtio0: msix shared
azalia0 at pci0 dev 4 function 0 "Intel 82801FB HD Audio" rev 0x01: apic
0 int 11
azalia0: No codecs found
uhci0 at pci0 dev 5 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 10
uhci1 at pci0 dev 5 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 10
uhci2 at pci0 dev 5 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 11
ehci0 at pci0 dev 5 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 11
usb0 at ehci0: USB revision 2.0
uhub0 at usb0
