Re: panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_UP)" failed: file "/usr/src/sys/net/route.c", line 506
On 26/11/21(Fri) 17:08, Alexander Bluhm wrote: > On Fri, Nov 26, 2021 at 12:22:39PM +0100, Claudio Jeker wrote: > > Guess someone introduced a double rtfree() somewhere. > > Only explenation for this panic. > > Here is a report with OpenBSD 6.9. Bug has been there for a long > time. > > https://marc.info/?l=openbsd-bugs=162435709704591=2 I wonder if there isn't a race with rtm_output() or a timeout. It would help a lot if one could monitor the routing messages to know which RTM_* command is issued to the kernel prior to the panic. If you could also figure out which route (dst, src, flags) is triggering the panic.
Re: panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_UP)" failed: file "/usr/src/sys/net/route.c", line 506
On Fri, Nov 26, 2021 at 12:22:39PM +0100, Claudio Jeker wrote: > Guess someone introduced a double rtfree() somewhere. > Only explenation for this panic. Here is a report with OpenBSD 6.9. Bug has been there for a long time. https://marc.info/?l=openbsd-bugs=162435709704591=2 bluhm
Re: panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_UP)" failed: file "/usr/src/sys/net/route.c", line 506
On Fri, Nov 26, 2021 at 12:09:48PM +0100, Alexander Bluhm wrote:
> On Fri, Nov 26, 2021 at 08:54:51AM +, Lucas wrote:
> > panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_UP)" failed:
> > file "/usr/src/sys/net/route.c", line 506
>
> During my performance tests I got the same trace once.
>
> login: panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_UP)"
> failed: file "
> /usr/src/sys/net/route.c", line 506
> Stopped at db_enter+0x10: popq%rbp
> TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
> *397403 91501 0 0x14000 0x2000 softnet
> db_enter() at db_enter+0x10
> panic(822af53c) at panic+0xbf
> __assert(8231ea01,822b69c9,1fa,82326314) at
> __assert+0x
> 25
> rtfree(fd8746702d98) at rtfree+0x298
> ip6_forward(fd80b545e300,fd8746702d98,0) at ip6_forward+0x118
> ip6_input_if(8000246b7d58,8000246b7d64,29,0,80767048) at
> ip6_in
> put_if+0x80d
> ipv6_input(80767048,fd80b545e300) at ipv6_input+0x39
> ether_input(80767048,fd80b545e300) at ether_input+0x39f
> if_input_process(80767048,8000246b7e48) at if_input_process+0x6f
> ifiq_process(8076ac00) at ifiq_process+0x69
> taskq_thread(80034080) at taskq_thread+0x81
> end trace frame: 0x0, count: 4
> https://www.openbsd.org/ddb.html describes the minimum info required in bug
> reports. Insufficient info makes it difficult to find and fix bugs.
> ddb{0}>
>
> OpenBSD 7.0-current (GENERIC.MP) #cvs : D2021.09.29.00.00.00: Sun Nov 21
> 08:09:21 CET 2021
> r...@ot31.obsd-lab.genua.de:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> It happened with a CVS checkout from Sep 29 2021.
>
> I was running this test command on a Linux box to generate traffic
> for btrace graphs.
>
> ssh_perform@lt13_iperf3_-6_-cfdd7:e83e:66bc:0726::36_-P10_-t60
> linux-openbsd-ipsec-tunnel4-tcp6-ip3fwd-ot31
>
> The means I have an iked IPv6 through IPv4 IPsec tunnel. The OpenBSD
> machine that panics is encrypting TCP traffic.
>
> Console outout:
> http://bluhm.genua.de/perform/results/7.0/2021-11-21T04%3A35%3A50Z/bsdcons-ot31.txt
>
Guess someone introduced a double rtfree() somewhere.
Only explenation for this panic.
--
:wq Claudio
Re: panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_UP)" failed: file "/usr/src/sys/net/route.c", line 506
On Fri, Nov 26, 2021 at 08:54:51AM +, Lucas wrote:
> panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_UP)" failed:
> file "/usr/src/sys/net/route.c", line 506
During my performance tests I got the same trace once.
login: panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_UP)"
failed: file "
/usr/src/sys/net/route.c", line 506
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
*397403 91501 0 0x14000 0x2000 softnet
db_enter() at db_enter+0x10
panic(822af53c) at panic+0xbf
__assert(8231ea01,822b69c9,1fa,82326314) at __assert+0x
25
rtfree(fd8746702d98) at rtfree+0x298
ip6_forward(fd80b545e300,fd8746702d98,0) at ip6_forward+0x118
ip6_input_if(8000246b7d58,8000246b7d64,29,0,80767048) at ip6_in
put_if+0x80d
ipv6_input(80767048,fd80b545e300) at ipv6_input+0x39
ether_input(80767048,fd80b545e300) at ether_input+0x39f
if_input_process(80767048,8000246b7e48) at if_input_process+0x6f
ifiq_process(8076ac00) at ifiq_process+0x69
taskq_thread(80034080) at taskq_thread+0x81
end trace frame: 0x0, count: 4
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}>
OpenBSD 7.0-current (GENERIC.MP) #cvs : D2021.09.29.00.00.00: Sun Nov 21
08:09:21 CET 2021
r...@ot31.obsd-lab.genua.de:/usr/src/sys/arch/amd64/compile/GENERIC.MP
It happened with a CVS checkout from Sep 29 2021.
I was running this test command on a Linux box to generate traffic
for btrace graphs.
ssh_perform@lt13_iperf3_-6_-cfdd7:e83e:66bc:0726::36_-P10_-t60
linux-openbsd-ipsec-tunnel4-tcp6-ip3fwd-ot31
The means I have an iked IPv6 through IPv4 IPsec tunnel. The OpenBSD
machine that panics is encrypting TCP traffic.
Console outout:
http://bluhm.genua.de/perform/results/7.0/2021-11-21T04%3A35%3A50Z/bsdcons-ot31.txt
bluhm
panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_UP)" failed: file "/usr/src/sys/net/route.c", line 506
Hello,
I got the following panic while trying out an iked(8) responder in a VPS
on Hetzner, type CX11. There is no dmesg yet because I left the ddb
console on in case I'm asked to run more commands there. It has 2 vCPUs
and it's running 7.0-stable with errata up to 004. Transcription
follows. Lemme know if I can provide with more information.
-Lucas
panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_UP)" failed: file
"/usr/src/sys/net/route.c", line 506
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
*258078 81837 0 0x14000 0x2000 softnet
db_enter() at db_enter+0x10
panic(81e54727) at panic+0xbf
__assert(81ec28fe,81e5fb1a,1fa,81ecf5fa) at
__assert+0x25
rtfree(fd806d13a540) at rtfree+0x298
ip6_forward(fd8047383c00,fd806d13a540,1) at ip6_forward+0x118
ip6_input_if(800020b130b8,800020b130c4,29,0,800c52a8) at
ip6_input_if+0x80d
ipv6_input(800c62a8,fd8047383c00) at ipv6_input+0x39
ether_input(800c62a8,800020b131a8) at ether_input+0x39f
if_input_process(800c62a8,800020b131a8) at if_input_process+0x6f
ifiq_process(800c66b8) at ifiq_process+0x69
taskq_thread(8002b080) at taskq_thread+0x81
end trace frame: 0x0, count: 4
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}> show panic
*cpu0: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_UP)" failed: file
"/usr/src/sys/net/route.c", line 506
ddb{0}> trace
db_enter() at db_enter+0x10
panic(81e54727) at panic+0xbf
__assert(81ec28fe,81e5fb1a,1fa,81ecf5fa) at
__assert+0x25
rtfree(fd806d13a540) at rtfree+0x298
ip6_forward(fd8047383c00,fd806d13a540,1) at ip6_forward+0x118
ip6_input_if(800020b130b8,800020b130c4,29,0,800c52a8) at
ip6_input_if+0x80d
ipv6_input(800c62a8,fd8047383c00) at ipv6_input+0x39
ether_input(800c62a8,800020b131a8) at ether_input+0x39f
if_input_process(800c62a8,800020b131a8) at if_input_process+0x6f
ifiq_process(800c66b8) at ifiq_process+0x69
taskq_thread(8002b080) at taskq_thread+0x81
end trace frame: 0x0, count: -11
ddb{0}> show reg
rdi0
rsi 0x14
rbp 0x800020b12d20
rbx 0xfd807d13a540
rdx 0xfe3f1e12
rcx0x286
rax 0x74
r80x800020b12b48
r9 0
r100
r11 0x4ff427155739d864
r12 0x8217aa00cpu_info_full_primary+0x2a00
r13 0x1
r140
r15 0x81e54727cmd0646_9_tim_udma+0x2b2a1
rip 0x818713b0db_enter+0x10
cs 0x8
rflags 0x202
rsp 0x800020b12d20
ss 0x10
db_enter+0x10: popq%rbp
ddb{0}> mach ddbcpu 1
Stopped at x86_ipi_db+0x12:leave
x86_ipi_db(8000209c8ff0) at x86_ipi_db+0x12
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
acpicpu_idle() at acpicpu_idle+0x281
sched_idle(8000209c8ff0) at sched_idle+0x27e
end trace frame: 0x0, count: 10
ddb{1}> show panic
*cpu0: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_UP)" failed: file
"/usr/src/sys/net/route.c", line 506
ddb{1}> trace
x86_ipi_db(8000209c8ff0) at x86_ipi_db+0x12
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
acpicpu_idle() at acpicpu_idle+0x281
sched_idle(8000209c8ff0) at sched_idle+0x27e
end trace frame: 0x0, count: -5
ddb{1}> show reg
rdi 0x8000209c8ff0
rsi0
rbp 0x800020b19240
rbx 0x82170ea8ipifunc+0x38
rdx0
rcx 0x7
rax 0xff7f
r8 0
r9 0
r100
r11 0x676da17a104f1a97
r12 0x7
r130
r14 0x8000209c8ff0
r150
rip 0x81871382x86_ipi_db+0x12
cs 0x8
rflags 0x202
rsp 0x800020b19230
ss 0x10
x86_ipi_db+0x12:leave
