https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=283137
Bug ID: 283137
Summary: pf: states corruption since 93c80b79ad65c leading to
kernel panic
Product: Base System
Version: 14.2-STABLE
Hardware: Any
URL: https://github.com/opnsense/src/issues/230
OS: Any
Status: New
Severity: Affects Some People
Priority: ---
Component: kern
Assignee: [email protected]
Reporter: [email protected]
Hi,
OPNsense users report a pf state corruption since the deployment of
93c80b79ad65 which ends up in at least one kernel panic, but due to the nature
of the situation it could actually be multiple.
The issue seems quite prevalent on production systems and may crash a system
after just a couple of minutes of runtime.
One user provided a kernel dump. I'm attaching the info for triage here:
(kgdb) bt
#0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
#1 doadump (textdump=textdump@entry=0) at
/usr/src/sys/kern/kern_shutdown.c:405
#2 0xffffffff8049c36a in db_dump (dummy=<optimized out>, dummy2=<optimized
out>, dummy3=<optimized out>, dummy4=<optimized out>) at
/usr/src/sys/ddb/db_command.c:591
#3 0xffffffff8049c16d in db_command (last_cmdp=<optimized out>,
cmd_table=<optimized out>, dopager=false) at /usr/src/sys/ddb/db_command.c:504
#4 0xffffffff8049c2b6 in db_command_script
(command=command@entry=0xffffffff81bbf6d3 <db_recursion_data+3> "dump") at
/usr/src/sys/ddb/db_command.c:569
#5 0xffffffff804a1528 in db_script_exec (scriptname=<optimized out>,
warnifnotfound=warnifnotfound@entry=0) at /usr/src/sys/ddb/db_script.c:302
#6 0xffffffff804a1435 in db_script_kdbenter (eventname=<optimized out>) at
/usr/src/sys/ddb/db_script.c:325
#7 0xffffffff8049f4f1 in db_trap (type=<optimized out>, code=<optimized out>)
at /usr/src/sys/ddb/db_main.c:267
#8 0xffffffff80c09868 in kdb_trap (type=type@entry=3, code=code@entry=0,
tf=tf@entry=0xfffffe00e206e2e0) at /usr/src/sys/kern/subr_kdb.c:790
#9 0xffffffff810e0419 in trap (frame=0xfffffe00e206e2e0) at
/usr/src/sys/amd64/amd64/trap.c:608
#10 <signal handler called>
#11 kdb_enter (why=<optimized out>, msg=<optimized out>) at
/usr/src/sys/kern/subr_kdb.c:556
#12 0xffffffff80bb91d2 in vpanic (fmt=0xffffffff823f5cbd "Bad link elm %p
prev->next != elm", ap=ap@entry=0xfffffe00e206e510) at
/usr/src/sys/kern/kern_shutdown.c:955
#13 0xffffffff80bb9283 in panic (fmt=0xffffffff81d82c18 <cnputs_mtx+24> "") at
/usr/src/sys/kern/kern_shutdown.c:891
#14 0xffffffff823c1dd0 in pf_state_key_detach (s=s@entry=0xfffff803cc297b00,
idx=idx@entry=0) at /usr/src/sys/netpfil/pf/pf.c:1456
#15 0xffffffff823ad0ef in pf_detach_state (s=s@entry=0xfffff803cc297b00) at
/usr/src/sys/netpfil/pf/pf.c:1442
#16 0xffffffff823ac6d9 in pf_state_key_attach (skw=0xfffff803cc2c4420, sks=0x0,
s=0xfffff803cc297b00) at /usr/src/sys/netpfil/pf/pf.c:1355
#17 pf_state_insert (kif=<optimized out>,
orig_kif=orig_kif@entry=0xfffff80002150600, skw=0xfffff803cc2c4420,
sks=<optimized out>, s=s@entry=0xfffff803cc297b00)
at /usr/src/sys/netpfil/pf/pf.c:1535
#18 0xffffffff823ba740 in pf_create_state (r=0xfffff80227b7e000,
nr=0xfffff80189e7a800, a=<optimized out>, pd=0xfffffe00e206eb00, nsn=0x0,
nk=0x12, sk=<optimized out>,
m=0xfffff8001dc64800, off=20, sport=4843, dport=59668,
rewrite=0xfffffe00e206ea0c, kif=0xfffff80002150600, sm=0xfffffe00e206ec18,
tag=-1, bproto_sum=25520,
bip_sum=979, hdrlen=8, match_rules=<optimized out>) at
/usr/src/sys/netpfil/pf/pf.c:5025
#19 pf_test_rule (rm=rm@entry=0xfffffe00e206ebf0,
sm=sm@entry=0xfffffe00e206ec18, kif=kif@entry=0xfffff80002150600,
m=0xfffff8001dc64800, off=20,
pd=pd@entry=0xfffffe00e206eb00, am=0xfffffe00e206ebd8,
rsm=0xfffffe00e206ebc8, inp=0x0) at /usr/src/sys/netpfil/pf/pf.c:4800
#20 0xffffffff823b4471 in pf_test (dir=dir@entry=1, pflags=<optimized out>,
ifp=0xfffff80001906000, m0=m0@entry=0xfffffe00e206ed08, inp=<optimized out>,
default_actions=default_actions@entry=0x0) at
/usr/src/sys/netpfil/pf/pf.c:8269
#21 0xffffffff823dc177 in pf_check_in (m=0xfffffe00e206ed08, ifp=0x12,
flags=-502865312, ruleset=<optimized out>, inp=0xffffffff80c10af0 <putchar>)
at /usr/src/sys/netpfil/pf/pf_ioctl.c:6575
#22 0xffffffff80d19e98 in pfil_mbuf_common (pch=<optimized out>,
m=0xfffffe00e206ed08, m@entry=0xfffffe00e206ec48, ifp=0xfffff80001906000,
flags=65536, inp=inp@entry=0x0)
at /usr/src/sys/net/pfil.c:212
#23 pfil_mbuf_in (head=<optimized out>, m=m@entry=0xfffffe00e206ed08,
ifp=0xfffff80001906000, inp=inp@entry=0x0) at /usr/src/sys/net/pfil.c:230
#24 0xffffffff80d9c59a in ip_tryforward (m=0xfffff8001dc64800) at
/usr/src/sys/netinet/ip_fastfwd.c:312
#25 0xffffffff80d9fa9c in ip_input (m=0xfffff8001dc64800) at
/usr/src/sys/netinet/ip_input.c:621
#26 0xffffffff80d1682b in netisr_process_workstream_proto
(nwsp=0xfffffe003a5eca40, proto=1) at /usr/src/sys/net/netisr.c:927
#27 swi_net (arg=0xfffffe003a5eca40) at /usr/src/sys/net/netisr.c:974
#28 0xffffffff80b6ffc6 in intr_event_execute_handlers (ie=0xfffff80001a59100,
p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1205
#29 ithread_execute_handlers (ie=0xfffff80001a59100, p=<optimized out>) at
/usr/src/sys/kern/kern_intr.c:1218
#30 ithread_loop (arg=arg@entry=0xfffff80001a7a620) at
/usr/src/sys/kern/kern_intr.c:1306
#31 0xffffffff80b6c402 in fork_exit (callout=0xffffffff80b6fd70 <ithread_loop>,
arg=0xfffff80001a7a620, frame=0xfffffe00e206ef40) at
/usr/src/sys/kern/kern_fork.c:1164
#32 <signal handler called>
(kgdb) frame 14
#14 0xffffffff823c1dd0 in pf_state_key_detach (s=s@entry=0xfffff803cc297b00,
idx=idx@entry=0) at /usr/src/sys/netpfil/pf/pf.c:1456
warning: Source file is more recent than executable.
1456 TAILQ_REMOVE(&sk->states[idx], s, key_list[idx]);
(kgdb) list
1451 #ifdef INVARIANTS
1452 struct pf_keyhash *kh = &V_pf_keyhash[pf_hashkey(sk)];
1453
1454 PF_HASHROW_ASSERT(kh);
1455 #endif
1456 TAILQ_REMOVE(&sk->states[idx], s, key_list[idx]);
1457 s->key[idx] = NULL;
1458
1459 if (TAILQ_EMPTY(&sk->states[0]) && TAILQ_EMPTY(&sk->states[1]))
{
1460 LIST_REMOVE(sk, entry);
(kgdb) p *sk
$1 = {addr = {{{v4 = {s_addr = XXX}, v6 = {__u6_addr = {__u6_addr8 = "XXX",
<incomplete sequence XXX>,
__u6_addr16 = {XXX, XXX, XXX, XXX, XXX, XXX, XXX, XXX}, __u6_addr32
= {XXX, XXX, XXX, XXX}}},
addr8 = "XXX", <incomplete sequence \XXX>, addr16 = {XXX, XXX, XXX,
XXX, XXX, XXX, XXX,
XXX}, addr32 = {XXX, XXX, XXX, XXX}}}, {{v4 = {s_addr = XXX}, v6 =
{__u6_addr = {
__u6_addr8 = "XXX", <incomplete sequence XXX>, __u6_addr16 = {XXX,
XXX, XXX, XXX, XXX,
XXX, XXX, XXX}, __u6_addr32 = {XXX, XXX, XXX, XXX}}},
addr8 = "XXX", <incomplete sequence XXX>, addr16 = {XXX, XXX, XXX, XXX,
XXX, XXX, XXX,
XXX}, addr32 = {XXX, XXX, XXX, XXX}}}}, port = {49374, 57005}, af =
222 '\336', proto = 192 '\300',
pad = "\255", <incomplete sequence \336>, entry = {le_next =
0xdeadc0dedeadc0de, le_prev = 0xdeadc0dedeadc0de}, states = {{tqh_first =
0xdeadc0dedeadc0de,
tqh_last = 0xdeadc0dedeadc0de}, {tqh_first = 0xdeadc0dedeadc0de, tqh_last
= 0xdeadc0dedeadc0de}}}
(kgdb) p *sk->states
$2 = {tqh_first = 0xdeadc0dedeadc0de, tqh_last = 0xdeadc0dedeadc0de}
(kgdb) p *s
$3 = {id = 10415225491559546880, creatorid = 1082503010, direction = 1 '\001',
pad = "\000\000", state_flags = 128, timeout = 27 '\033', sync_state = 255
'\377',
sync_updates = 0 '\000', refs = 0, lock = 0xfffffe0109794688, sync_list =
{tqe_next = 0x0, tqe_prev = 0x0}, key_list = {{tqe_next = 0x0,
tqe_prev = 0xfffff803cc2c4458}, {tqe_next = 0x0, tqe_prev = 0x0}}, entry
= {le_next = 0x0, le_prev = 0x0}, src = {scrub = 0x0, seqlo = 0, seqhi = 0,
seqdiff = 0,
max_win = 0, mss = 0, state = 1 '\001', wscale = 0 '\000', tcp_est = 0
'\000', pad = ""}, dst = {scrub = 0x0, seqlo = 0, seqhi = 0, seqdiff = 0,
max_win = 0,
mss = 0, state = 0 '\000', wscale = 0 '\000', tcp_est = 0 '\000', pad =
""}, match_rules = {slh_first = 0x0}, rule = {ptr = 0xfffff80227b7e000, nr =
666361856},
anchor = {ptr = 0x0, nr = 0}, nat_rule = {ptr = 0xfffff80189e7a800, nr =
2313660416}, rt_addr = {{v4 = {s_addr = 0}, v6 = {__u6_addr = {
__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0,
0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, addr8 = '\000' <repeats 15 times>,
addr16 = {0,
0, 0, 0, 0, 0, 0, 0}, addr32 = {0, 0, 0, 0}}}, key =
{0xfffff803cc2c4420, 0x0}, kif = 0xfffff80002150600, orig_kif =
0xfffff80002150600, rt_kif = 0x0,
src_node = 0x0, nat_src_node = 0x0, packets = {0, 0}, bytes = {0, 0},
creation = 127, expire = 127, pfsync_time = 0, act = {rtableid = -1, qid = 0,
pqid = 0,
max_mss = 0, log = 0 '\000', set_tos = 0 '\000', min_ttl = 0 '\000', dnpipe
= 0, dnrpipe = 0, flags = 128, set_prio = "\000"}, tag = 0, rt = 0 '\000'}
Cheers,
Franco
--
You are receiving this mail because:
You are the assignee for the bug.
