Hi all, sashan@ and dlg@ I'm sorry if your eyes bleeds from pfsync panics. I just wanted to send panic log here after pf_purge backout. I'm testing pfsync with NET_TASKQ=6 because I have 6 core boxes and that way panics are faster to trigger. In this test I sure that pf is hitting state limit quite hard and I'm only generating traffic and nothing more. I'm generating traffic with cisco t-rex so traffic is real as what you can get in lab environment. Now I'm trying to trigger panic with WITNESS but it seems that it hard.
I just need to say that with all pf, pfsync and with pf_purge diffs after hackaton + this diff on tech@ https://www.mail-archive.com/tech@openbsd.org/msg72582.html my production firewall seems stable and it wasn't without that diff I'm not sure if we have same diffs but even Josmar Pierri on bugs@ https://www.mail-archive.com/bugs@openbsd.org/msg18994.html who had panics quite regularly with that diff on tech@ seems to have stable firewall now. r620-1# uvm_fault(0xffffffff82374288, 0x17, 0, 2) -> e kernel: page fault trap, code=0 Stopped at pfsync_q_del+0x96: movq %rdx,0x8(%rax) TID PID UID PRFLAGS PFLAGS CPU COMMAND *192892 19920 0 0x14000 0x200 5K softnet pfsync_q_del(fffffd82e8a4ce20) at pfsync_q_del+0x96 pf_remove_state(fffffd82e8a4ce20) at pf_remove_state+0x14b pfsync_in_del_c(fffffd8006d843b8,c,79,0) at pfsync_in_del_c+0x6f pfsync_input(ffff800022d60ad8,ffff800022d60ae4,f0,2) at pfsync_input+0x33c ip_deliver(ffff800022d60ad8,ffff800022d60ae4,f0,2) at ip_deliver+0x113 ipintr() at ipintr+0x69 if_netisr(0) at if_netisr+0xea taskq_thread(ffff800000030000) at taskq_thread+0x100 end trace frame: 0x0, count: 7 https://www.openbsd.org/ddb.html describes the minimum info required in bug reports. Insufficient info makes it difficult to find and fix bugs. ddb{5}> ddb{5}> show panic *cpu5: uvm_fault(0xffffffff82374288, 0x17, 0, 2) -> e ddb{5}> ddb{5}> show reg rdi 0 rsi 0xf rbp 0xffff800022d60920 rbx 0x3cc rdx 0xffffffffffffffff rcx 0x10 rax 0xf r8 0xfffffd82b9c8a8d0 r9 0xfffffd82cd8efe38 r10 0xfffffd82cd8efe38 r11 0x56c3d350aa4ea3d2 r12 0xffff8000008c4000 r13 0x28 r14 0xfffffd82e8a4ce20 r15 0xffff8000008c4720 rip 0xffffffff81c78626 pfsync_q_del+0x96 cs 0x8 rflags 0x10286 __ALIGN_SIZE+0xf286 rsp 0xffff800022d608f0 ss 0x10 pfsync_q_del+0x96: movq %rdx,0x8(%rax) ddb{5}> ddb{5}> ps /o TID PID UID PRFLAGS PFLAGS CPU COMMAND *192892 19920 0 0x14000 0x200 5K softnet ddb{5}> trace /t 0t 192892 sleep_finish(ffffffff8272dea0,1) at sleep_finish+0xfe tsleep(ffffffff823738f0,4,ffffffff81f880db,0) at tsleep+0xb2 main(0) at main+0x775 end trace frame: 0x0, count: -3 ddb{5}> ps PID TID PPID UID S FLAGS WAIT COMMAND 4058 502031 1 0 3 0x100083 ttyin ksh 48707 291695 1 0 3 0x100098 kqread cron 37722 219650 96592 95 3 0x1100092 kqread smtpd 38110 501136 96592 103 3 0x1100092 kqread smtpd 6191 274846 96592 95 3 0x1100092 kqread smtpd 93001 113813 96592 95 3 0x100092 kqread smtpd 48938 178445 96592 95 3 0x1100092 kqread smtpd 60938 255136 96592 95 3 0x1100092 kqread smtpd 96592 417634 1 0 3 0x100080 kqread smtpd 37963 127625 1 0 3 0x88 kqread sshd 32077 367571 1 0 3 0x100080 kqread ntpd 47614 473827 5112 83 3 0x100092 kqread ntpd 5112 74565 1 83 3 0x1100012 netlock ntpd 88022 357328 31934 74 3 0x1100092 bpf pflogd 31934 430624 1 0 3 0x80 netio pflogd 88174 57576 57658 73 3 0x1100090 kqread syslogd 57658 5426 1 0 3 0x100082 netio syslogd 93915 271655 0 0 3 0x14200 bored smr 60765 222732 0 0 3 0x14200 pgzero zerothread 75107 75302 0 0 3 0x14200 aiodoned aiodoned 26715 517414 0 0 3 0x14200 syncer update 30007 265724 0 0 3 0x14200 cleaner cleaner 53800 389208 0 0 3 0x14200 reaper reaper 19611 77488 0 0 3 0x14200 pgdaemon pagedaemon 38998 273899 0 0 3 0x14200 usbtsk usbtask 92490 66823 0 0 3 0x14200 usbatsk usbatsk 37775 10602 0 0 3 0x40014200 acpi0 acpi0 70481 440307 0 0 3 0x40014200 idle5 32082 94674 0 0 7 0x40014200 idle4 73245 209471 0 0 7 0x40014200 idle3 79232 28371 0 0 7 0x40014200 idle2 4208 272620 0 0 7 0x40014200 idle1 43275 465866 0 0 3 0x14200 bored sensors 90184 330319 0 0 3 0x14200 netlock softnet 39657 1663 0 0 3 0x14200 netlock softnet 18386 339074 0 0 3 0x14200 netlock softnet 87166 175644 0 0 3 0x14200 netlock softnet 71477 23850 0 0 3 0x14200 netlock softnet *19920 192892 0 0 7 0x14200 softnet 62868 70840 0 0 3 0x14200 bored systqmp 80577 254539 0 0 3 0x14200 netlock systq 61833 25332 0 0 3 0x40014200 netlock softclock 1618 140616 0 0 7 0x40014200 idle0 1 363826 0 0 3 0x82 wait init 0 0 -1 0 3 0x10200 scheduler swapper ddb{5}> ddb{5}> mach ddbcpu 0 Stopped at x86_ipi_db+0x12: leave x86_ipi_db(ffffffff822c1ff0) at x86_ipi_db+0x12 x86_ipi_handler() at x86_ipi_handler+0x80 Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23 _kernel_lock() at _kernel_lock+0xa9 softintr_dispatch(0) at softintr_dispatch+0x49 Xsoftclock() at Xsoftclock+0x1f acpicpu_idle() at acpicpu_idle+0x281 sched_idle(ffffffff822c1ff0) at sched_idle+0x280 end trace frame: 0x0, count: 7 ddb{0}> mach ddbcpu 1 Stopped at x86_ipi_db+0x12: leave x86_ipi_db(ffff800022509ff0) at x86_ipi_db+0x12 x86_ipi_handler() at x86_ipi_handler+0x80 Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23 acpicpu_idle() at acpicpu_idle+0x281 sched_idle(ffff800022509ff0) at sched_idle+0x280 end trace frame: 0x0, count: 10 ddb{1}> mach ddbcpu 2 Stopped at x86_ipi_db+0x12: leave x86_ipi_db(ffff800022512ff0) at x86_ipi_db+0x12 x86_ipi_handler() at x86_ipi_handler+0x80 Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23 acpicpu_idle() at acpicpu_idle+0x281 sched_idle(ffff800022512ff0) at sched_idle+0x280 end trace frame: 0x0, count: 10 ddb{2}> mach ddbcpu 3 Stopped at x86_ipi_db+0x12: leave x86_ipi_db(ffff80002251bff0) at x86_ipi_db+0x12 x86_ipi_handler() at x86_ipi_handler+0x80 Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23 acpicpu_idle() at acpicpu_idle+0x281 sched_idle(ffff80002251bff0) at sched_idle+0x280 end trace frame: 0x0, count: 10 ddb{3}> mach ddbcpu 4 Stopped at x86_ipi_db+0x12: leave x86_ipi_db(ffff800022524ff0) at x86_ipi_db+0x12 x86_ipi_handler() at x86_ipi_handler+0x80 Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23 acpicpu_idle() at acpicpu_idle+0x281 sched_idle(ffff800022524ff0) at sched_idle+0x280 end trace frame: 0x0, count: 10 ddb{4}> mach ddbcpu 5 Stopped at pfsync_q_del+0x96: movq %rdx,0x8(%rax) pfsync_q_del(fffffd82e8a4ce20) at pfsync_q_del+0x96 pf_remove_state(fffffd82e8a4ce20) at pf_remove_state+0x14b pfsync_in_del_c(fffffd8006d843b8,c,79,0) at pfsync_in_del_c+0x6f pfsync_input(ffff800022d60ad8,ffff800022d60ae4,f0,2) at pfsync_input+0x33c ip_deliver(ffff800022d60ad8,ffff800022d60ae4,f0,2) at ip_deliver+0x113 ipintr() at ipintr+0x69 if_netisr(0) at if_netisr+0xea taskq_thread(ffff800000030000) at taskq_thread+0x100 end trace frame: 0x0, count: 7
