[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

--- Comment #13 from Dmitry Petrov  ---
Getting very similar (with upd6_bind) crashes on multiple machines (vmware
guests) every few days after upgrade to recent 14/stable.

Kernel was compiled with "nooptions SCTP", otherwise it is generic.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

--- Comment #12 from Weldon Godfrey  ---
(In reply to Gleb Smirnoff from comment #11)
thanks, email with link to file sent to you.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

--- Comment #11 from Gleb Smirnoff  ---
Thanks for adding me, Richard! This unlikely is related to the mbuf field reuse
that we discussed. However, it is very likely the problem is in the area I
recently worked on.

Weldon, I'm interested in the vmcore file + contents of /boot/kernel + contents
of /usr/lib/debug/boot/kernel. Do you have an opportunity to put them on web?
Please compress them aggressively and you can use my PGP key to encrypt it. Key
id is 5D05CC22 and you can obtain or verify it here
https://docs.freebsd.org/en/articles/pgpkeys/#_gleb_smirnoff_glebiusfreebsd_org

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

Richard Scheffenegger  changed:

   What|Removed |Added

 CC||gleb...@freebsd.org

--- Comment #10 from Richard Scheffenegger  ---
Adding gleb; the latest backtrace has no hint of TCP in there, but UDP; maybe
the mbuf field reuse we've been discussing this week may have something to do
with this?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

--- Comment #9 from Weldon Godfrey  ---
I ran kdbg on the vmcore file, I got this...I don't know if this helps.

root@venom-f5:/var/crash # kgdb /boot/kernel/kernel /var/crash/vmcore.3
GNU gdb (GDB) 13.2 [GDB v13.2 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd14.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:
cpuid = 1; apic id = 01
fault virtual address   = 0xb8
fault code  = supervisor read data, page not present
instruction pointer = 0x20:0x80d4e890
stack pointer   = 0x28:0xfe01017ddc70
frame pointer   = 0x28:0xfe01017ddcf0
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 676 (isc-net-0005)
rdi: 816c7a30 rsi: 816c7a30 rdx: 00010200
rcx:   r8: f8001b0c1400  r9: 
rax: f8014f794a80 rbx: f8010c069540 rbp: fe01017ddcf0
r10:  r11: fe0103a59700 r12: f80245b80e20
r13: c2e5 r14: c2e5 r15: f8001b0c1400
trap number = 12
panic: page fault
cpuid = 1
time = 1702438402
KDB: stack backtrace:
#0 0x80b9302d at kdb_backtrace+0x5d
#1 0x80b46132 at vpanic+0x132
#2 0x80b45ff3 at panic+0x43
#3 0x8103285c at trap_fatal+0x40c
#4 0x810328af at trap_pfault+0x4f
#5 0x810098e8 at calltrap+0x8
#6 0x80d708ac at udp6_bind+0x13c
#7 0x80be9132 at sobind+0x32
#8 0x80bf05c5 at kern_bindat+0xc5
#9 0x80bf045b at sys_bind+0x9b
#10 0x81033119 at amd64_syscall+0x109
#11 0x8100a1fb at fast_syscall_common+0xf8
Uptime: 19d16h43m33s
Dumping 1669 out of 16338 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
57  __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct
pcpu,

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

Weldon Godfrey  changed:

   What|Removed |Added

Version|14.0-STABLE |14.0-RELEASE

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

--- Comment #8 from Weldon Godfrey  ---
It has happened to me 3 times since the original report on various releases. 
Not all that often but it is a lightly used webserver  I am just now installing
gdb, should have done that earlier, sorry.   I do have vmcore files, but they
are too large to attach, if someone wants one and has another way for me to
send, let me know.   Otherwise, I will attach core.txt on next crash since i
should have gdb installed then.

Last crash yesterday, this is the version info.   I will be upgrading latest
RELENG shortly
FreeBSD venom-f5 14.0-RELEASE FreeBSD 14.0-RELEASE #2
releng/14.0-n265380-f9716eee8ab: Tue Nov 21 13:04:48 CST 2023
root@venom-f5:/usr/obj/usr/src/amd64.amd64/sys/VENOM-F5 amd64

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

--- Comment #7 from Vedran Miletic  ---
(In reply to Mark Johnston from comment #5)
Unfortunately, not really, and this is the first time that it happened so I am
not that worried. I'll CC to bug 268699.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

--- Comment #6 from Richard Scheffenegger  ---
A full core would certainly be helpful to see the mbufs in play at the time of
the panic.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

Mark Johnston  changed:

   What|Removed |Added

 CC||rsch...@freebsd.org,
   ||tue...@freebsd.org

--- Comment #5 from Mark Johnston  ---
(In reply to Vedran Miletic from comment #4)
Thanks.  There is no need to compress core.txt.

Are you able to test FreeBSD-CURRENT?  It has assertions enabled which may
catch the problem more usefully.

CC a couple of TCP folks.  PR 268699 looks similar.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

--- Comment #4 from Vedran Miletic  ---
Created attachment 246274
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=246274=edit
xz compressed core.txt

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

--- Comment #3 from Mark Johnston  ---
(In reply to Vedran Miletic from comment #2)
Assuming you have coredumps enabled (e.g., "dumpdev" is set in /etc/rc.conf),
they should appear in /var/crash after a reboot.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

Vedran Miletic  changed:

   What|Removed |Added

 CC||ved...@miletic.net

--- Comment #2 from Vedran Miletic  ---
(In reply to Mark Johnston from comment #1)
I got this error on 14.0-RC4:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 10
fault virtual address   = 0x8
fault code  = supervisor read data, page not present
instruction pointer = 0x20:0x80be4343
stack pointer   = 0x28:0xfe0080454ae0
frame pointer   = 0x28:0xfe0080454b10
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 12 (swi1: netisr 0)
rdi: f801d713c290 rsi: f8018fed5100 rdx: 82a10090
rcx: 1000  r8: 82a10090  r9: fe0080454b50
rax: 0001 rbx: f801d713c290 rbp: fe0080454b10
r10: 000a r11: fe0080454740 r12: 
r13:  r14: f8018fed5100 r15: 1000
trap number = 12
panic: page fault
cpuid = 0
time = 1699775882
KDB: stack backtrace:
#0 0x80b9002d at kdb_backtrace+0x5d
#1 0x80b43132 at vpanic+0x132
#2 0x80b42ff3 at panic+0x43
#3 0x8100c85c at trap_fatal+0x40c
#4 0x8100c8af at trap_pfault+0x4f
#5 0x80fe3828 at calltrap+0x8
#6 0x80be454b at sbdrop+0x3b
#7 0x80d11c34 at tcp_do_segment+0x2e04
#8 0x80d0e4e5 at tcp_input_with_port+0xfe5
#9 0x80d0ee1b at tcp_input+0xb
#10 0x80cfe40d at ip_input+0x23d
#11 0x80c845b8 at swi_net+0x128
#12 0x80b01497 at ithread_loop+0x257
#13 0x80afdb0f at fork_exit+0x7f
#14 0x80fe488e at fork_trampoline+0xe

Where are the core files?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

Mark Johnston  changed:

   What|Removed |Added

 CC||ma...@freebsd.org

--- Comment #1 from Mark Johnston  ---
Do you have a backtrace with line number information?  This can be obtained
from the full dump, in core.*.txt.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

Graham Perrin  changed:

   What|Removed |Added

 Status|New |Open
 CC||grahamper...@gmail.com

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

Mark Linimon  changed:

   What|Removed |Added

   Keywords||crash

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 273890] Fatal trap 12: page fault while in kernel mode

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273890

Bug ID: 273890
   Summary: Fatal trap 12: page fault while in kernel mode
   Product: Base System
   Version: 14.0-STABLE
  Hardware: amd64
OS: Any
Status: New
  Severity: Affects Only Me
  Priority: ---
 Component: kern
  Assignee: b...@freebsd.org
  Reporter: wel...@excelsus.com

I know I am pushing things, I am running FreeBSD 14-BETA2 as a VM on TrueNAS
Scale (I also had this happen on BETA1).   But I thought you may want to look
at it.

I have three VMs running on the TrueNAS Scale hypervisor (same machine).   All
are not doing much yet at all (one just fresh install, another doing light DNS,
third running DNS and running a very lightly used apache web server).  So far
only the web server is panicing, the other two hasnt panic yet.  

I can provide a core dump if needed, it is too large to upload to this ticket.



% uname -a
FreeBSD venom-f5 14.0-BETA2 FreeBSD 14.0-BETA2 #1
releng/14.0-n265099-8d60ede293e: Sat Sep 16 11:18:03 CDT 2023
root@venom-f5:/usr/obj/usr/src/amd64.amd64/sys/VENOM-F5 amd64
___

Sep 16 20:26:24 venom-f5 named[684]: clients-per-query decreased to 18
Sep 16 20:30:38 venom-f5 sshd[1647]: error: Fssh_kex_exchange_identification:
Connection closed by remote host
Sep 16 20:46:24 venom-f5 named[684]: clients-per-query decreased to 17
Sep 16 20:53:07 venom-f5 syslogd: last message repeated 1 times
Sep 16 20:59:05 venom-f5 kernel:
Sep 16 20:56:00 venom-f5 syslogd: kernel boot file is /boot/kernel/kernel
Sep 16 20:56:00 venom-f5 kernel:
Sep 16 20:56:00 venom-f5 kernel: Fatal trap 12: page fault while in kernel mode
Sep 16 20:56:00 venom-f5 kernel: cpuid = 5; apic id = 05
Sep 16 20:56:00 venom-f5 kernel: fault virtual address  = 0xb8
Sep 16 20:56:00 venom-f5 kernel: fault code = supervisor read data,
page not present
Sep 16 20:56:00 venom-f5 kernel: instruction pointer=
0x20:0x80d4e090
Sep 16 20:56:00 venom-f5 kernel: stack pointer  =
0x28:0xfe01017a7c70
Sep 16 20:56:00 venom-f5 kernel: frame pointer  =
0x28:0xfe01017a7cf0
Sep 16 20:56:00 venom-f5 kernel: code segment   = base 0x0, limit
0xf, type 0x1b
Sep 16 20:56:00 venom-f5 kernel:= DPL 0, pres 1, long
1, def32 0, gran 1
Sep 16 20:56:00 venom-f5 kernel: processor eflags   = interrupt enabled,
resume, IOPL = 0
Sep 16 20:56:00 venom-f5 kernel: current process= 684
(isc-net-0001)
Sep 16 20:56:00 venom-f5 kernel: rdi: 816c79c0 rsi: 816c79c0
rdx: 00010200
Sep 16 20:56:00 venom-f5 kernel: rcx:   r8: f800073aa800 
r9: 
Sep 16 20:56:00 venom-f5 kernel: rax: f801dc915e00 rbx: f801dc8f8c40
rbp: fe01017a7cf0
Sep 16 20:56:00 venom-f5 kernel: r10:  r11: fe0102c14c60
r12: f801a252f220
Sep 16 20:56:00 venom-f5 kernel: r13: 13fe r14: 13fe
r15: f800073aa800
Sep 16 20:56:00 venom-f5 kernel: trap number= 12
Sep 16 20:56:00 venom-f5 kernel: panic: page fault
Sep 16 20:56:00 venom-f5 kernel: cpuid = 5
Sep 16 20:56:00 venom-f5 kernel: time = 1694915945
Sep 16 20:56:00 venom-f5 kernel: KDB: stack backtrace:
Sep 16 20:56:00 venom-f5 kernel: #0 0x80b92cad at kdb_backtrace+0x5d
Sep 16 20:56:00 venom-f5 kernel: #1 0x80b45db2 at vpanic+0x132
Sep 16 20:56:00 venom-f5 kernel: #2 0x80b45c73 at panic+0x43
Sep 16 20:56:00 venom-f5 kernel: #3 0x8103181c at trap_fatal+0x40c
Sep 16 20:56:00 venom-f5 kernel: #4 0x8103186f at trap_pfault+0x4f
Sep 16 20:56:00 venom-f5 kernel: #5 0x81008298 at calltrap+0x8
Sep 16 20:56:00 venom-f5 kernel: #6 0x80d700ac at udp6_bind+0x13c
Sep 16 20:56:00 venom-f5 kernel: #7 0x80be8b12 at sobind+0x32
Sep 16 20:56:00 venom-f5 kernel: #8 0x80bf0015 at kern_bindat+0xc5
Sep 16 20:56:00 venom-f5 kernel: #9 0x80befeab at sys_bind+0x9b
Sep 16 20:56:00 venom-f5 kernel: #10 0x810320d9 at amd64_syscall+0x109
Sep 16 20:56:00 venom-f5 kernel: #11 0x81008bab at
fast_syscall_common+0xf8
Sep 16 20:56:00 venom-f5 kernel: Uptime: 9h25m35s
Sep 16 20:56:00 venom-f5 kernel: Dumping 722 out of 16338
MB:..3%..12%..23%..32%..43%..51%..63%..71%..82%..91%
Sep 16 20:56:00 venom-f5 kernel: Dump complete
Sep 16 20:56:00 venom-f5 kernel: Automatic reboot in 15 seconds - press a key
on the console to abort
Sep 16 20:56:00 venom-f5 kernel: Rebooting...
Sep 16 20:56:00 venom-f5 kernel: cpu_reset: Restarting BSP
Sep 16 20:56:00 venom-f5 kernel: cpu_reset_proxy: Stopped CPU 5
Sep 16 20:56:00 venom-f5 kernel: ---<>---
Sep 16 20:56:00 venom-f5 kernel: Copyright (c) 1992-2023 The FreeBSD Project.
Sep 16 20:56:00 venom-f5 kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988,
1989, 1991, 1992, 1993, 1994
Sep 16 20:56:00 venom-f5 kernel:The Regents of the University of