Re: Page fault in _mca_init during startup
On Thu, Feb 4, 2021 at 7:40 PM Konstantin Belousov wrote: > On Thu, Feb 04, 2021 at 07:01:30PM -0700, Alan Somers wrote: > > On Thu, Feb 4, 2021 at 5:59 PM Konstantin Belousov > > wrote: > > > Do you have INVARIANTS enabled? If not, I am curious if enabling them > > > would convert that rare page fault into rare "CPU %d has more MC banks" > > > assert. > > > > > > Also might be the output of the > > > # for x in $(jot $(sysctl -n hw.ncpu) 0) ; do cpucontrol -m 0x179 > > > /dev/cpuctl$x; done > > > command will show the issue (0x179 is the MCG_CAP MSR). > > > You need to load cpuctl(4) if it is not loaded yet. > > > > > > > I don't have INVARIANTS enabled, and I can't enable it on the production > > servers. However, I can turn those three KASSERTs into VERIFYs and see > > what happens. Here is what your command shows on the server that > panicked: > > $ for x in $(jot $(sysctl -n hw.ncpu) 0) ; do sudo cpucontrol -m 0x179 > > /dev/cpuctl$x; done | uniq -c > > 16 MSR 0x179: 0x 0x0f000c14 > > 16 MSR 0x179: 0x 0x0f000814 > > It probably explains it, but it would be more telling if you left the > output as is, so that we can see which CPUs have MCG_CMCI_P (10) bit set. > I didn't sort them, so the first 16 have bit 10 set and the second 16 don't. > > I suspect that your machine has two sockets, and processor in one socket > has CPUs reporting MCG_CMCI_P, while other processor does not. Your SMP > is not quite symmetric, perhaps processors were from different bins? > Could be. Is there some MSR that reports a more specific version number? > > If BSP is selected on reporting socket, everything boots well. If > other socket wins the BSP selection race, cmci is not initialized, but > when per-cpu mca_init() sees CMCI_P bit, it calls cmci_setup() without > allocated cmc state, because BSP did not needed it. > > If I am right, then unconditionally allocating the memory is probably the > only choice there. > > commit 2e2c925ac3b626edc6492a57a80f6b87895801c2 > Author: Konstantin Belousov > Date: Fri Feb 5 04:32:05 2021 +0200 > > x86 mca: unconditionally allocate memory for cmc state > > diff --git a/sys/x86/x86/mca.c b/sys/x86/x86/mca.c > index 03100e77d455..dff3f7631f5c 100644 > --- a/sys/x86/x86/mca.c > +++ b/sys/x86/x86/mca.c > @@ -1047,7 +1047,7 @@ mca_setup(uint64_t mcg_cap) > "force_scan", CTLTYPE_INT | CTLFLAG_RW | CTLFLAG_MPSAFE, NULL, > 0, > sysctl_mca_scan, "I", "Force an immediate scan for machine > checks"); > #ifdef DEV_APIC > - if (cmci_supported(mcg_cap)) > + if (cpu_vendor_id == CPU_VENDOR_INTEL) > cmci_setup(); > else if (amd_thresholding_supported()) > amd_thresholding_setup(); > ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Page fault in _mca_init during startup
On Thu, Feb 04, 2021 at 07:01:30PM -0700, Alan Somers wrote: > On Thu, Feb 4, 2021 at 5:59 PM Konstantin Belousov > wrote: > > Do you have INVARIANTS enabled? If not, I am curious if enabling them > > would convert that rare page fault into rare "CPU %d has more MC banks" > > assert. > > > > Also might be the output of the > > # for x in $(jot $(sysctl -n hw.ncpu) 0) ; do cpucontrol -m 0x179 > > /dev/cpuctl$x; done > > command will show the issue (0x179 is the MCG_CAP MSR). > > You need to load cpuctl(4) if it is not loaded yet. > > > > I don't have INVARIANTS enabled, and I can't enable it on the production > servers. However, I can turn those three KASSERTs into VERIFYs and see > what happens. Here is what your command shows on the server that panicked: > $ for x in $(jot $(sysctl -n hw.ncpu) 0) ; do sudo cpucontrol -m 0x179 > /dev/cpuctl$x; done | uniq -c > 16 MSR 0x179: 0x 0x0f000c14 > 16 MSR 0x179: 0x 0x0f000814 It probably explains it, but it would be more telling if you left the output as is, so that we can see which CPUs have MCG_CMCI_P (10) bit set. I suspect that your machine has two sockets, and processor in one socket has CPUs reporting MCG_CMCI_P, while other processor does not. Your SMP is not quite symmetric, perhaps processors were from different bins? If BSP is selected on reporting socket, everything boots well. If other socket wins the BSP selection race, cmci is not initialized, but when per-cpu mca_init() sees CMCI_P bit, it calls cmci_setup() without allocated cmc state, because BSP did not needed it. If I am right, then unconditionally allocating the memory is probably the only choice there. commit 2e2c925ac3b626edc6492a57a80f6b87895801c2 Author: Konstantin Belousov Date: Fri Feb 5 04:32:05 2021 +0200 x86 mca: unconditionally allocate memory for cmc state diff --git a/sys/x86/x86/mca.c b/sys/x86/x86/mca.c index 03100e77d455..dff3f7631f5c 100644 --- a/sys/x86/x86/mca.c +++ b/sys/x86/x86/mca.c @@ -1047,7 +1047,7 @@ mca_setup(uint64_t mcg_cap) "force_scan", CTLTYPE_INT | CTLFLAG_RW | CTLFLAG_MPSAFE, NULL, 0, sysctl_mca_scan, "I", "Force an immediate scan for machine checks"); #ifdef DEV_APIC - if (cmci_supported(mcg_cap)) + if (cpu_vendor_id == CPU_VENDOR_INTEL) cmci_setup(); else if (amd_thresholding_supported()) amd_thresholding_setup(); ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Page fault in _mca_init during startup
On Thu, Feb 4, 2021 at 5:59 PM Konstantin Belousov
wrote:
> On Thu, Feb 04, 2021 at 05:19:43PM -0700, Alan Somers wrote:
> > On Thu, Feb 4, 2021 at 4:27 PM Mark Johnston wrote:
> >
> > > On Fri, Feb 05, 2021 at 12:58:34AM +0200, Konstantin Belousov wrote:
> > > > On Thu, Feb 04, 2021 at 01:34:13PM -0800, Matthew Macy wrote:
> > > > > On Thu, Feb 4, 2021 at 1:31 PM Alan Somers
> > > wrote:
> > > > > >
> > > > > > After upgrading a machine to FreeBSD, 12.2, it hit the following
> > > panic on
> > > > > > its first reboot. I suspect that a few other servers have hit
> this
> > > too,
> > > > > > but since it happens before swap is mounted there are no core
> dumps,
> > > and
> > > > > > they usually reboot immediately. The code in question hasn't
> > > changed since
> > > > > > 2018. The panic happened in cmci_monitor at line 930. Does
> anybody
> > > have
> > > > > > any suggestions for how I could debug further? I can't readily
> > > reproduce
> > > > > > it, and I can't dump core, but I'd like to investigate it any
> way I
> > > can.
> > > > > > The server in question has dual Xeon Gold 6142 CPUs.
> > > > > >
> > > > Try this.
> > > >
> > > > I think that there is no other dependencies in the startup order, but
> > > > cannot know it for sure.
> > > >
> > > > commit 19584e3d3e9606d591fa30999b370ed758960e8c
> > > > Author: Konstantin Belousov
> > > > Date: Fri Feb 5 00:56:09 2021 +0200
> > > >
> > > > x86: init mca before APs are started
> > >
> > > APs only call mca_init() after they have been released by the BSP
> > > though, and that happens later in SI_SUB_SMP.
> > >
> > > > diff --git a/sys/x86/x86/mca.c b/sys/x86/x86/mca.c
> > > > index 03100e77d455..e2bf2673cf69 100644
> > > > --- a/sys/x86/x86/mca.c
> > > > +++ b/sys/x86/x86/mca.c
> > > > @@ -1371,7 +1371,7 @@ mca_init_bsp(void *arg __unused)
> > > >
> > > > mca_init();
> > > > }
> > > > -SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_ANY, mca_init_bsp, NULL);
> > > > +SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_SECOND, mca_init_bsp,
> NULL);
> > > >
> > > > /* Called when a machine check exception fires. */
> > > > void
> > >
> >
> > kib's patch causes a different problem, and this one is reproducible:
> >
> > Fatal trap 12: page fault while in kernel mode
> > cpuid = 0; apic id = 00
> > fault virtual address = 0x18
> > fault code = supervisor read data, page not present
> > instruction pointer = 0x20:0x8125762c
> > stack pointer= 0x28:0x828dad90
> > frame pointer= 0x28:0x828dad90
> > code segment = base 0x0, limit 0xf, type 0x1b
> > = DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags = resume, IOPL = 0
> > current process = 0 ()
> > trap number = 12
> > panic: page fault
> > cpuid = 0
> > time = 1
> > KDB: stack backtrace:
> > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> > 0x828daa50
> > vpanic() at vpanic+0x17b/frame 0x828daaa0
> > panic() at panic+0x43/frame 0x828dab00
> > trap_fatal() at trap_fatal+0x391/frame 0x828dab60
> > trap_pfault() at trap_pfault+0x4f/frame 0x828dabb0
> > trap() at trap+0x286/frame 0x828dacc0
> > calltrap() at calltrap+0x8/frame 0x828dacc0
> > --- trap 0xc, rip = 0x8125762c, rsp = 0x828dad90, rbp =
> > 0x828dad90 ---
> > native_lapic_enable_cmc() at native_lapic_enable_cmc+0x1c/frame
> > 0x828dad90
> > _mca_init() at _mca_init+0x94c/frame 0x828dadd0
> > mi_startup() at mi_startup+0xdf/frame 0x828dadf0
> > btext() at btext+0x2c
> > KDB: enter: panic
> > [ thread pid 0 tid 0 ]
> > Stopped at kdb_enter+0x37: movq$0,0x12bc396(%rip)
> >
> > If you're wondering, the panic happens at this point in
> > native_lapic_enable_cmc:
> >
> > apic_id = PCPU_GET(apic_id);
> > KASSERT(lapics[apic_id].la_present,
> >("%s: missing APIC %u", __func__, apic_id));
> > lapics[apic_id].la_lvts[APIC_LVT_CMCI].lvt_masked = 0;<- panic here
> > lapics[apic_id].la_lvts[APIC_LVT_CMCI].lvt_active = 1;
> > if (bootverbose)
> > printf("lapic%u: CMCI unmasked\n", apic_id);
> > }
>
> Scratch this patch.
>
> Do you have INVARIANTS enabled? If not, I am curious if enabling them
> would convert that rare page fault into rare "CPU %d has more MC banks"
> assert.
>
> Also might be the output of the
> # for x in $(jot $(sysctl -n hw.ncpu) 0) ; do cpucontrol -m 0x179
> /dev/cpuctl$x; done
> command will show the issue (0x179 is the MCG_CAP MSR).
> You need to load cpuctl(4) if it is not loaded yet.
>
I don't have INVARIANTS enabled, and I can't enable it on the production
servers. However, I can turn those three KASSERTs into VERIFYs and see
what happens. Here is what your command shows on the server that panicked:
$ for x in $(jot $(sysctl -n hw.ncpu) 0) ; do sudo cpucontrol -m 0x179
/dev/cpuctl$x; done | uniq -c
16 MSR 0x179: 0x 0x0f000c14
16 MSR 0x179: 0x 0x0f000814
-Alan
___
freebsd
Re: Page fault in _mca_init during startup
On Thu, Feb 04, 2021 at 05:19:43PM -0700, Alan Somers wrote:
> On Thu, Feb 4, 2021 at 4:27 PM Mark Johnston wrote:
>
> > On Fri, Feb 05, 2021 at 12:58:34AM +0200, Konstantin Belousov wrote:
> > > On Thu, Feb 04, 2021 at 01:34:13PM -0800, Matthew Macy wrote:
> > > > On Thu, Feb 4, 2021 at 1:31 PM Alan Somers
> > wrote:
> > > > >
> > > > > After upgrading a machine to FreeBSD, 12.2, it hit the following
> > panic on
> > > > > its first reboot. I suspect that a few other servers have hit this
> > too,
> > > > > but since it happens before swap is mounted there are no core dumps,
> > and
> > > > > they usually reboot immediately. The code in question hasn't
> > changed since
> > > > > 2018. The panic happened in cmci_monitor at line 930. Does anybody
> > have
> > > > > any suggestions for how I could debug further? I can't readily
> > reproduce
> > > > > it, and I can't dump core, but I'd like to investigate it any way I
> > can.
> > > > > The server in question has dual Xeon Gold 6142 CPUs.
> > > > >
> > > Try this.
> > >
> > > I think that there is no other dependencies in the startup order, but
> > > cannot know it for sure.
> > >
> > > commit 19584e3d3e9606d591fa30999b370ed758960e8c
> > > Author: Konstantin Belousov
> > > Date: Fri Feb 5 00:56:09 2021 +0200
> > >
> > > x86: init mca before APs are started
> >
> > APs only call mca_init() after they have been released by the BSP
> > though, and that happens later in SI_SUB_SMP.
> >
> > > diff --git a/sys/x86/x86/mca.c b/sys/x86/x86/mca.c
> > > index 03100e77d455..e2bf2673cf69 100644
> > > --- a/sys/x86/x86/mca.c
> > > +++ b/sys/x86/x86/mca.c
> > > @@ -1371,7 +1371,7 @@ mca_init_bsp(void *arg __unused)
> > >
> > > mca_init();
> > > }
> > > -SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_ANY, mca_init_bsp, NULL);
> > > +SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_SECOND, mca_init_bsp, NULL);
> > >
> > > /* Called when a machine check exception fires. */
> > > void
> >
>
> kib's patch causes a different problem, and this one is reproducible:
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address = 0x18
> fault code = supervisor read data, page not present
> instruction pointer = 0x20:0x8125762c
> stack pointer= 0x28:0x828dad90
> frame pointer= 0x28:0x828dad90
> code segment = base 0x0, limit 0xf, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags = resume, IOPL = 0
> current process = 0 ()
> trap number = 12
> panic: page fault
> cpuid = 0
> time = 1
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0x828daa50
> vpanic() at vpanic+0x17b/frame 0x828daaa0
> panic() at panic+0x43/frame 0x828dab00
> trap_fatal() at trap_fatal+0x391/frame 0x828dab60
> trap_pfault() at trap_pfault+0x4f/frame 0x828dabb0
> trap() at trap+0x286/frame 0x828dacc0
> calltrap() at calltrap+0x8/frame 0x828dacc0
> --- trap 0xc, rip = 0x8125762c, rsp = 0x828dad90, rbp =
> 0x828dad90 ---
> native_lapic_enable_cmc() at native_lapic_enable_cmc+0x1c/frame
> 0x828dad90
> _mca_init() at _mca_init+0x94c/frame 0x828dadd0
> mi_startup() at mi_startup+0xdf/frame 0x828dadf0
> btext() at btext+0x2c
> KDB: enter: panic
> [ thread pid 0 tid 0 ]
> Stopped at kdb_enter+0x37: movq$0,0x12bc396(%rip)
>
> If you're wondering, the panic happens at this point in
> native_lapic_enable_cmc:
>
> apic_id = PCPU_GET(apic_id);
> KASSERT(lapics[apic_id].la_present,
>("%s: missing APIC %u", __func__, apic_id));
> lapics[apic_id].la_lvts[APIC_LVT_CMCI].lvt_masked = 0;<- panic here
> lapics[apic_id].la_lvts[APIC_LVT_CMCI].lvt_active = 1;
> if (bootverbose)
> printf("lapic%u: CMCI unmasked\n", apic_id);
> }
Scratch this patch.
Do you have INVARIANTS enabled? If not, I am curious if enabling them
would convert that rare page fault into rare "CPU %d has more MC banks"
assert.
Also might be the output of the
# for x in $(jot $(sysctl -n hw.ncpu) 0) ; do cpucontrol -m 0x179
/dev/cpuctl$x; done
command will show the issue (0x179 is the MCG_CAP MSR).
You need to load cpuctl(4) if it is not loaded yet.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Page fault in _mca_init during startup
On Thu, Feb 4, 2021 at 4:27 PM Mark Johnston wrote:
> On Fri, Feb 05, 2021 at 12:58:34AM +0200, Konstantin Belousov wrote:
> > On Thu, Feb 04, 2021 at 01:34:13PM -0800, Matthew Macy wrote:
> > > On Thu, Feb 4, 2021 at 1:31 PM Alan Somers
> wrote:
> > > >
> > > > After upgrading a machine to FreeBSD, 12.2, it hit the following
> panic on
> > > > its first reboot. I suspect that a few other servers have hit this
> too,
> > > > but since it happens before swap is mounted there are no core dumps,
> and
> > > > they usually reboot immediately. The code in question hasn't
> changed since
> > > > 2018. The panic happened in cmci_monitor at line 930. Does anybody
> have
> > > > any suggestions for how I could debug further? I can't readily
> reproduce
> > > > it, and I can't dump core, but I'd like to investigate it any way I
> can.
> > > > The server in question has dual Xeon Gold 6142 CPUs.
> > > >
> > Try this.
> >
> > I think that there is no other dependencies in the startup order, but
> > cannot know it for sure.
> >
> > commit 19584e3d3e9606d591fa30999b370ed758960e8c
> > Author: Konstantin Belousov
> > Date: Fri Feb 5 00:56:09 2021 +0200
> >
> > x86: init mca before APs are started
>
> APs only call mca_init() after they have been released by the BSP
> though, and that happens later in SI_SUB_SMP.
>
> > diff --git a/sys/x86/x86/mca.c b/sys/x86/x86/mca.c
> > index 03100e77d455..e2bf2673cf69 100644
> > --- a/sys/x86/x86/mca.c
> > +++ b/sys/x86/x86/mca.c
> > @@ -1371,7 +1371,7 @@ mca_init_bsp(void *arg __unused)
> >
> > mca_init();
> > }
> > -SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_ANY, mca_init_bsp, NULL);
> > +SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_SECOND, mca_init_bsp, NULL);
> >
> > /* Called when a machine check exception fires. */
> > void
>
kib's patch causes a different problem, and this one is reproducible:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x18
fault code = supervisor read data, page not present
instruction pointer = 0x20:0x8125762c
stack pointer= 0x28:0x828dad90
frame pointer= 0x28:0x828dad90
code segment = base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 0 ()
trap number = 12
panic: page fault
cpuid = 0
time = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0x828daa50
vpanic() at vpanic+0x17b/frame 0x828daaa0
panic() at panic+0x43/frame 0x828dab00
trap_fatal() at trap_fatal+0x391/frame 0x828dab60
trap_pfault() at trap_pfault+0x4f/frame 0x828dabb0
trap() at trap+0x286/frame 0x828dacc0
calltrap() at calltrap+0x8/frame 0x828dacc0
--- trap 0xc, rip = 0x8125762c, rsp = 0x828dad90, rbp =
0x828dad90 ---
native_lapic_enable_cmc() at native_lapic_enable_cmc+0x1c/frame
0x828dad90
_mca_init() at _mca_init+0x94c/frame 0x828dadd0
mi_startup() at mi_startup+0xdf/frame 0x828dadf0
btext() at btext+0x2c
KDB: enter: panic
[ thread pid 0 tid 0 ]
Stopped at kdb_enter+0x37: movq$0,0x12bc396(%rip)
If you're wondering, the panic happens at this point in
native_lapic_enable_cmc:
apic_id = PCPU_GET(apic_id);
KASSERT(lapics[apic_id].la_present,
("%s: missing APIC %u", __func__, apic_id));
lapics[apic_id].la_lvts[APIC_LVT_CMCI].lvt_masked = 0;<- panic here
lapics[apic_id].la_lvts[APIC_LVT_CMCI].lvt_active = 1;
if (bootverbose)
printf("lapic%u: CMCI unmasked\n", apic_id);
}
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Page fault in _mca_init during startup
On Fri, Feb 05, 2021 at 12:58:34AM +0200, Konstantin Belousov wrote: > On Thu, Feb 04, 2021 at 01:34:13PM -0800, Matthew Macy wrote: > > On Thu, Feb 4, 2021 at 1:31 PM Alan Somers wrote: > > > > > > After upgrading a machine to FreeBSD, 12.2, it hit the following panic on > > > its first reboot. I suspect that a few other servers have hit this too, > > > but since it happens before swap is mounted there are no core dumps, and > > > they usually reboot immediately. The code in question hasn't changed > > > since > > > 2018. The panic happened in cmci_monitor at line 930. Does anybody have > > > any suggestions for how I could debug further? I can't readily reproduce > > > it, and I can't dump core, but I'd like to investigate it any way I can. > > > The server in question has dual Xeon Gold 6142 CPUs. > > > > Try this. > > I think that there is no other dependencies in the startup order, but > cannot know it for sure. > > commit 19584e3d3e9606d591fa30999b370ed758960e8c > Author: Konstantin Belousov > Date: Fri Feb 5 00:56:09 2021 +0200 > > x86: init mca before APs are started APs only call mca_init() after they have been released by the BSP though, and that happens later in SI_SUB_SMP. > diff --git a/sys/x86/x86/mca.c b/sys/x86/x86/mca.c > index 03100e77d455..e2bf2673cf69 100644 > --- a/sys/x86/x86/mca.c > +++ b/sys/x86/x86/mca.c > @@ -1371,7 +1371,7 @@ mca_init_bsp(void *arg __unused) > > mca_init(); > } > -SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_ANY, mca_init_bsp, NULL); > +SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_SECOND, mca_init_bsp, NULL); > > /* Called when a machine check exception fires. */ > void > ___ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Page fault in _mca_init during startup
On Thu, Feb 04, 2021 at 04:05:42PM -0700, Alan Somers wrote: > On Thu, Feb 4, 2021 at 3:58 PM Konstantin Belousov > wrote: > > > On Thu, Feb 04, 2021 at 01:34:13PM -0800, Matthew Macy wrote: > > > On Thu, Feb 4, 2021 at 1:31 PM Alan Somers wrote: > > > > > > > > After upgrading a machine to FreeBSD, 12.2, it hit the following panic > > on > > > > its first reboot. I suspect that a few other servers have hit this > > too, > > > > but since it happens before swap is mounted there are no core dumps, > > and > > > > they usually reboot immediately. The code in question hasn't changed > > since > > > > 2018. The panic happened in cmci_monitor at line 930. Does anybody > > have > > > > any suggestions for how I could debug further? I can't readily > > reproduce > > > > it, and I can't dump core, but I'd like to investigate it any way I > > can. > > > > The server in question has dual Xeon Gold 6142 CPUs. > > > > > > > > > > I can't actually help :( but I can add a +1 with similar hardware or > > > equivalent specs. It's not frequent, but it's often enough to be > > > annoying. > > > -M > > > > > > > if (!(ctl & MC_CTL2_CMCI_EN)) > > > > /* This bank does not support CMCI. */ > > > > return; > > > > > > > > cc = &cmc_state[PCPU_GET(cpuid)][i];// <- panic here > > > > > > > > /* Determine maximum threshold. */ > > > > > > > > > > > > Fatal trap 12: page fault while in kernel mode > > > > cpuid = 26; apic id = 34 > > > > fault virtual address = 0xd0 > > > > fault code = supervisor read data, page not present > > > > instruction pointer = 0x20:0x8125a009 > > > > stack pointer= 0x28:0xfeb65f20 > > > > frame pointer= 0x28:0xfeb65f50 > > > > code segment = base 0x0, limit 0xf, type 0x1b > > > > = DPL 0, pres 1, long 1, def32 0, gran 1 > > > > processor eflags = resume, IOPL = 0 > > > > current process = 11 (idle: cpu26) > > > > trap number = 12 > > > > panic: page fault > > > > cpuid = 26 > > > > time = 1 > > > > KDB: stack backtrace: > > > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > > > > 0xfeb65be0 > > > > vpanic() at vpanic+0x17b/frame 0xfeb65c30 > > > > panic() at panic+0x43/frame 0xfeb65c90 > > > > trap_fatal() at trap_fatal+0x391/frame 0xfeb65cf0 > > > > trap_pfault() at trap_pfault+0x4f/frame 0xfeb65d40 > > > > trap() at trap+0x286/frame 0xfeb65e50 > > > > calltrap() at calltrap+0x8/frame 0xfeb65e50 > > > > --- trap 0xc, rip = 0x8125a009, rsp = 0xfeb65f20, rbp = > > > > 0xfeb65f50 --- > > > > _mca_init() at _mca_init+0x5d9/frame 0xfeb65f50 > > > > init_secondary_tail() at init_secondary_tail+0xfd/frame > > 0xfeb65f80 > > > > init_secondary() at init_secondary+0x2d1/frame 0xfeb65ff0 > > > > KDB: enter: panic > > > > [ thread pid 11 tid 100029 ] > > > > Stopped at kdb_enter+0x37: movq$0,0x12bc1f6(%rip) > > > > Try this. > > > > I think that there is no other dependencies in the startup order, but > > cannot know it for sure. > > > > commit 19584e3d3e9606d591fa30999b370ed758960e8c > > Author: Konstantin Belousov > > Date: Fri Feb 5 00:56:09 2021 +0200 > > > > x86: init mca before APs are started > > > > diff --git a/sys/x86/x86/mca.c b/sys/x86/x86/mca.c > > index 03100e77d455..e2bf2673cf69 100644 > > --- a/sys/x86/x86/mca.c > > +++ b/sys/x86/x86/mca.c > > @@ -1371,7 +1371,7 @@ mca_init_bsp(void *arg __unused) > > > > mca_init(); > > } > > -SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_ANY, mca_init_bsp, NULL); > > +SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_SECOND, mca_init_bsp, NULL); > > > > /* Called when a machine check exception fires. */ > > void > > > > I can test this patch on development servers, but so far I've only seen the > crash on production servers. Do you have any suggestions for how to force > the crash, or how to test this patch besides simply making sure that my dev > servers can boot? The race, as I see it, is that we call mca_init() on BSP too late, so malloc() that provides the storage for cmc_state array, could be called too late, before one of the APs was IPIed for startup. Patch ensures that mca_init_bsp() SYSINIT is finished before we go to start the APs. I do not think there is any reliable way to trigger the panic while keeping the patch usable, except to observe enough successfull boots. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Page fault in _mca_init during startup
On Thu, Feb 4, 2021 at 3:58 PM Konstantin Belousov wrote: > On Thu, Feb 04, 2021 at 01:34:13PM -0800, Matthew Macy wrote: > > On Thu, Feb 4, 2021 at 1:31 PM Alan Somers wrote: > > > > > > After upgrading a machine to FreeBSD, 12.2, it hit the following panic > on > > > its first reboot. I suspect that a few other servers have hit this > too, > > > but since it happens before swap is mounted there are no core dumps, > and > > > they usually reboot immediately. The code in question hasn't changed > since > > > 2018. The panic happened in cmci_monitor at line 930. Does anybody > have > > > any suggestions for how I could debug further? I can't readily > reproduce > > > it, and I can't dump core, but I'd like to investigate it any way I > can. > > > The server in question has dual Xeon Gold 6142 CPUs. > > > > > > > I can't actually help :( but I can add a +1 with similar hardware or > > equivalent specs. It's not frequent, but it's often enough to be > > annoying. > > -M > > > > > if (!(ctl & MC_CTL2_CMCI_EN)) > > > /* This bank does not support CMCI. */ > > > return; > > > > > > cc = &cmc_state[PCPU_GET(cpuid)][i];// <- panic here > > > > > > /* Determine maximum threshold. */ > > > > > > > > > Fatal trap 12: page fault while in kernel mode > > > cpuid = 26; apic id = 34 > > > fault virtual address = 0xd0 > > > fault code = supervisor read data, page not present > > > instruction pointer = 0x20:0x8125a009 > > > stack pointer= 0x28:0xfeb65f20 > > > frame pointer= 0x28:0xfeb65f50 > > > code segment = base 0x0, limit 0xf, type 0x1b > > > = DPL 0, pres 1, long 1, def32 0, gran 1 > > > processor eflags = resume, IOPL = 0 > > > current process = 11 (idle: cpu26) > > > trap number = 12 > > > panic: page fault > > > cpuid = 26 > > > time = 1 > > > KDB: stack backtrace: > > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > > > 0xfeb65be0 > > > vpanic() at vpanic+0x17b/frame 0xfeb65c30 > > > panic() at panic+0x43/frame 0xfeb65c90 > > > trap_fatal() at trap_fatal+0x391/frame 0xfeb65cf0 > > > trap_pfault() at trap_pfault+0x4f/frame 0xfeb65d40 > > > trap() at trap+0x286/frame 0xfeb65e50 > > > calltrap() at calltrap+0x8/frame 0xfeb65e50 > > > --- trap 0xc, rip = 0x8125a009, rsp = 0xfeb65f20, rbp = > > > 0xfeb65f50 --- > > > _mca_init() at _mca_init+0x5d9/frame 0xfeb65f50 > > > init_secondary_tail() at init_secondary_tail+0xfd/frame > 0xfeb65f80 > > > init_secondary() at init_secondary+0x2d1/frame 0xfeb65ff0 > > > KDB: enter: panic > > > [ thread pid 11 tid 100029 ] > > > Stopped at kdb_enter+0x37: movq$0,0x12bc1f6(%rip) > > Try this. > > I think that there is no other dependencies in the startup order, but > cannot know it for sure. > > commit 19584e3d3e9606d591fa30999b370ed758960e8c > Author: Konstantin Belousov > Date: Fri Feb 5 00:56:09 2021 +0200 > > x86: init mca before APs are started > > diff --git a/sys/x86/x86/mca.c b/sys/x86/x86/mca.c > index 03100e77d455..e2bf2673cf69 100644 > --- a/sys/x86/x86/mca.c > +++ b/sys/x86/x86/mca.c > @@ -1371,7 +1371,7 @@ mca_init_bsp(void *arg __unused) > > mca_init(); > } > -SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_ANY, mca_init_bsp, NULL); > +SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_SECOND, mca_init_bsp, NULL); > > /* Called when a machine check exception fires. */ > void > I can test this patch on development servers, but so far I've only seen the crash on production servers. Do you have any suggestions for how to force the crash, or how to test this patch besides simply making sure that my dev servers can boot? -Alan ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Page fault in _mca_init during startup
On Thu, Feb 04, 2021 at 01:34:13PM -0800, Matthew Macy wrote: > On Thu, Feb 4, 2021 at 1:31 PM Alan Somers wrote: > > > > After upgrading a machine to FreeBSD, 12.2, it hit the following panic on > > its first reboot. I suspect that a few other servers have hit this too, > > but since it happens before swap is mounted there are no core dumps, and > > they usually reboot immediately. The code in question hasn't changed since > > 2018. The panic happened in cmci_monitor at line 930. Does anybody have > > any suggestions for how I could debug further? I can't readily reproduce > > it, and I can't dump core, but I'd like to investigate it any way I can. > > The server in question has dual Xeon Gold 6142 CPUs. > > > > I can't actually help :( but I can add a +1 with similar hardware or > equivalent specs. It's not frequent, but it's often enough to be > annoying. > -M > > > if (!(ctl & MC_CTL2_CMCI_EN)) > > /* This bank does not support CMCI. */ > > return; > > > > cc = &cmc_state[PCPU_GET(cpuid)][i];// <- panic here > > > > /* Determine maximum threshold. */ > > > > > > Fatal trap 12: page fault while in kernel mode > > cpuid = 26; apic id = 34 > > fault virtual address = 0xd0 > > fault code = supervisor read data, page not present > > instruction pointer = 0x20:0x8125a009 > > stack pointer= 0x28:0xfeb65f20 > > frame pointer= 0x28:0xfeb65f50 > > code segment = base 0x0, limit 0xf, type 0x1b > > = DPL 0, pres 1, long 1, def32 0, gran 1 > > processor eflags = resume, IOPL = 0 > > current process = 11 (idle: cpu26) > > trap number = 12 > > panic: page fault > > cpuid = 26 > > time = 1 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > > 0xfeb65be0 > > vpanic() at vpanic+0x17b/frame 0xfeb65c30 > > panic() at panic+0x43/frame 0xfeb65c90 > > trap_fatal() at trap_fatal+0x391/frame 0xfeb65cf0 > > trap_pfault() at trap_pfault+0x4f/frame 0xfeb65d40 > > trap() at trap+0x286/frame 0xfeb65e50 > > calltrap() at calltrap+0x8/frame 0xfeb65e50 > > --- trap 0xc, rip = 0x8125a009, rsp = 0xfeb65f20, rbp = > > 0xfeb65f50 --- > > _mca_init() at _mca_init+0x5d9/frame 0xfeb65f50 > > init_secondary_tail() at init_secondary_tail+0xfd/frame 0xfeb65f80 > > init_secondary() at init_secondary+0x2d1/frame 0xfeb65ff0 > > KDB: enter: panic > > [ thread pid 11 tid 100029 ] > > Stopped at kdb_enter+0x37: movq$0,0x12bc1f6(%rip) Try this. I think that there is no other dependencies in the startup order, but cannot know it for sure. commit 19584e3d3e9606d591fa30999b370ed758960e8c Author: Konstantin Belousov Date: Fri Feb 5 00:56:09 2021 +0200 x86: init mca before APs are started diff --git a/sys/x86/x86/mca.c b/sys/x86/x86/mca.c index 03100e77d455..e2bf2673cf69 100644 --- a/sys/x86/x86/mca.c +++ b/sys/x86/x86/mca.c @@ -1371,7 +1371,7 @@ mca_init_bsp(void *arg __unused) mca_init(); } -SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_ANY, mca_init_bsp, NULL); +SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_SECOND, mca_init_bsp, NULL); /* Called when a machine check exception fires. */ void ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Page fault in _mca_init during startup
On Thu, Feb 4, 2021 at 1:31 PM Alan Somers wrote: > > After upgrading a machine to FreeBSD, 12.2, it hit the following panic on > its first reboot. I suspect that a few other servers have hit this too, > but since it happens before swap is mounted there are no core dumps, and > they usually reboot immediately. The code in question hasn't changed since > 2018. The panic happened in cmci_monitor at line 930. Does anybody have > any suggestions for how I could debug further? I can't readily reproduce > it, and I can't dump core, but I'd like to investigate it any way I can. > The server in question has dual Xeon Gold 6142 CPUs. > I can't actually help :( but I can add a +1 with similar hardware or equivalent specs. It's not frequent, but it's often enough to be annoying. -M > if (!(ctl & MC_CTL2_CMCI_EN)) > /* This bank does not support CMCI. */ > return; > > cc = &cmc_state[PCPU_GET(cpuid)][i];// <- panic here > > /* Determine maximum threshold. */ > > > Fatal trap 12: page fault while in kernel mode > cpuid = 26; apic id = 34 > fault virtual address = 0xd0 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0x8125a009 > stack pointer= 0x28:0xfeb65f20 > frame pointer= 0x28:0xfeb65f50 > code segment = base 0x0, limit 0xf, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = resume, IOPL = 0 > current process = 11 (idle: cpu26) > trap number = 12 > panic: page fault > cpuid = 26 > time = 1 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfeb65be0 > vpanic() at vpanic+0x17b/frame 0xfeb65c30 > panic() at panic+0x43/frame 0xfeb65c90 > trap_fatal() at trap_fatal+0x391/frame 0xfeb65cf0 > trap_pfault() at trap_pfault+0x4f/frame 0xfeb65d40 > trap() at trap+0x286/frame 0xfeb65e50 > calltrap() at calltrap+0x8/frame 0xfeb65e50 > --- trap 0xc, rip = 0x8125a009, rsp = 0xfeb65f20, rbp = > 0xfeb65f50 --- > _mca_init() at _mca_init+0x5d9/frame 0xfeb65f50 > init_secondary_tail() at init_secondary_tail+0xfd/frame 0xfeb65f80 > init_secondary() at init_secondary+0x2d1/frame 0xfeb65ff0 > KDB: enter: panic > [ thread pid 11 tid 100029 ] > Stopped at kdb_enter+0x37: movq$0,0x12bc1f6(%rip) > ___ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Page fault in _mca_init during startup
After upgrading a machine to FreeBSD, 12.2, it hit the following panic on its first reboot. I suspect that a few other servers have hit this too, but since it happens before swap is mounted there are no core dumps, and they usually reboot immediately. The code in question hasn't changed since 2018. The panic happened in cmci_monitor at line 930. Does anybody have any suggestions for how I could debug further? I can't readily reproduce it, and I can't dump core, but I'd like to investigate it any way I can. The server in question has dual Xeon Gold 6142 CPUs. if (!(ctl & MC_CTL2_CMCI_EN)) /* This bank does not support CMCI. */ return; cc = &cmc_state[PCPU_GET(cpuid)][i];// <- panic here /* Determine maximum threshold. */ Fatal trap 12: page fault while in kernel mode cpuid = 26; apic id = 34 fault virtual address = 0xd0 fault code = supervisor read data, page not present instruction pointer = 0x20:0x8125a009 stack pointer= 0x28:0xfeb65f20 frame pointer= 0x28:0xfeb65f50 code segment = base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 11 (idle: cpu26) trap number = 12 panic: page fault cpuid = 26 time = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfeb65be0 vpanic() at vpanic+0x17b/frame 0xfeb65c30 panic() at panic+0x43/frame 0xfeb65c90 trap_fatal() at trap_fatal+0x391/frame 0xfeb65cf0 trap_pfault() at trap_pfault+0x4f/frame 0xfeb65d40 trap() at trap+0x286/frame 0xfeb65e50 calltrap() at calltrap+0x8/frame 0xfeb65e50 --- trap 0xc, rip = 0x8125a009, rsp = 0xfeb65f20, rbp = 0xfeb65f50 --- _mca_init() at _mca_init+0x5d9/frame 0xfeb65f50 init_secondary_tail() at init_secondary_tail+0xfd/frame 0xfeb65f80 init_secondary() at init_secondary+0x2d1/frame 0xfeb65ff0 KDB: enter: panic [ thread pid 11 tid 100029 ] Stopped at kdb_enter+0x37: movq$0,0x12bc1f6(%rip) ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Suspected mbuf leak with Nginx + sendfile + TLS in 12.2-STABLE
On 2/4/21 8:08 AM, GomoR wrote: Dear FreeBSD community, we are encountering a DoS condition on our production machines. Our use case is an Nginx reverse proxy serving large files via HTTPS. This problem arose when switching kernel and userland from 12.1-RELEASE to 12.2-RELEASE. Ports were not upgraded (at first). Each time a user downloads a file, mbuf & mbuf_clusters are raising to reach the maximum limit in a matter of seconds. Those values are asserted by 'netstat -m' as follows: Normal situation: mbuf: 256, 26031105, 16767,5974,428087938, 0, 0 mbuf_cluster: 2048, 8135232, 18408,2704,101644203, 0, 0 Warning situtation: mbuf: 256, 26031105, 2981516, 151205,1109483561, 0, 0 mbuf_cluster: 2048, 8135232, 2983155,4201,319714617, 0, 0 We have seen a patch related to sendfile + KTLS + mbuf at the below link and we updated to -STABLE to apply: None of the sendfile or KTLS changes from Netflix are in 12, they are only in 13 and later. Don't transmit mbufs that aren't yet ready on TOE sockets. This includes mbufs waiting for data from sendfile() I/O requests, or mbufs awaiting encryption for KTLS. https://github.com/freebsd/freebsd-src/commit/14c77f30b201bf76119d59678e72051c09c2 This patch only applies to Chelsio T5/T6 NICs when using TOE (TCP offload) and doesn't affect freeing mbufs, it just fixes a race when the NIC could potentially send random garbage if it sends the mbuf before the scheduled disk I/O to populate it with data from disk has completed. NIC is: ix0: What can we do to help you find the root cause? The first step I would do if possible would be to bisect between the last known working version and the version that is known to be broken to determine which commit introduced the problem. One thing that could help here is to see if you can reproduce the problem using a 12.2 kernel on a 12.1 world + ports. If you can, then you can limit your bisecting to just building new kernels which will make that process quicker. You might also see if using a different NIC shows the same problem. If not, then it might point to a regression in the NIC driver (or perhaps in iflib as ix uses iflib I believe). -- John Baldwin ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Suspected mbuf leak with Nginx + sendfile + TLS in 12.2-STABLE
Dear FreeBSD community, we are encountering a DoS condition on our production machines. Our use case is an Nginx reverse proxy serving large files via HTTPS. This problem arose when switching kernel and userland from 12.1-RELEASE to 12.2-RELEASE. Ports were not upgraded (at first). Each time a user downloads a file, mbuf & mbuf_clusters are raising to reach the maximum limit in a matter of seconds. Those values are asserted by 'netstat -m' as follows: Normal situation: mbuf: 256, 26031105, 16767,5974,428087938, 0, 0 mbuf_cluster: 2048, 8135232, 18408,2704,101644203, 0, 0 Warning situtation: mbuf: 256, 26031105, 2981516, 151205,1109483561, 0, 0 mbuf_cluster: 2048, 8135232, 2983155,4201,319714617, 0, 0 We have seen a patch related to sendfile + KTLS + mbuf at the below link and we updated to -STABLE to apply: Don't transmit mbufs that aren't yet ready on TOE sockets. This includes mbufs waiting for data from sendfile() I/O requests, or mbufs awaiting encryption for KTLS. https://github.com/freebsd/freebsd-src/commit/14c77f30b201bf76119d59678e72051c09c2 Unfortunately for us, applying it didn't solve the issue. When we stop the download early, mbufs are freed. But past a threshold, we must reboot the server. The only remaining thing we can do is to ping the server, it is no more possible to connect with SSH, for instance. We also tried to set some loader.conf values which fixed nothing: hw.ix.enable_msix=0 hw.pci.enable_msix=0 hw.pci.enable_msi=0 net.inet.tcp.tso=0 hw.ix.flow_control=0 We also updated Nginx & OpenSSL to latest versions and tried Nginx to compile against FreeBSD shipped OpenSSL library. It did change nothing. Versions: openssl-1.1.1i,1 nginx-1.18.0_45,2 # ldd /usr/local/sbin/nginx /usr/local/sbin/nginx: libcrypt.so.5 => /lib/libcrypt.so.5 (0x800323000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x800344000) libssl.so.11 => /usr/local/lib/libssl.so.11 (0x8003e7000) libcrypto.so.11 => /usr/local/lib/libcrypto.so.11 (0x80047e000) libz.so.6 => /lib/libz.so.6 (0x800772000) libc.so.7 => /lib/libc.so.7 (0x80078e000) libthr.so.3 => /lib/libthr.so.3 (0x800b84000) NIC is: ix0: What can we do to help you find the root cause? Best regards, P.S.: adding jhb@ in Cc from bapt@ suggestion ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
