Meta Mode (was: Re: BOOT CRASH -- Current -CURRENT)

2022-10-02 Thread Cy Schubert 
In message 
, Warner Losh writes:
> --65ac9c05ea048b2a
> Content-Type: text/plain; charset="UTF-8"
> Content-Transfer-Encoding: quoted-printable
>
> On Sat, Oct 1, 2022 at 9:06 PM Larry Rosenman  wrote:
>
> > On 10/01/2022 10:04 pm, Warner Losh wrote:
> >
> > Do  you have a /boot tarball that can be loaded in a VM that recreates th=
> e
> > problem (along with a clean hash)?
> >
> > But before you try that, have you tried a completely clean rebuild of the
> > kernel to preclude the possibility that something is somehow cross thread=
> ed?
> >
> > Warner
> >
> > On Sat, Oct 1, 2022 at 8:39 PM Larry Rosenman  wrote:
> >
> >
> > =E2=9D=AF more info.11
> > Dump header from device: /dev/mfid0p3
> >Architecture: amd64
> >Architecture Version: 2
> >Dump Length: 126748815
> >Blocksize: 512
> >Compression: zstd
> >Dumptime: 2022-10-01 21:26:40 -0500
> >Hostname:
> >Magic: FreeBSD Kernel Dump
> >Version String: FreeBSD 14.0-CURRENT #168
> > ler/freebsd-main-changes-n258354-6cdd871ebc4: Sat Oct  1 21:13:01 CDT
> > 2022
> >  r...@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL
> >Panic String: page fault
> >Dump Parity: 501115454
> >Bounds: 11
> >Dump Status: good
> >
> > I do have source and debug stuff, BUT kgdb croaks on me.
> >
> > I *CAN* give access to the machine.
> >
> > the console backtrace showed something about the kld load of
> > dependencies.
> >
> >
> >
> > --
> > Larry Rosenman http://people.freebsd.org/~ler
> > Phone: +1 214-642-9640 E-Mail: l...@freebsd.org
> > US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
> >
> > let me wipe /usr/obj, and rebuild everything (I *DO* use meta-mode).
> >
>
> I've had fewer problems with it than non-meta mode, but this looks like a
> 'corruption' or 'cross threaded' crash I've chased in the past that went
> away with a rebuild. So it's better to be sure...

I think so too. What may appear to be a gratuitous rebuild of llvm, for 
example, is in fact meta mode rebuilding because of some makefile change. 
Without meta mode I've experienced odd weirdnesses that are fixed through a 
subsequent clean build.

I just started using meta mode again this week after a few years hiatus to 
see if it addresses the occasional weird behaviour due to something not 
being rebuilt when it should have been.


-- 
Cheers,
Cy Schubert 
FreeBSD UNIX: Web:  http://www.FreeBSD.org
NTP:   Web:  https://nwtime.org

e^(i*pi)+1=0

Re: BOOT CRASH -- Current -CURRENT

2022-10-01 Thread Larry Rosenman 



On 10/01/2022 10:43 pm, Larry Rosenman wrote:


On 10/01/2022 10:08 pm, Warner Losh wrote:

On Sat, Oct 1, 2022 at 9:06 PM Larry Rosenman  wrote:

On 10/01/2022 10:04 pm, Warner Losh wrote:

Do  you have a /boot tarball that can be loaded in a VM that recreates 
the problem (along with a clean hash)?


But before you try that, have you tried a completely clean rebuild of 
the kernel to preclude the possibility that something is somehow cross 
threaded?


Warner

On Sat, Oct 1, 2022 at 8:39 PM Larry Rosenman  wrote:
❯ more info.11
Dump header from device: /dev/mfid0p3
Architecture: amd64
Architecture Version: 2
Dump Length: 126748815
Blocksize: 512
Compression: zstd
Dumptime: 2022-10-01 21:26:40 -0500
Hostname:
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 14.0-CURRENT #168
ler/freebsd-main-changes-n258354-6cdd871ebc4: Sat Oct  1 21:13:01 CDT
2022
r...@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL
Panic String: page fault
Dump Parity: 501115454
Bounds: 11
Dump Status: good

I do have source and debug stuff, BUT kgdb croaks on me.

I *CAN* give access to the machine.

the console backtrace showed something about the kld load of
dependencies.

--
Larry Rosenman http://people.freebsd.org/~ler
Phone: +1 214-642-9640 E-Mail: l...@freebsd.org
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106


let me wipe /usr/obj, and rebuild everything (I *DO* use meta-mode).

I've had fewer problems with it than non-meta mode, but this looks like 
a 'corruption' or 'cross threaded' crash I've chased in the past that 
went away with a rebuild. So it's better to be sure...


Warner

Still breaks -- did someone(tm) forget to make netlink a module?

--
Larry Rosenman http://people.freebsd.org/~ler
Phone: +1 214-642-9640 E-Mail: l...@freebsd.org
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

❯ sudo kgdb -c vmcore.12 /mnt/usr/lib/debug/boot/kernel/kernel.debug
GNU gdb (GDB) 12.1 [GDB v12.1 for FreeBSD]
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd14.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /mnt/usr/lib/debug/boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:
---<>---
Copyright (c) 1992-2022 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #0 ler/freebsd-main-changes-n258354-6cdd871ebc4: 
Sat Oct  1 22:30:48 CDT 2022
r...@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL 
amd64
FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git 
llvmorg-14.0.5-0-gc12386ae247c)

VT(efifb): resolution 640x480
CPU: Intel(R) Xeon(R) CPU   X5660  @ 2.80GHz (2793.16-MHz 
K8-class CPU)

  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
  
Features=0xbfebfbff
  
Features2=0x29ee3ff

  AMD Features=0x2c100800
  AMD Features2=0x1
  Structured Extended Features3=0x9c00
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 137438953472 (131072 MB)
avail memory = 133789515776 (127591 MB)
CPU microcode: no matching update found
Event timer "LAPIC" quality 600
ACPI APIC Table: 
FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs
FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 hardware threads
random: unblocking device.
ioapic1: MADT APIC ID 1 != hw id 0
ioapic0  irqs 0-23
ioapic1  irqs 32-55
Launching APs: 1 14 12 21 2 6 17 10 18 15 4 19 7 3 8 20 13 5 23 11 9 16 
22

TCP_ratelimit: Is now initialized
TCP Hpts created 24 swi interrupt threads and bound 24 to NUMA domains
random: entropy device external interface
kbd1 at kbdmux0
acpi0: 
acpi0: Power Button (fixed)
apei0:  on acpi0
cpu0:  on acpi0
atrtc0:  port 0x70-0x7f irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.00s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0:  port 0x40-0x5f irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0:  iomem 0xfed0-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency

Re: BOOT CRASH -- Current -CURRENT

2022-10-01 Thread Larry Rosenman 



On 10/01/2022 10:08 pm, Warner Losh wrote:


On Sat, Oct 1, 2022 at 9:06 PM Larry Rosenman  wrote:

On 10/01/2022 10:04 pm, Warner Losh wrote:

Do  you have a /boot tarball that can be loaded in a VM that recreates 
the problem (along with a clean hash)?


But before you try that, have you tried a completely clean rebuild of 
the kernel to preclude the possibility that something is somehow cross 
threaded?


Warner

On Sat, Oct 1, 2022 at 8:39 PM Larry Rosenman  wrote:
❯ more info.11
Dump header from device: /dev/mfid0p3
Architecture: amd64
Architecture Version: 2
Dump Length: 126748815
Blocksize: 512
Compression: zstd
Dumptime: 2022-10-01 21:26:40 -0500
Hostname:
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 14.0-CURRENT #168
ler/freebsd-main-changes-n258354-6cdd871ebc4: Sat Oct  1 21:13:01 CDT
2022
r...@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL
Panic String: page fault
Dump Parity: 501115454
Bounds: 11
Dump Status: good

I do have source and debug stuff, BUT kgdb croaks on me.

I *CAN* give access to the machine.

the console backtrace showed something about the kld load of
dependencies.

--
Larry Rosenman http://people.freebsd.org/~ler
Phone: +1 214-642-9640 E-Mail: l...@freebsd.org
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106


let me wipe /usr/obj, and rebuild everything (I *DO* use meta-mode).

I've had fewer problems with it than non-meta mode, but this looks like 
a 'corruption' or 'cross threaded' crash I've chased in the past that 
went away with a rebuild. So it's better to be sure...


Warner

Still breaks -- did someone(tm) forget to make netlink a module?

--
Larry Rosenman http://people.freebsd.org/~ler
Phone: +1 214-642-9640 E-Mail: l...@freebsd.org
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Re: BOOT CRASH -- Current -CURRENT

2022-10-01 Thread Warner Losh 
On Sat, Oct 1, 2022 at 9:06 PM Larry Rosenman  wrote:

> On 10/01/2022 10:04 pm, Warner Losh wrote:
>
> Do  you have a /boot tarball that can be loaded in a VM that recreates the
> problem (along with a clean hash)?
>
> But before you try that, have you tried a completely clean rebuild of the
> kernel to preclude the possibility that something is somehow cross threaded?
>
> Warner
>
> On Sat, Oct 1, 2022 at 8:39 PM Larry Rosenman  wrote:
>
>
> ❯ more info.11
> Dump header from device: /dev/mfid0p3
>Architecture: amd64
>Architecture Version: 2
>Dump Length: 126748815
>Blocksize: 512
>Compression: zstd
>Dumptime: 2022-10-01 21:26:40 -0500
>Hostname:
>Magic: FreeBSD Kernel Dump
>Version String: FreeBSD 14.0-CURRENT #168
> ler/freebsd-main-changes-n258354-6cdd871ebc4: Sat Oct  1 21:13:01 CDT
> 2022
>  r...@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL
>Panic String: page fault
>Dump Parity: 501115454
>Bounds: 11
>Dump Status: good
>
> I do have source and debug stuff, BUT kgdb croaks on me.
>
> I *CAN* give access to the machine.
>
> the console backtrace showed something about the kld load of
> dependencies.
>
>
>
> --
> Larry Rosenman http://people.freebsd.org/~ler
> Phone: +1 214-642-9640 E-Mail: l...@freebsd.org
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
>
> let me wipe /usr/obj, and rebuild everything (I *DO* use meta-mode).
>

I've had fewer problems with it than non-meta mode, but this looks like a
'corruption' or 'cross threaded' crash I've chased in the past that went
away with a rebuild. So it's better to be sure...

Warner

Re: BOOT CRASH -- Current -CURRENT

2022-10-01 Thread Larry Rosenman 



On 10/01/2022 10:04 pm, Warner Losh wrote:

Do  you have a /boot tarball that can be loaded in a VM that recreates 
the problem (along with a clean hash)?


But before you try that, have you tried a completely clean rebuild of 
the kernel to preclude the possibility that something is somehow cross 
threaded?


Warner

On Sat, Oct 1, 2022 at 8:39 PM Larry Rosenman  wrote:


❯ more info.11
Dump header from device: /dev/mfid0p3
Architecture: amd64
Architecture Version: 2
Dump Length: 126748815
Blocksize: 512
Compression: zstd
Dumptime: 2022-10-01 21:26:40 -0500
Hostname:
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 14.0-CURRENT #168
ler/freebsd-main-changes-n258354-6cdd871ebc4: Sat Oct  1 21:13:01 CDT
2022
r...@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL
Panic String: page fault
Dump Parity: 501115454
Bounds: 11
Dump Status: good

I do have source and debug stuff, BUT kgdb croaks on me.

I *CAN* give access to the machine.

the console backtrace showed something about the kld load of
dependencies.

--
Larry Rosenman http://people.freebsd.org/~ler
Phone: +1 214-642-9640 E-Mail: l...@freebsd.org
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106


let me wipe /usr/obj, and rebuild everything (I *DO* use meta-mode).

--
Larry Rosenman http://people.freebsd.org/~ler
Phone: +1 214-642-9640 E-Mail: l...@freebsd.org
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Re: BOOT CRASH -- Current -CURRENT

2022-10-01 Thread Warner Losh 
Do  you have a /boot tarball that can be loaded in a VM that recreates the
problem (along with a clean hash)?

But before you try that, have you tried a completely clean rebuild of the
kernel to preclude the possibility that something is somehow cross threaded?

Warner

On Sat, Oct 1, 2022 at 8:39 PM Larry Rosenman  wrote:

>
> ❯ more info.11
> Dump header from device: /dev/mfid0p3
>Architecture: amd64
>Architecture Version: 2
>Dump Length: 126748815
>Blocksize: 512
>Compression: zstd
>Dumptime: 2022-10-01 21:26:40 -0500
>Hostname:
>Magic: FreeBSD Kernel Dump
>Version String: FreeBSD 14.0-CURRENT #168
> ler/freebsd-main-changes-n258354-6cdd871ebc4: Sat Oct  1 21:13:01 CDT
> 2022
>  r...@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL
>Panic String: page fault
>Dump Parity: 501115454
>Bounds: 11
>Dump Status: good
>
> I do have source and debug stuff, BUT kgdb croaks on me.
>
> I *CAN* give access to the machine.
>
> the console backtrace showed something about the kld load of
> dependencies.
>
>
>
> --
> Larry Rosenman http://people.freebsd.org/~ler
> Phone: +1 214-642-9640 E-Mail: l...@freebsd.org
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
>
>

Re: BOOT CRASH -- Current -CURRENT

2022-10-01 Thread Larry Rosenman 

On 10/01/2022 9:39 pm, Larry Rosenman wrote:

❯ more info.11
Dump header from device: /dev/mfid0p3
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 126748815
  Blocksize: 512
  Compression: zstd
  Dumptime: 2022-10-01 21:26:40 -0500
  Hostname:
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 14.0-CURRENT #168 
ler/freebsd-main-changes-n258354-6cdd871ebc4: Sat Oct  1 21:13:01 CDT 
2022

r...@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL
  Panic String: page fault
  Dump Parity: 501115454
  Bounds: 11
  Dump Status: good

I do have source and debug stuff, BUT kgdb croaks on me.

I *CAN* give access to the machine.

the console backtrace showed something about the kld load of 
dependencies.


Here's the BT:
❯ sudo kgdb -c vmcore.11 /mnt/usr/lib/debug/boot/kernel/kernel.debug
GNU gdb (GDB) 12.1 [GDB v12.1 for FreeBSD]
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd14.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /mnt/usr/lib/debug/boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:
---<>---
Copyright (c) 1992-2022 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #168 ler/freebsd-main-changes-n258354-6cdd871ebc4: 
Sat Oct  1 21:13:01 CDT 2022
r...@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL 
amd64
FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git 
llvmorg-14.0.5-0-gc12386ae247c)

VT(efifb): resolution 640x480
CPU: Intel(R) Xeon(R) CPU   X5660  @ 2.80GHz (2793.07-MHz 
K8-class CPU)

  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
  
Features=0xbfebfbff
  
Features2=0x29ee3ff

  AMD Features=0x2c100800
  AMD Features2=0x1
  Structured Extended Features3=0x9c00
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 137438953472 (131072 MB)
avail memory = 133789515776 (127591 MB)
CPU microcode: no matching update found
Event timer "LAPIC" quality 600
ACPI APIC Table: 
FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs
FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 hardware threads
random: unblocking device.
ioapic1: MADT APIC ID 1 != hw id 0
ioapic0  irqs 0-23
ioapic1  irqs 32-55
Launching APs: 1 8 7 5 2 12 15 17 14 20 3 18 13 4 19 10 22 11 6 9 16 23 
21

TCP_ratelimit: Is now initialized
TCP Hpts created 24 swi interrupt threads and bound 24 to NUMA domains
random: entropy device external interface
kbd1 at kbdmux0
acpi0: 
acpi0: Power Button (fixed)
apei0:  on acpi0
cpu0:  on acpi0
atrtc0:  port 0x70-0x7f irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.00s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0:  port 0x40-0x5f irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0:  iomem 0xfed0-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0:  port 0xcf8-0xcff on acpi0
pci0:  on pcib0
pcib1:  at device 1.0 on pci0
pci1:  on pcib1
pci1:  at device 0.0 (no driver attached)
pci1:  at device 0.1 (no driver attached)
pcib2:  at device 3.0 on pci0
pci2:  on pcib2
pci2:  at device 0.0 (no driver attached)
pci2:  at device 0.1 (no driver attached)
pcib3:  at device 4.0 on pci0
pci3:  on pcib3
mfi0:  port 0xfc00-0xfcff mem 
0xdf1bc000-0xdf1b,0xdf1c-0xdf1f irq 33 at device 0.0 on pci3

mfi0: Using MSI
mfi0: Megaraid SAS driver Ver 4.23
mfi0: FW MaxCmds = 1008, limiting to 128
mfi0: 55158 (717992596s/0x0020/info) - Shutdown command received from 
host
pcib4: mfi0: 55159 (boot + 33s/0x0020/info) - 
Firmware initialization started (PCI ID 0079/1000/1f17/1028)

mfi0: 55160 (boot + 33s/0x0020/info) - Firmware version 2.100.03-4651
 at device 5.0 on pci0
pci4: mfi0: 55161 (boot + 35s/0x0008/info) - Battery 
Present

mfi0: 55162 (boot + 35s/0x0020/info)

BOOT CRASH -- Current -CURRENT

2022-10-01 Thread Larry Rosenman 



❯ more info.11
Dump header from device: /dev/mfid0p3
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 126748815
  Blocksize: 512
  Compression: zstd
  Dumptime: 2022-10-01 21:26:40 -0500
  Hostname:
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 14.0-CURRENT #168 
ler/freebsd-main-changes-n258354-6cdd871ebc4: Sat Oct  1 21:13:01 CDT 
2022

r...@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL
  Panic String: page fault
  Dump Parity: 501115454
  Bounds: 11
  Dump Status: good

I do have source and debug stuff, BUT kgdb croaks on me.

I *CAN* give access to the machine.

the console backtrace showed something about the kld load of 
dependencies.




--
Larry Rosenman http://people.freebsd.org/~ler
Phone: +1 214-642-9640 E-Mail: l...@freebsd.org
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106