Re: top-of-tree alpha kernel panics during boot
Dag-Erling Smorgrav wrote: > Andrew Gallatin <[EMAIL PROTECTED]> writes: > > Damn. I'm sorry then, I think I've done all I can to try to duplicate > > it. Would you mind doing a binary search to find out when your problem > > started? > > I'd rather not, the machine is essential to my home network and > downtime affects not only me but also my SO. And the segfaults seem > to have gone away as well... I am now running a ToT kernel w/o pcm, > and it's already gone halfway through a buildworld without a single > segfault. If all you are replacing is the kernel, you should be able to do a single test in an hour. If you are willing to spend 8 hours on this, then you should be able to go back 32 days, for a 1 day increment. If you are willing to spend 4 hours on it, and you pick a 4 day increment, you can go back 64 days (2 months). Do you have a bounding range? This may not take that much time, or you can put your SO in a bubble bath on two occasions (8-)), etc., and get it solved. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Andrew Gallatin <[EMAIL PROTECTED]> writes: > Damn. I'm sorry then, I think I've done all I can to try to duplicate > it. Would you mind doing a binary search to find out when your problem > started? I'd rather not, the machine is essential to my home network and downtime affects not only me but also my SO. And the segfaults seem to have gone away as well... I am now running a ToT kernel w/o pcm, and it's already gone halfway through a buildworld without a single segfault. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Dag-Erling Smorgrav writes: > Andrew Gallatin <[EMAIL PROTECTED]> writes: > > Do you preload any/all of the things you've marked as klds? > > No... Damn. I'm sorry then, I think I've done all I can to try to duplicate it. Would you mind doing a binary search to find out when your problem started? Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Andrew Gallatin <[EMAIL PROTECTED]> writes: > Do you preload any/all of the things you've marked as klds? No... DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
On Wed, Feb 19, 2003 at 08:42:40PM +0100, Dag-Erling Smorgrav wrote: > 3) a fresh kernel without pcm boots but exhibits the same symptoms > kris reported, i.e. programs segfaulting for no apparent reason; > if / when they produce a core file, it is corrupted and useless > for debugging. I suspect a problem in the I/O system, possibly > similar to the one tegge discovered and fixed last week. My problem (seen on the alpha package cluster in the chroots used to build packages) may have been a bad userland. I used a snapshot from snapshots.jp.freebsd.org to populate the chroots and saw tons of sig11s during the package builds, but when I built a world myself and used that the problems seem to have ceased. I haven't updated the kernel on those machines. Kris msg52746/pgp0.pgp Description: PGP signature
Re: top-of-tree alpha kernel panics during boot
Dag-Erling Smorgrav writes: > #options CD9660 #kld Do you preload any/all of the things you've marked as klds? I've tried to duplicate your crash on my xp1000. I've used your kernel config file. I've reduced my memory size to 254MB. I've got roughly similar hardware (w/the exception of the chipset and cpu). The only thing I've not done is to preload any klds. I don't doubt there's a problem, but I'm having a hell of a time finding it :-( Drew Copyright (c) 1992-2003 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.0-CURRENT #2: Wed Feb 19 20:34:33 EST 2003 gallatin@monet:/usr/tmp/sys/alpha/compile/DES Preloaded elf kernel "/kernel.test" at 0xfc762000. ST6600 COMPAQ Professional Workstation XP1000, 500MHz 8192 byte page size, 1 processor. CPU: EV6 (21264) major=8 minor=3 extensions=0x303 OSF PAL rev: 0x1001600020153 real memory = 266338304 (254 MB) avail memory = 251936768 (240 MB) tsunami0: <21271 Core Logic chipset> pcib0: <21271 PCI host bus adapter> on tsunami0 pci0: on pcib0 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x1-0x1000f,0x3f4-0x3f7,0x1f0-0x1f7 irq 238 at device 7.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ata0: interrupting at ISA irq 14 ata1: at 0x170 irq 15 on atapci0 ata1: interrupting at ISA irq 15 atapci1: port 0x374-0x377,0x170-0x177 mem 0x108-0x108 irq 239 at device 7.2 on pci0 atapci1: Busmastering DMA not configured ohci0: mem 0x109-0x1090fff irq 234 at device 7.3 on pci0 ohci0: interrupting at ISA irq 10 usb0: OHCI version 1.0, legacy support usb0: on ohci0 usb0: USB revision 1.0 uhub0: (0x1080) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered pci0: at device 13.0 (no driver attached) pcib1: <21271 PCI host bus adapter> on tsunami0 pci1: on pcib1 dc0: port 0x1-0x1007f mem 0x1051000-0x10513ff irq 45 at device 3.0 on pci1 dc0: Ethernet address: 00:00:f8:71:ae:00 miibus0: on dc0 dcphy0: on miibus0 dcphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto dc0: interrupting at TSUNAMI irq 45 isp0: port 0x1000-0x10ff mem 0x105-0x1050fff irq 47 at device 6.0 on pci1 isp0: interrupting at TSUNAMI irq 47 pcib2: at device 8.0 on pci1 pci2: on pcib2 atkbdc0: at port 0x64,0x60 on isa0 atkbd0: irq 1 on atkbdc0 atkbd0: interrupting at ISA irq 1 fdc0: at port 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on isa0 fdc0: interrupting at ISA irq 6 mcclock0: at port 0x70-0x71 on isa0 ppc0: at port 0x3bc-0x3c3 irq 7 on isa0 ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode lpt0: on ppbus0 lpt0: Polled port ppi0: on ppbus0 ppc0: interrupting at ISA irq 7 sio0 at port 0x3f8-0x3ff irq 4 on isa0 sio0: type 16550A, console sio0: interrupting at ISA irq 4 sio1 at port 0x2f8-0x2ff irq 3 on isa0 sio1: type 16550A sio1: interrupting at ISA irq 3 sbc0: at port 0x220-0x22f irq 5 drq 1 on isa0 sbc0: interrupting at ISA irq 5 pcm0: on sbc0 Timecounter "i8254" frequency 1193182 Hz Timecounter "alpha" frequency 53025 Hz Timecounters tick every 0.976 msec ad0: 9787MB [19885/16/63] at ata0-slave WDMA2 acd0: CDROM at ata0-master PIO4 Waiting 2 seconds for SCSI devices to settle da0 at isp0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device da0: 40.000MB/s transfers (20.000MHz, offset 8, 16bit), Tagged Queueing Enabled da0: 4091MB (8380080 512 byte sectors: 255H 63S/T 521C) da1 at isp0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device da1: 40.000MB/s transfers (20.000MHz, offset 8, 16bit), Tagged Queueing Enabled da1: 8683MB (17783240 512 byte sectors: 255H 63S/T 1106C) da2 at isp0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-2 device da2: 40.000MB/s transfers (20.000MHz, offset 8, 16bit), Tagged Queueing Enabled da2: 8678MB (17773524 512 byte sectors: 255H 63S/T 1106C) Mounting root from ufs:/dev/da1a To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Andrew Gallatin <[EMAIL PROTECTED]> writes: > Can you post your kernel config please, along with with, if any, > CPUTYPE you have set in make.conf and a description of your machine > (mem size in particular)? Digital Personal WorkStation 600au, 598MHz 8192 byte page size, 1 processor. CPU: EV56 (21164A) major=7 minor=0 extensions=0x1 OSF PAL rev: 0x100020116 real memory = 266493952 (254 MB) avail memory = 251985920 (240 MB) des@dsa ~% grep CPU /etc/make.conf CPUTYPE ?= ev56 Full dmesg (from a working kernel built from Jan 9th sources) and kernel config attached. Note that: 1) the problem disappears if I remove the pcm driver from the kernel (yet I don't think it's directly at fault since the panic happens when the ess driver is about to be initialized) 2) the problem also disappears if I enable KTR (see commented-out entries in config file) 3) a fresh kernel without pcm boots but exhibits the same symptoms kris reported, i.e. programs segfaulting for no apparent reason; if / when they produce a core file, it is corrupted and useless for debugging. I suspect a problem in the I/O system, possibly similar to the one tegge discovered and fixed last week. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] Copyright (c) 1992-2003 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.0-CURRENT #30: Thu Jan 9 13:58:33 CET 2003 [EMAIL PROTECTED]:/usr/src/sys/alpha/compile/DSA Preloaded elf kernel "/boot/kernel.ok/kernel" at 0xfc77. Digital Personal Workstation (Miata) Digital Personal WorkStation 600au, 598MHz 8192 byte page size, 1 processor. CPU: EV56 (21164A) major=7 minor=0 extensions=0x1 OSF PAL rev: 0x100020116 real memory = 266493952 (254 MB) avail memory = 251985920 (240 MB) Initializing GEOMetry subsystem cia0: <2117x Core Logic chipset> cia0: Pyxis, pass 1 cia0: extended capabilities: 1 pcib0: <2117x PCI host bus adapter> on cia0 pci0: on pcib0 dc0: port 0x9000-0x907f mem 0x80151000-0x8015107f irq 0 at device 3.0 on pci0 dc0: interrupting at CIA irq 0 dc0: Ethernet address: 08:00:2b:86:88:55 miibus0: on dc0 nsphy0: on miibus0 nsphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x90a0-0x90af,0x3f4-0x3f7,0x1f0-0x1f7 irq 238 at device 7.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ata0: interrupting at ISA irq 14 ata1: at 0x170 irq 15 on atapci0 ata1: interrupting at ISA irq 15 atapci1: port 0x374-0x377,0x170-0x177 mem 0x8014-0x8014 irq 239 at device 7.2 on pci0 atapci1: Busmastering DMA not configured ohci0: mem 0x8015-0x80150fff irq 234 at device 7.3 on pci0 ohci0: interrupting at ISA irq 10 usb0: OHCI version 1.0, legacy support usb0: on ohci0 usb0: USB revision 1.0 uhub0: (0x1080) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered pcib1: at device 20.0 on pci0 pci1: on pcib1 isp0: port 0x8000-0x80ff mem 0x8001-0x80010fff irq 3 at device 4.0 on pci1 isp0: interrupting at CIA irq 3 atkbdc0: at port 0x64,0x60 on isa0 atkbd0: irq 1 on atkbdc0 atkbd0: interrupting at ISA irq 1 fdc0: at port 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on isa0 fdc0: interrupting at ISA irq 6 mcclock0: at port 0x70-0x71 on isa0 ppc0: at port 0x3bc-0x3c3 irq 7 on isa0 ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode lpt0: on ppbus0 lpt0: Polled port ppi0: on ppbus0 ppc0: interrupting at ISA irq 7 sio0 at port 0x3f8-0x3ff irq 4 on isa0 sio0: type 16550A, console sio0: interrupting at ISA irq 4 sio1 at port 0x2f8-0x2ff irq 3 on isa0 sio1: type 16550A sio1: interrupting at ISA irq 3 sbc0: at port 0x220-0x22f irq 5 drq 1 on isa0 sbc0: interrupting at ISA irq 5 pcm0: on sbc0 Timecounter "i8254" frequency 1193182 Hz Timecounter "alpha" frequency 599860139 Hz Timecounters tick every 0.976 msec Waiting 2 seconds for SCSI devices to settle cd0 at isp0 bus 0 target 6 lun 0 cd0: Removable CD-ROM SCSI-2 device cd0: 4.237MB/s transfers (4.237MHz, offset 8) cd0: Attempt to query device size failed: NOT READY, Medium not present da1 at isp0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device da1: 40.000MB/s transfers (20.000MHz, offset 8, 16bit), Tagged Queueing Enabled da1: 4341MB (8890760 512 byte sectors: 255H 63S/T 553C) da0 at isp0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 40.000MB/s transfers (20.000MHz, offset 8, 16bit), Tagged Queueing Enabled da0: 8748MB (17916240 512 byte sectors: 255H 63S/T 1115C) Mounting root from ufs:/dev/da0a # Kernel configuration for dsa.thinksec.com machine alpha cpu EV5 options DEC_ST550 ident DSA maxusers0 makeoptions DEBUG=-g options DDB #optionsDEBUG_VFS_LOCKS options INVARIANT_SUPPORT, INVARIANTS #optionsWITNESS, WITNESS_SKIPSPIN #option
Re: top-of-tree alpha kernel panics during boot
Can you post your kernel config please, along with with, if any, CPUTYPE you have set in make.conf and a description of your machine (mem size in particular)? I'm unable to reproduce the boot panic with a GENERIC as of today. (rest of machine is from Nov 1st). Rebuilding without CPUTYPE now.. Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Dag-Erling Smorgrav writes: > Andrew Gallatin <[EMAIL PROTECTED]> writes: > > You might be able to get some idea of what's happening by enabling KTR > > and tracing everything, then dumping the trace buffer at your > > breakpoint. > > Of course, the KTR-enabled kernel fails to crash. > > *sigh* > > but I bet it'll segfault like nobody's business if I let it boot to > multiuser, so I'm stuck with my Jan 9 kernel. Ugh. About the only thing I can suggest is to pepper the startup code with Debugger() calls, and do something of a binary search. Having just done that to track some vm faults before vm is setup, I know that's about as much fun as eating dirt. But I can't think of a better idea. Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Andrew Gallatin <[EMAIL PROTECTED]> writes: > You might be able to get some idea of what's happening by enabling KTR > and tracing everything, then dumping the trace buffer at your > breakpoint. Of course, the KTR-enabled kernel fails to crash. *sigh* but I bet it'll segfault like nobody's business if I let it boot to multiuser, so I'm stuck with my Jan 9 kernel. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Andrew Gallatin <[EMAIL PROTECTED]> writes: > You might be able to get some idea of what's happening by enabling KTR > and tracing everything, then dumping the trace buffer at your > breakpoint. Hmm, how do I dump the KTR buffer from DDB? I've done it before, but it's ages ago and I don't remember how... DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Dag-Erling Smorgrav writes: > Andrew Gallatin <[EMAIL PROTECTED]> writes: > > Can you look at the registers and match $ra with a line number using > > addr2line or gdb? (sorry, forgot if ddb can even look at registers) > > Uh, I know *where* it stops since I added the call to Debugger() in > the first place. The problem is figuring out where it was called > from, three or four levels up. > Yeah, if its 3 levels, you're screwed. You might be able to get some idea of what's happening by enabling KTR and tracing everything, then dumping the trace buffer at your breakpoint. BTW, thanks for persisting in this. I appreciate it. Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Andrew Gallatin <[EMAIL PROTECTED]> writes: > Can you look at the registers and match $ra with a line number using > addr2line or gdb? (sorry, forgot if ddb can even look at registers) Uh, I know *where* it stops since I added the call to Debugger() in the first place. The problem is figuring out where it was called from, three or four levels up. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Dag-Erling Smorgrav writes: > Dag-Erling Smorgrav <[EMAIL PROTECTED]> writes: > > Any suggestions as to how I can figure out who used that block of > > memory before it was allocated to the ess driver? > > I threw in a call to Debugger(), but... > > mtrash_dtor(0xfc7b6000, 8192, 0) > here's the culprit! > Stopped at 0xfc577578: zapnot v0,#0xf,v0 > db> trace > db> c > > no backtrace :( > Can you look at the registers and match $ra with a line number using addr2line or gdb? (sorry, forgot if ddb can even look at registers) Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Dag-Erling Smorgrav <[EMAIL PROTECTED]> writes: > Any suggestions as to how I can figure out who used that block of > memory before it was allocated to the ess driver? I threw in a call to Debugger(), but... mtrash_dtor(0xfc7b6000, 8192, 0) here's the culprit! Stopped at 0xfc577578: zapnot v0,#0xf,v0 db> trace db> c no backtrace :( DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
OK, I've played around with my kernel trying to figure out what causes this. The one thing I know is that it disappears if I leave pcm out of my kernel, but I don't think pcm is the culprit. Since the address at fault is constant, I've added code to mtrash_[cd]tor() which calls db_print_backtrace() every time we hit that address. I also added code to module_regiser_init() to identify the module being processed: > module_register_init: registering 'isa/esscontrol' > mtrash_ctor(0xfc7aab80, 64, 0) > mtrash_ctor(0xfc7a8ba0, 32, 0) > mtrash_ctor(0xfc7b6000, 8192, 0) > db_print_backtrace() at db_print_backtrace+0x18 > mtrash_ctor() at mtrash_ctor+0x68 > uma_zalloc_arg() at uma_zalloc_arg+0x160 > malloc() at malloc+0x94 > kobj_class_compile() at kobj_class_compile+0x2c > devclass_add_driver() at devclass_add_driver+0x64 > driver_module_handler() at driver_module_handler+0xb4 > module_register_init() at module_register_init+0xc0 > mi_startup() at mi_startup+0x144 > locorestart() at locorestart+0x64 > --- root of call graph --- > Memory modified after free 0xfc7b6000(8184) > panic: Most recently used by none > > panic > Stopped at Debugger+0x38: zapnot v0,#0xf,v0 The problem is that this doesn't really tell me what I want (i.e. who held that block before it was allocated to isa/esscontrol). All I know about that is: mtrash_dtor(0xfc7b6000, 8192, 0) because it seems that db_print_backtrace() is a nop at that point. Any suggestions as to how I can figure out who used that block of memory before it was allocated to the ess driver? DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Dag-Erling Smorgrav writes: > Andrew Gallatin <[EMAIL PROTECTED]> writes: > > This is a UP box right? > > Yes, a PWS 600au. > OK, good.. ;) There had been SMP problems with the UMA debug code a long time ago. Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Andrew Gallatin <[EMAIL PROTECTED]> writes: > This is a UP box right? Yes, a PWS 600au. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: top-of-tree alpha kernel panics during boot
Dag-Erling Smorgrav writes: > Memory modified after free 0xfc7b6000(8184) > panic: Most recently used by none It might be interesting to add some printfs to see what the value is currently, vs what was expected, and where in the zone the modification happened.. This is a UP box right? Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
