Re: panic - ffs_write - AMD64/7.4 to current
On 2024/08/24 12:13, Martin Pieuchot wrote: > Hugh, > > If you can reproduce this easily, please send a new panic with the > outputs of: > - show uvm > - show bcstats > - And the traces of all running processes... In the two reports below we > only have the trace of pax(1) which is running on CPU2. > > The two panics are due to corruptions of two different global data > structures related to buffers: the tree of pages and the tree of buffers. > > In both cases it happens when the buffer cache reaches low DMA watermark > and tries to flip a buffer high. The fact that global data structures > are corrupted and the given buffer cannot be found tends to indicate > there is a race. And this is coherent with the use of pax | nc which > are currently running on two different CPUs. > > I fear there's a sleeping point somewhere, we could try converting the > splbio() to a mutex which should help. > > On 24/08/24(Sat) 01:11, Hugh Graham wrote: > > On Fri, Aug 23, 2024 at 01:52:52PM -0600, Bob Beck wrote: > > > My immediate suspicion would also fall there. Nothing in here has > > > recently changed. > > > > > > You should probably share this with a wider audience, like bugs@ or tech@ > > > instead of just > > > Mailing individuals. > > > > Apologies for the lack of process. I am only barely awake after a > > long slumber. > > > > It ran all day, but I did manage to reproduce the crash on 7.4, > > so that absolves a whole bunch of "recent" changes. > > > > Also, as yet, I have only the single machine for testing and > > can't exclude hardware. If anyone wants to make an independent > > confirmation, sending a ports tree with plenty of packages and > > distfiles might be a successful recipe. > > > > pax -w ports | network | pax -r > > > > Where the receiver's network media is forced to 10BaseT, or the > > sending machine is just that slow. My latest crash was near the > > 25GB mark, but this varies greatly and is usually sooner. I > > will confirm this recipe when I see my next crash. > > > > /Hugh > > > > >> OpenBSD/amd64 BOOTX64 3.65 > > boot> boot bsd.mp.74.dist -s > > booting hd0a:bsd.mp.74.dist: 17249612+4142096+368672+0+1241088 > > [1340407+128+1321080+1013316]=0x1973738 > > entry point at 0x1001000 > > [ using 3675960 bytes of bsd ELF symbol table ] > > Copyright (c) 1982, 1986, 1989, 1991, 1993 > > The Regents of the University of California. All rights reserved. > > Copyright (c) 1995-2023 OpenBSD. All rights reserved. > > https://www.OpenBSD.org > > > > OpenBSD 7.4 (GENERIC.MP) #1397: Tue Oct 10 09:02:37 MDT 2023 > > [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP > > real mem = 33551818752 (31997MB) > > ... > > panic: kernel diagnostic assertion "tpg != NULL" failed: file > > "/usr/src/sys/uvm > > /uvm_page.c", line 855 > > Stopped at db_enter+0x14: popq%rbp > > TIDPIDUID PRFLAGS PFLAGS CPU COMMAND > > * 1154 46918 00x13 02 pax > > 401298 12308 00x13 03 nc > > 404052 38135 0 0x14000 0x2001 softnet0 > > db_enter() at db_enter+0x14 > > panic(820a9e1f) at panic+0xc3 > > __assert(82122f8e,8207775c,357,8215caa2) at > > __assert+0x > > 29 > > uvm_pagerealloc_multi(fd8741fa1218,40,4000,22,8250d8d0) > > at u > > vm_pagerealloc_multi+0x2f8 > > buf_realloc_pages(fd8741fa1158,8250d8d0,2) at > > buf_realloc_pages+0xb > > f > > buf_flip_high(fd8741fa1158) at buf_flip_high+0x7e > > bufcache_recover_dmapages(0,4) at bufcache_recover_dmapages+0x12b > > buf_get(fd873280ab58,3be4,4000) at buf_get+0xcb > > getblk(fd873280ab58,3be4,4000,0,) at getblk+0x71 > > ffs2_balloc(fd872b47be18,ef9,2d,fd880dad9ea0,1,8000443a11a8) > > at > > ffs2_balloc+0xeef > > ffs_write(8000443a1228) at ffs_write+0x229 > > VOP_WRITE(fd873280ab58,8000443a1388,1,fd880dad9ea0) at > > VOP_WRITE+0x > > 45 > > vn_write(fd8718901708,8000443a1388,0) at vn_write+0xcc > > dofilewritev(80004436e2b0,6,8000443a1388,0,8000443a1460) at > > dofilew > > ritev+0x151 > > end trace frame: 0x8000443a13f0, count: 0 > > https://www.openbsd.org/ddb.html describes the minimum info required in bug > > reports. Insufficient info makes it difficult to find and fix bugs. > > ddb{2}> > > > > > > > > Very likely related to come of the changes being made in uvm. I've just been sent a photo from a crashed machine (not local to me - it's running 7.6-beta from Aug 19) with a trace which doesn't look entirely dissimilar to this first one from Hugh. It would have been idling at the time with X, mate, possibly chromium running but not actively used. Sadly I don't have any further information from DDB beyond what was on-screen, the machine was already rebooted so I can't get it now, so I'm afraid this is probably not all that a useful report.. Hand-retyped
Re: panic - ffs_write - AMD64/7.4 to current
Hugh, If you can reproduce this easily, please send a new panic with the outputs of: - show uvm - show bcstats - And the traces of all running processes... In the two reports below we only have the trace of pax(1) which is running on CPU2. The two panics are due to corruptions of two different global data structures related to buffers: the tree of pages and the tree of buffers. In both cases it happens when the buffer cache reaches low DMA watermark and tries to flip a buffer high. The fact that global data structures are corrupted and the given buffer cannot be found tends to indicate there is a race. And this is coherent with the use of pax | nc which are currently running on two different CPUs. I fear there's a sleeping point somewhere, we could try converting the splbio() to a mutex which should help. On 24/08/24(Sat) 01:11, Hugh Graham wrote: > On Fri, Aug 23, 2024 at 01:52:52PM -0600, Bob Beck wrote: > > My immediate suspicion would also fall there. Nothing in here has recently > > changed. > > > > You should probably share this with a wider audience, like bugs@ or tech@ > > instead of just > > Mailing individuals. > > Apologies for the lack of process. I am only barely awake after a > long slumber. > > It ran all day, but I did manage to reproduce the crash on 7.4, > so that absolves a whole bunch of "recent" changes. > > Also, as yet, I have only the single machine for testing and > can't exclude hardware. If anyone wants to make an independent > confirmation, sending a ports tree with plenty of packages and > distfiles might be a successful recipe. > > pax -w ports | network | pax -r > > Where the receiver's network media is forced to 10BaseT, or the > sending machine is just that slow. My latest crash was near the > 25GB mark, but this varies greatly and is usually sooner. I > will confirm this recipe when I see my next crash. > > /Hugh > > >> OpenBSD/amd64 BOOTX64 3.65 > boot> boot bsd.mp.74.dist -s > booting hd0a:bsd.mp.74.dist: 17249612+4142096+368672+0+1241088 > [1340407+128+1321080+1013316]=0x1973738 > entry point at 0x1001000 > [ using 3675960 bytes of bsd ELF symbol table ] > Copyright (c) 1982, 1986, 1989, 1991, 1993 > The Regents of the University of California. All rights reserved. > Copyright (c) 1995-2023 OpenBSD. All rights reserved. https://www.OpenBSD.org > > OpenBSD 7.4 (GENERIC.MP) #1397: Tue Oct 10 09:02:37 MDT 2023 > [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 33551818752 (31997MB) > ... > panic: kernel diagnostic assertion "tpg != NULL" failed: file > "/usr/src/sys/uvm > /uvm_page.c", line 855 > Stopped at db_enter+0x14: popq%rbp > TIDPIDUID PRFLAGS PFLAGS CPU COMMAND > * 1154 46918 00x13 02 pax > 401298 12308 00x13 03 nc > 404052 38135 0 0x14000 0x2001 softnet0 > db_enter() at db_enter+0x14 > panic(820a9e1f) at panic+0xc3 > __assert(82122f8e,8207775c,357,8215caa2) at > __assert+0x > 29 > uvm_pagerealloc_multi(fd8741fa1218,40,4000,22,8250d8d0) > at u > vm_pagerealloc_multi+0x2f8 > buf_realloc_pages(fd8741fa1158,8250d8d0,2) at > buf_realloc_pages+0xb > f > buf_flip_high(fd8741fa1158) at buf_flip_high+0x7e > bufcache_recover_dmapages(0,4) at bufcache_recover_dmapages+0x12b > buf_get(fd873280ab58,3be4,4000) at buf_get+0xcb > getblk(fd873280ab58,3be4,4000,0,) at getblk+0x71 > ffs2_balloc(fd872b47be18,ef9,2d,fd880dad9ea0,1,8000443a11a8) > at > ffs2_balloc+0xeef > ffs_write(8000443a1228) at ffs_write+0x229 > VOP_WRITE(fd873280ab58,8000443a1388,1,fd880dad9ea0) at > VOP_WRITE+0x > 45 > vn_write(fd8718901708,8000443a1388,0) at vn_write+0xcc > dofilewritev(80004436e2b0,6,8000443a1388,0,8000443a1460) at > dofilew > ritev+0x151 > end trace frame: 0x8000443a13f0, count: 0 > https://www.openbsd.org/ddb.html describes the minimum info required in bug > reports. Insufficient info makes it difficult to find and fix bugs. > ddb{2}> > > > > > Very likely related to come of the changes being made in uvm. > > > > > On Aug 23, 2024, at 1:25???PM, Kenneth Westerback > > > wrote: > > > > > > And of course it is beck@, not bob@. :-) > > > > > > Ken > > > > > > On Fri, Aug 23, 2024, 15:01 Kenneth Westerback > > > wrote: > > > Hmmm. Not sure I can see anything there I might have worked on recently. > > > My immediate suspicion falls on recent reports about and subsequent > > > repairs to amd64 low memory area memory handling. You might want to loop > > > in mpi@ and others who made changes to uvm recently. > > > > > > Happy to try and reproduce given sufficiently detailed instructions. Note > > > that I have never used pax or cpio and all my network gear and boxes are > > > 1GB devices. :-) > > > > > > A particularly interesting test
