This problem is back. Oct 24 19:23:53 zipper kernel: nvme0: Resetting controller due to a timeout and possible hot unplug. Oct 24 19:23:53 zipper syslogd: last message repeated 1 times Oct 24 19:23:53 zipper kernel: nvme0: event="start" Oct 24 19:23:54 zipper kernel: nvme0: Waiting for reset to complete Oct 24 19:23:54 zipper kernel: nvme0: event="timed_out" Oct 24 19:23:54 zipper kernel: nvme0: Waiting for reset to complete Oct 24 19:23:54 zipper kernel: nvme0: failing queued i/o Oct 24 19:23:54 zipper kernel: nvme0: READ (02) sqid:8 cid:0 nsid:1 lba:1180280 len:72 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296104 len:16 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=120278 0 47 0 0 0 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296120 len:16 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296136 len:16 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296152 len:16 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296168 len:8 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296176 len:16 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 5, Retries exhausted Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296192 len:16 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296208 len:16 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296224 len:8 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296232 len:16 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: swap_pager: I/O error - pagein failed; blkno 147279,size 36864, error 5 Oct 24 19:23:54 zipper kernel: vm_fault: pager read error, pid 1916 (devd) Oct 24 19:23:54 zipper kernel: nda0 at nvme0 bus 0 scbus9 target 0 lun 1 Oct 24 19:23:54 zipper kernel: nda0: <INTEL SSDPEKNW512G8 002C BTNH940617WE512A> s/n BTNH940617WE512A detached Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e806768 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated Oct 24 19:23:54 zipper kernel: swap_pager: I/O error - pagein failed; blkno 1286155,size 8192, error 6 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e806778 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: vm_fault: pager read error, pid 1916 (devd) Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e806788 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e806798 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e8067a8 0 7 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e8067b0 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e8067c0 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e8067d0 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error
The machine never recovered until I did a reset. FreeBSD zipper.catspoiler.org 16.0-CURRENT FreeBSD 16.0-CURRENT #52 main-n290318-a3d1d4e88ae7: Sat Oct 24 10:54:51 PDT 2026 [email protected]:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 nda0 at nvme0 bus 0 scbus9 target 0 lun 1 nda0: <INTEL SSDPEKNW512G8 002C BTNH940617WE512A> nda0: Serial Number BTNH940617WE512A nda0: nvme version 1.3 nda0: 488386MB (1000215216 512 byte sectors) # nvmecontrol logpage -p 0x1 nvme0 Error Information Log ===================== No error entries found # nvmecontrol logpage -p 0x2 nvme0 SMART/Health Information Log ============================ Critical Warning State: 0x00 Available spare: 0 Temperature: 0 Device reliability: 0 Read only: 0 Volatile memory backup: 0 Temperature: 306 K, 32.85 C, 91.13 F Available spare: 100 Available spare threshold: 10 Percentage used: 11 Data units (512,000 byte) read: 45037975 Data units written: 80040521 Host read commands: 2359001540 Host write commands: 1489208174 Controller busy time (minutes): 92832 Power cycles: 131 Power on hours: 42510 Unsafe shutdowns: 16 Media errors: 0 No. error info log entries: 0 Warning Temp Composite Time: 0 Error Temp Composite Time: 0 Temperature 1 Transition Count: 5231 Temperature 2 Transition Count: 0 Total Time For Temperature 1: 41213 Total Time For Temperature 2: 0 The Temperature 1 Count and Time are identical to the values from 2024 (see below), now that the drive has a finned heatsink. In both cases, the failures have occurred at low ambient temperatures. Yesterday's failure happened with the room at about 66 F. I had no problems under the same load in July with the room at about 85 F. On 13 Feb, Don Lewis wrote: > On 12 Feb, Warner Losh wrote: >> On Mon, Feb 12, 2024 at 9:15 PM Don Lewis <[email protected]> wrote: >> >>> On 12 Feb, Maxim Sobolev wrote: >>> > Might be an overheating. Today's nvme drives are notoriously flaky if you >>> > run them without proper heat sink attached to it. >>> >>> I don't think it is a thermal problem. According to the drive health >>> page, the device temperature has never reached Temperature 2, whatever >>> that is. The room temperature is around 65F. The system was stable >>> last summer when the room temperature spent a lot of time in the 80-85F >>> range. The device temperature depends a lot on the I/O rate, and the >>> last panic happened when the I/O rate had been below 40tps for quite a >>> while. >>> >> >> It did reach temperature 1, though. That's the 'Warning this drive is too >> hot' temperature. It has spent 41213 minutes of your 19297 hours of up >> time, or an average of 2 minutes per hour. That's too much. Temperature >> 2 is critical error: we are about to shut down completely due to it >> being too hot. It's only a couple degrees below hardware power off >> due to temperature in many drives. Some really cheap ones don't really >> implement it at all. On my card with the bad heat sink, Warning temp is >> 70C while critical is 75C while IIRC thermal shutdown is 78C or 80C. >> >> I don't think we report these values in nvmecontrol identify. But you can >> do a raw dump with -x look at bytes 266:267 for warning and 268:269 >> for critical. >> >> In contrast, the few dozen drives that I have, all of which have been >> abused in various ways, And only one of them has any heat issues, >> and that one is an engineering special / sample with what I think is >> a damaged heat sink. If your card has no heat sink, this could well >> be what's going on. >> >> This panic means "the nvme card lost its mind and stopped talking >> to the host". Its status registers read 0xff's, which means that the card >> isn't decoding bus signals. Usually this means that the firmware on the >> card has faulted and rebooted. If the card is overheating, then this could >> well be what's happening. >> >> There's a tiny chance that this could be something more exotic, >> but my money is on hardware gone bad after 2 years of service. I don't think >> this is 'wear out' of the NAND (it's only 15TB written, but it could be if >> this >> drive is really really crappy nand: first generation QLC maybe, but it seems >> too new). It might also be a connector problem that's developed over time. >> There might be a few other things too, but I don't think this is a U.2 drive >> with funky cables. > > The system was probably idle the majority of those two years of power on > time. > > It's one of these: > https://www.techpowerup.com/ssd-specs/intel-660p-512-gb.d437 > I've seen comments that these generally don't need cooling. > > I just ordered a heatsink with some nice big fins, but it will take a > week or more to arrive. > >> >>> > On Mon, Feb 12, 2024, 4:28 PM Don Lewis <[email protected]> wrote: >>> > >>> >> I just upgraded my package build machine to: >>> >> FreeBSD 15.0-CURRENT #110 main-n268161-4015c064200e >>> >> from: >>> >> FreeBSD 15.0-CURRENT #106 main-n265953-a5ed6a815e38 >>> >> and I've had two nvme-triggered panics in the last day. >>> >> >>> >> nvme is being used for swap and L2ARC. I'm not able to get a crash >>> >> dump, probably because the nvme device has gone away and I get an error >>> >> about not having a dump device. It looks like a low-memory panic >>> >> because free memory is low and zfs is calling malloc(). >>> >> >>> >> This shows up in the log leading up to the panic: >>> >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a >>> >> timeout a >>> >> nd possible hot unplug. >>> >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times >>> >> Feb 12 10:07:41 zipper kernel: nvme0: resetting controller >>> >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a >>> >> timeout a >>> >> nd possible hot unplug. >>> >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times >>> >> Feb 12 10:07:41 zipper kernel: nvme0: Waiting for reset to complete >>> >> Feb 12 10:07:41 zipper syslogd: last message repeated 2 times >>> >> Feb 12 10:07:41 zipper kernel: nvme0: failing queued i/o >>> >> Feb 12 10:07:41 zipper kernel: nvme0: Failed controller, stopping >>> watchdog >>> >> ti >>> >> meout. >>> >> >>> >> The device looks healthy to me: >>> >> SMART/Health Information Log >>> >> ============================ >>> >> Critical Warning State: 0x00 >>> >> Available spare: 0 >>> >> Temperature: 0 >>> >> Device reliability: 0 >>> >> Read only: 0 >>> >> Volatile memory backup: 0 >>> >> Temperature: 312 K, 38.85 C, 101.93 F >>> >> Available spare: 100 >>> >> Available spare threshold: 10 >>> >> Percentage used: 3 >>> >> Data units (512,000 byte) read: 5761183 >>> >> Data units written: 29911502 >>> >> Host read commands: 471921188 >>> >> Host write commands: 605394753 >>> >> Controller busy time (minutes): 32359 >>> >> Power cycles: 110 >>> >> Power on hours: 19297 >>> >> Unsafe shutdowns: 14 >>> >> Media errors: 0 >>> >> No. error info log entries: 0 >>> >> Warning Temp Composite Time: 0 >>> >> Error Temp Composite Time: 0 >>> >> Temperature 1 Transition Count: 5231 >>> >> Temperature 2 Transition Count: 0 >>> >> Total Time For Temperature 1: 41213 >>> >> Total Time For Temperature 2: 0 >>> >> >>> >> >>> >> >>> >>> >>> > >
