Re: nvme controller reset failures on recent -CURRENT
Hi all, > Am 13.02.2024 um 20:56 schrieb Pete Wright : > 1. M.2 nvme really does need proper cooling, much more so than traditional > SATA/SAS/SCSI drives. I recently found a tool named "Scrutiny" that presents a nice dashboard of all your disk devices and their SMART data including crucial points like temperature. Pros: Open source Nice web UI Uses smartmontools to gather the data, not reinventing the wheel Agents that can be called from cron jobs for many OSes including FreeBSD Alerting via a variety of communication channels Cons: Central hub best run on Linux plus docker compose No authentication whatsoever, so strictly internal use No grouping or any organisation of systems so does not scale beyond tens of servers I found a couple of problematic HDDs and SSDs right after deploying it which regular SMART tests overlooked. https://github.com/AnalogJ/scrutiny Look for the Hub/Spoke deployment if you are willing to use e.g. a Linux VM to run the tool, then point your FreeBSD systems at that. It probably can be deployed strictly on FreeBSD, too, using the manual installation instructions. HTH, kind regards, Patrick
Re: nvme controller reset failures on recent -CURRENT
I had issues with a nvme drive in an intel nuc. When I asked freebsd-hackers, overheating was the first guess: https://lists.freebsd.org/pipermail/freebsd-hackers/2018-May/052783.html I blew the dust out of the fan assembly and changed the bios fan settings to be more aggressive and the system has been rock solid since. Craig
Re: nvme controller reset failures on recent -CURRENT
There's a tiny chance that this could be something more exotic, but my money is on hardware gone bad after 2 years of service. I don't think this is 'wear out' of the NAND (it's only 15TB written, but it could be if this drive is really really crappy nand: first generation QLC maybe, but it seems too new). It might also be a connector problem that's developed over time. There might be a few other things too, but I don't think this is a U.2 drive with funky cables. The system was probably idle the majority of those two years of power on time. It's one of these: https://www.techpowerup.com/ssd-specs/intel-660p-512-gb.d437 I've seen comments that these generally don't need cooling. I just ordered a heatsink with some nice big fins, but it will take a week or more to arrive. just wanted to add another data-point to this discussion. i had a crucial NVME drive on my workstation that recently was showing similar problems. after much debugging i came to the same conclusion that it was getting too hot. i went ahead an purchased a Sabrent NVME drive that came with a heat sink. i've also starting making much more use of my workstation (and the disk subsystem) and have had zero issues. so lessons learnt: 1. M.2 nvme really does need proper cooling, much more so than traditional SATA/SAS/SCSI drives. 2. not all vendors do a great job reporting the health of devices -pete -- Pete Wright p...@nomadlogic.org
Re: nvme controller reset failures on recent -CURRENT
On 12 Feb, Warner Losh wrote: > On Mon, Feb 12, 2024 at 9:15 PM Don Lewis wrote: > >> On 12 Feb, Maxim Sobolev wrote: >> > Might be an overheating. Today's nvme drives are notoriously flaky if you >> > run them without proper heat sink attached to it. >> >> I don't think it is a thermal problem. According to the drive health >> page, the device temperature has never reached Temperature 2, whatever >> that is. The room temperature is around 65F. The system was stable >> last summer when the room temperature spent a lot of time in the 80-85F >> range. The device temperature depends a lot on the I/O rate, and the >> last panic happened when the I/O rate had been below 40tps for quite a >> while. >> > > It did reach temperature 1, though. That's the 'Warning this drive is too > hot' temperature. It has spent 41213 minutes of your 19297 hours of up > time, or an average of 2 minutes per hour. That's too much. Temperature > 2 is critical error: we are about to shut down completely due to it > being too hot. It's only a couple degrees below hardware power off > due to temperature in many drives. Some really cheap ones don't really > implement it at all. On my card with the bad heat sink, Warning temp is > 70C while critical is 75C while IIRC thermal shutdown is 78C or 80C. > > I don't think we report these values in nvmecontrol identify. But you can > do a raw dump with -x look at bytes 266:267 for warning and 268:269 > for critical. > > In contrast, the few dozen drives that I have, all of which have been > abused in various ways, And only one of them has any heat issues, > and that one is an engineering special / sample with what I think is > a damaged heat sink. If your card has no heat sink, this could well > be what's going on. > > This panic means "the nvme card lost its mind and stopped talking > to the host". Its status registers read 0xff's, which means that the card > isn't decoding bus signals. Usually this means that the firmware on the > card has faulted and rebooted. If the card is overheating, then this could > well be what's happening. > > There's a tiny chance that this could be something more exotic, > but my money is on hardware gone bad after 2 years of service. I don't think > this is 'wear out' of the NAND (it's only 15TB written, but it could be if > this > drive is really really crappy nand: first generation QLC maybe, but it seems > too new). It might also be a connector problem that's developed over time. > There might be a few other things too, but I don't think this is a U.2 drive > with funky cables. The system was probably idle the majority of those two years of power on time. It's one of these: https://www.techpowerup.com/ssd-specs/intel-660p-512-gb.d437 I've seen comments that these generally don't need cooling. I just ordered a heatsink with some nice big fins, but it will take a week or more to arrive. > >> > On Mon, Feb 12, 2024, 4:28 PM Don Lewis wrote: >> > >> >> I just upgraded my package build machine to: >> >> FreeBSD 15.0-CURRENT #110 main-n268161-4015c064200e >> >> from: >> >> FreeBSD 15.0-CURRENT #106 main-n265953-a5ed6a815e38 >> >> and I've had two nvme-triggered panics in the last day. >> >> >> >> nvme is being used for swap and L2ARC. I'm not able to get a crash >> >> dump, probably because the nvme device has gone away and I get an error >> >> about not having a dump device. It looks like a low-memory panic >> >> because free memory is low and zfs is calling malloc(). >> >> >> >> This shows up in the log leading up to the panic: >> >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a >> >> timeout a >> >> nd possible hot unplug. >> >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times >> >> Feb 12 10:07:41 zipper kernel: nvme0: resetting controller >> >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a >> >> timeout a >> >> nd possible hot unplug. >> >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times >> >> Feb 12 10:07:41 zipper kernel: nvme0: Waiting for reset to complete >> >> Feb 12 10:07:41 zipper syslogd: last message repeated 2 times >> >> Feb 12 10:07:41 zipper kernel: nvme0: failing queued i/o >> >> Feb 12 10:07:41 zipper kernel: nvme0: Failed controller, stopping >> watchdog >> >> ti >> >> meout. >> >> >> >> The device looks healthy to me: >> >> SMART/Health Information Log >> >> >> >> Critical Warning State: 0x00 >> >> Available spare: 0 >> >> Temperature: 0 >> >> Device reliability:0 >> >> Read only: 0 >> >> Volatile memory backup:0 >> >> Temperature:312 K, 38.85 C, 101.93 F >> >> Available spare:100 >> >> Available spare threshold: 10 >> >> Percentage used:3 >> >> Data units (512,000 byte) read: 5761183 >> >> Data units written: 29911502 >> >> Host read commands: 471921188 >> >>
Re: nvme controller reset failures on recent -CURRENT
On Mon, Feb 12, 2024 at 9:15 PM Don Lewis wrote: > On 12 Feb, Maxim Sobolev wrote: > > Might be an overheating. Today's nvme drives are notoriously flaky if you > > run them without proper heat sink attached to it. > > I don't think it is a thermal problem. According to the drive health > page, the device temperature has never reached Temperature 2, whatever > that is. The room temperature is around 65F. The system was stable > last summer when the room temperature spent a lot of time in the 80-85F > range. The device temperature depends a lot on the I/O rate, and the > last panic happened when the I/O rate had been below 40tps for quite a > while. > It did reach temperature 1, though. That's the 'Warning this drive is too hot' temperature. It has spent 41213 minutes of your 19297 hours of up time, or an average of 2 minutes per hour. That's too much. Temperature 2 is critical error: we are about to shut down completely due to it being too hot. It's only a couple degrees below hardware power off due to temperature in many drives. Some really cheap ones don't really implement it at all. On my card with the bad heat sink, Warning temp is 70C while critical is 75C while IIRC thermal shutdown is 78C or 80C. I don't think we report these values in nvmecontrol identify. But you can do a raw dump with -x look at bytes 266:267 for warning and 268:269 for critical. In contrast, the few dozen drives that I have, all of which have been abused in various ways, And only one of them has any heat issues, and that one is an engineering special / sample with what I think is a damaged heat sink. If your card has no heat sink, this could well be what's going on. This panic means "the nvme card lost its mind and stopped talking to the host". Its status registers read 0xff's, which means that the card isn't decoding bus signals. Usually this means that the firmware on the card has faulted and rebooted. If the card is overheating, then this could well be what's happening. There's a tiny chance that this could be something more exotic, but my money is on hardware gone bad after 2 years of service. I don't think this is 'wear out' of the NAND (it's only 15TB written, but it could be if this drive is really really crappy nand: first generation QLC maybe, but it seems too new). It might also be a connector problem that's developed over time. There might be a few other things too, but I don't think this is a U.2 drive with funky cables. Warner > > On Mon, Feb 12, 2024, 4:28 PM Don Lewis wrote: > > > >> I just upgraded my package build machine to: > >> FreeBSD 15.0-CURRENT #110 main-n268161-4015c064200e > >> from: > >> FreeBSD 15.0-CURRENT #106 main-n265953-a5ed6a815e38 > >> and I've had two nvme-triggered panics in the last day. > >> > >> nvme is being used for swap and L2ARC. I'm not able to get a crash > >> dump, probably because the nvme device has gone away and I get an error > >> about not having a dump device. It looks like a low-memory panic > >> because free memory is low and zfs is calling malloc(). > >> > >> This shows up in the log leading up to the panic: > >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a > >> timeout a > >> nd possible hot unplug. > >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times > >> Feb 12 10:07:41 zipper kernel: nvme0: resetting controller > >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a > >> timeout a > >> nd possible hot unplug. > >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times > >> Feb 12 10:07:41 zipper kernel: nvme0: Waiting for reset to complete > >> Feb 12 10:07:41 zipper syslogd: last message repeated 2 times > >> Feb 12 10:07:41 zipper kernel: nvme0: failing queued i/o > >> Feb 12 10:07:41 zipper kernel: nvme0: Failed controller, stopping > watchdog > >> ti > >> meout. > >> > >> The device looks healthy to me: > >> SMART/Health Information Log > >> > >> Critical Warning State: 0x00 > >> Available spare: 0 > >> Temperature: 0 > >> Device reliability:0 > >> Read only: 0 > >> Volatile memory backup:0 > >> Temperature:312 K, 38.85 C, 101.93 F > >> Available spare:100 > >> Available spare threshold: 10 > >> Percentage used:3 > >> Data units (512,000 byte) read: 5761183 > >> Data units written: 29911502 > >> Host read commands: 471921188 > >> Host write commands:605394753 > >> Controller busy time (minutes): 32359 > >> Power cycles: 110 > >> Power on hours: 19297 > >> Unsafe shutdowns: 14 > >> Media errors: 0 > >> No. error info log entries: 0 > >> Warning Temp Composite Time:0 > >> Error Temp Composite Time: 0 > >> Temperature 1 Transition Count: 5231 > >> Temperature 2 Transition Count: 0 > >> Total Time For
Re: nvme controller reset failures on recent -CURRENT
On 12 Feb, Maxim Sobolev wrote: > Might be an overheating. Today's nvme drives are notoriously flaky if you > run them without proper heat sink attached to it. I don't think it is a thermal problem. According to the drive health page, the device temperature has never reached Temperature 2, whatever that is. The room temperature is around 65F. The system was stable last summer when the room temperature spent a lot of time in the 80-85F range. The device temperature depends a lot on the I/O rate, and the last panic happened when the I/O rate had been below 40tps for quite a while. > On Mon, Feb 12, 2024, 4:28 PM Don Lewis wrote: > >> I just upgraded my package build machine to: >> FreeBSD 15.0-CURRENT #110 main-n268161-4015c064200e >> from: >> FreeBSD 15.0-CURRENT #106 main-n265953-a5ed6a815e38 >> and I've had two nvme-triggered panics in the last day. >> >> nvme is being used for swap and L2ARC. I'm not able to get a crash >> dump, probably because the nvme device has gone away and I get an error >> about not having a dump device. It looks like a low-memory panic >> because free memory is low and zfs is calling malloc(). >> >> This shows up in the log leading up to the panic: >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a >> timeout a >> nd possible hot unplug. >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times >> Feb 12 10:07:41 zipper kernel: nvme0: resetting controller >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a >> timeout a >> nd possible hot unplug. >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times >> Feb 12 10:07:41 zipper kernel: nvme0: Waiting for reset to complete >> Feb 12 10:07:41 zipper syslogd: last message repeated 2 times >> Feb 12 10:07:41 zipper kernel: nvme0: failing queued i/o >> Feb 12 10:07:41 zipper kernel: nvme0: Failed controller, stopping watchdog >> ti >> meout. >> >> The device looks healthy to me: >> SMART/Health Information Log >> >> Critical Warning State: 0x00 >> Available spare: 0 >> Temperature: 0 >> Device reliability:0 >> Read only: 0 >> Volatile memory backup:0 >> Temperature:312 K, 38.85 C, 101.93 F >> Available spare:100 >> Available spare threshold: 10 >> Percentage used:3 >> Data units (512,000 byte) read: 5761183 >> Data units written: 29911502 >> Host read commands: 471921188 >> Host write commands:605394753 >> Controller busy time (minutes): 32359 >> Power cycles: 110 >> Power on hours: 19297 >> Unsafe shutdowns: 14 >> Media errors: 0 >> No. error info log entries: 0 >> Warning Temp Composite Time:0 >> Error Temp Composite Time: 0 >> Temperature 1 Transition Count: 5231 >> Temperature 2 Transition Count: 0 >> Total Time For Temperature 1: 41213 >> Total Time For Temperature 2: 0 >> >> >>
Re: nvme controller reset failures on recent -CURRENT
On 12 Feb, Mark Johnston wrote: > On Mon, Feb 12, 2024 at 04:28:10PM -0800, Don Lewis wrote: >> I just upgraded my package build machine to: >> FreeBSD 15.0-CURRENT #110 main-n268161-4015c064200e >> from: >> FreeBSD 15.0-CURRENT #106 main-n265953-a5ed6a815e38 >> and I've had two nvme-triggered panics in the last day. >> >> nvme is being used for swap and L2ARC. I'm not able to get a crash >> dump, probably because the nvme device has gone away and I get an error >> about not having a dump device. It looks like a low-memory panic >> because free memory is low and zfs is calling malloc(). >> >> This shows up in the log leading up to the panic: >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a timeout a >> nd possible hot unplug. >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times >> Feb 12 10:07:41 zipper kernel: nvme0: resetting controller >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a timeout a >> nd possible hot unplug. >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times >> Feb 12 10:07:41 zipper kernel: nvme0: Waiting for reset to complete >> Feb 12 10:07:41 zipper syslogd: last message repeated 2 times >> Feb 12 10:07:41 zipper kernel: nvme0: failing queued i/o >> Feb 12 10:07:41 zipper kernel: nvme0: Failed controller, stopping watchdog ti >> meout. > > Are you by chance using the drive mentioned here? > https://github.com/openzfs/zfs/discussions/14793 > > I was bitten by that and ended up replacing the drive with a different > model. The crash manifested exactly as you describe, though I didn't > have L2ARC or swap enabled on it. Nope: nda0 at nvme0 bus 0 scbus9 target 0 lun 1 nda0: nda0: Serial Number BTNH940617WE512A nda0: nvme version 1.3 nda0: 488386MB (1000215216 512 byte sectors) I'm not seeing super high I/O rates> I happened to have iostat running when the machine paniced: 0 584 88.431 2.68 65.8 112 7.18 68.2 107 7.13 80 0 20 0 0 0 565 99.132 3.06 27.974 2.01 30.570 2.08 80 0 20 0 0 0 612 92.831 2.77 18.9 148 2.74 18.9 148 2.73 86 0 14 0 0 0 618 88.613 1.17 25.059 1.44 24.261 1.44 89 0 11 0 0 0 586 45.4 5 0.22 31.455 1.70 30.857 1.70 84 0 16 0 0 0 598 12.7 3 0.03 38.164 2.40 37.166 2.40 84 0 16 0 0 0 675 36.1 6 0.21 23.7 156 3.62 22.7 164 3.63 88 0 12 0 0 0 641 6.9 6 0.04 25.7 243 6.10 25.3 246 6.08 71 0 29 0 0 0 737 20.1 9 0.18 36.4 148 5.24 37.2 144 5.24 78 0 22 0 0 0 578 44.723 1.03 25.1 164 4.01 25.5 161 3.99 86 0 14 0 0 0 608 70.315 1.06 51.164 3.19 51.364 3.19 89 0 11 0 0 0 624 38.6 9 0.35 32.3 121 3.80 32.2 121 3.79 90 0 10 0 0 0 577 80.616 1.28 37.866 2.44 36.569 2.46 90 0 10 0 0 tty nda0 ada0 ada1 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 566 87.716 1.39 27.260 1.60 25.366 1.62 87 0 13 0 0 0 599 77.211 0.83 17.4 391 6.66 17.3 395 6.66 74 0 26 0 0 0 660 45.0 7 0.31 18.7 575 10.51 18.6 578 10.49 76 0 24 0 0 0 615 37.7 8 0.31 24.0 303 7.11 24.0 303 7.11 58 0 42 0 0 Fssh_packet_write_wait: ... port 22: Broken pipe ada* are old and slow spinning rust. That report does mention something else that could also be a cause. I upgraded the motherboard BIOS around the same time. When I get a chance, I'll drop back to the older FreeBSD version and see if the problem goes away.
Re: nvme controller reset failures on recent -CURRENT
On Mon, Feb 12, 2024 at 04:28:10PM -0800, Don Lewis wrote: > I just upgraded my package build machine to: > FreeBSD 15.0-CURRENT #110 main-n268161-4015c064200e > from: > FreeBSD 15.0-CURRENT #106 main-n265953-a5ed6a815e38 > and I've had two nvme-triggered panics in the last day. > > nvme is being used for swap and L2ARC. I'm not able to get a crash > dump, probably because the nvme device has gone away and I get an error > about not having a dump device. It looks like a low-memory panic > because free memory is low and zfs is calling malloc(). > > This shows up in the log leading up to the panic: > Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a timeout a > nd possible hot unplug. > Feb 12 10:07:41 zipper syslogd: last message repeated 1 times > Feb 12 10:07:41 zipper kernel: nvme0: resetting controller > Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a timeout a > nd possible hot unplug. > Feb 12 10:07:41 zipper syslogd: last message repeated 1 times > Feb 12 10:07:41 zipper kernel: nvme0: Waiting for reset to complete > Feb 12 10:07:41 zipper syslogd: last message repeated 2 times > Feb 12 10:07:41 zipper kernel: nvme0: failing queued i/o > Feb 12 10:07:41 zipper kernel: nvme0: Failed controller, stopping watchdog ti > meout. Are you by chance using the drive mentioned here? https://github.com/openzfs/zfs/discussions/14793 I was bitten by that and ended up replacing the drive with a different model. The crash manifested exactly as you describe, though I didn't have L2ARC or swap enabled on it. > The device looks healthy to me: > SMART/Health Information Log > > Critical Warning State: 0x00 > Available spare: 0 > Temperature: 0 > Device reliability:0 > Read only: 0 > Volatile memory backup:0 > Temperature:312 K, 38.85 C, 101.93 F > Available spare:100 > Available spare threshold: 10 > Percentage used:3 > Data units (512,000 byte) read: 5761183 > Data units written: 29911502 > Host read commands: 471921188 > Host write commands:605394753 > Controller busy time (minutes): 32359 > Power cycles: 110 > Power on hours: 19297 > Unsafe shutdowns: 14 > Media errors: 0 > No. error info log entries: 0 > Warning Temp Composite Time:0 > Error Temp Composite Time: 0 > Temperature 1 Transition Count: 5231 > Temperature 2 Transition Count: 0 > Total Time For Temperature 1: 41213 > Total Time For Temperature 2: 0 > >
Re: nvme controller reset failures on recent -CURRENT
Might be an overheating. Today's nvme drives are notoriously flaky if you run them without proper heat sink attached to it. -Max On Mon, Feb 12, 2024, 4:28 PM Don Lewis wrote: > I just upgraded my package build machine to: > FreeBSD 15.0-CURRENT #110 main-n268161-4015c064200e > from: > FreeBSD 15.0-CURRENT #106 main-n265953-a5ed6a815e38 > and I've had two nvme-triggered panics in the last day. > > nvme is being used for swap and L2ARC. I'm not able to get a crash > dump, probably because the nvme device has gone away and I get an error > about not having a dump device. It looks like a low-memory panic > because free memory is low and zfs is calling malloc(). > > This shows up in the log leading up to the panic: > Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a > timeout a > nd possible hot unplug. > Feb 12 10:07:41 zipper syslogd: last message repeated 1 times > Feb 12 10:07:41 zipper kernel: nvme0: resetting controller > Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a > timeout a > nd possible hot unplug. > Feb 12 10:07:41 zipper syslogd: last message repeated 1 times > Feb 12 10:07:41 zipper kernel: nvme0: Waiting for reset to complete > Feb 12 10:07:41 zipper syslogd: last message repeated 2 times > Feb 12 10:07:41 zipper kernel: nvme0: failing queued i/o > Feb 12 10:07:41 zipper kernel: nvme0: Failed controller, stopping watchdog > ti > meout. > > The device looks healthy to me: > SMART/Health Information Log > > Critical Warning State: 0x00 > Available spare: 0 > Temperature: 0 > Device reliability:0 > Read only: 0 > Volatile memory backup:0 > Temperature:312 K, 38.85 C, 101.93 F > Available spare:100 > Available spare threshold: 10 > Percentage used:3 > Data units (512,000 byte) read: 5761183 > Data units written: 29911502 > Host read commands: 471921188 > Host write commands:605394753 > Controller busy time (minutes): 32359 > Power cycles: 110 > Power on hours: 19297 > Unsafe shutdowns: 14 > Media errors: 0 > No. error info log entries: 0 > Warning Temp Composite Time:0 > Error Temp Composite Time: 0 > Temperature 1 Transition Count: 5231 > Temperature 2 Transition Count: 0 > Total Time For Temperature 1: 41213 > Total Time For Temperature 2: 0 > > >
