[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 Wes Morgan changed: What|Removed |Added CC||morg...@gmail.com --- Comment #18 from Wes Morgan --- Exact same behavior for me with two WD SN770s running in a ZFS mirror, with no apparent reason for triggering, but happened every boot. After reading this and some discussions in the forums, I immediately replaced them with two Corsair MP600 PROs and the problem vanished. The two SN770s are still plugged into the device, but with the pool exported and idle, with no errors. A WD SN850X running as a root pool has no issues. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 Mark Johnston changed: What|Removed |Added CC||ma...@freebsd.org Status|New |Open --- Comment #17 from Mark Johnston --- Just a "me too" on a newly built system with a SN770 drive (actually two but I haven't tested the other one yet) used by ZFS. My BIOS doesn't report any problem with the 3.3V rail. The PSU is brand new and I'm a bit reluctant to replace it at this point. The problem isn't reproducible for me but has been been happening overnight, even when I'm not running and the system should be idle. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 Thierry Thomas changed: What|Removed |Added See Also||https://bugs.freebsd.org/bu ||gzilla/show_bug.cgi?id=2704 ||09 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 --- Comment #16 from Thierry Thomas --- (In reply to Marcus Oliveira from comment #15) See also https://forums.tomshardware.com/threads/nvme-ssd-sometimes-does-not-boot-because-of-weak-psu.3692473/ -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 Marcus Oliveira changed: What|Removed |Added CC||marcu...@gmail.com --- Comment #15 from Marcus Oliveira --- Writing for someone who might come after me... After reading this last post about 3.3V rail of the PSU, I changed the PSU and the problem with my NVME and Freebsd disappeared. Funny enough, even before replacing the PSU the problem wouldn't happen on Windows 11. Marcus -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 --- Comment #14 from Timothy Guo --- (In reply to crb from comment #11) I would like to share my follow up experience of this issue. In short, the problem magically goes away after I wipe the disk and recreated the pool from backup. The same system (hardware and SW) has been working without issue for about half a year now. Unfortunately, I couldn't locate a conclusive offender during the entire procedure. One thing I would like to note also is the 3.3V rail of the PSU. When I was still suffering from the issue, I also discovered 3.3V rail under-voltage, probably thanks to the hint from @crb's bug. I first read the out of range Voltage value from BIOS, and then confirmed the issue through direct measurement with a Voltage meter directly from the PSU pin-out. So it's true that the issue could really be power related. But it's unfortunate that I can't tell who is the offender, is the NVME drawing too much power due to firmware bug? Or is a failing PSU leading to NVME failure? I contacted my PSU vendor and got the feedback that the wire connector may be aged and increased the resistance. Maybe my Voltage measuring attempt fixed the wiring connection, maybe the wipe-out and rebuild worked-around a potential firmware bug. The issue just suddenly goes away, as it suddenly comes (Note: I couldn't remember any re-assembling of the hardware build when it suddenly comes, though.) The only part that I'm sure is the power failure is real and highly related. A stronger PSU might have simply avoided the problem altogether? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 Ed Maste changed: What|Removed |Added See Also|https://bugs.freebsd.org/bu | |gzilla/show_bug.cgi?id=2641 | |41 | --- Comment #13 from Ed Maste --- (In reply to crb from comment #11) Thanks for adding the followup and it seems I was probably hasty in adding the see-also. I'll remove it. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 --- Comment #12 from Lucas Holt --- It appears this is a problem with the WD SN770 in general. https://github.com/openzfs/zfs/discussions/14793 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 crb changed: What|Removed |Added CC||c...@chrisbowman.com --- Comment #11 from crb --- It looks like ed posted a comment referring to a similar bug I had submitted. The TLDR of that bug was that I was a 3.3v rail that was out of spec (as reported by the bios) and I replaced the power supply, the BIOS reported a compliant value for the 3.3 rail and that system has been ROCK solid ever since with the same SSD. I'm not suggesting that is your issue, but I'm trying to save people time reading my bug report and determining what the problem was on that one. Christopher -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 Ed Maste changed: What|Removed |Added See Also||https://bugs.freebsd.org/bu ||gzilla/show_bug.cgi?id=2641 ||41 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 Lucas Holt changed: What|Removed |Added CC||l...@foolishgames.com --- Comment #10 from Lucas Holt --- I've had the same issue on FreeBSD 13.1 release. Drive worked fine for about 2 months. The zfs pool completely flipped out (one device only) and system had to be powered off. reboot, zpool status and other commands would just hang. WD_BLACK™ SN770 NVMe™ SSD - 2TB SKU: WDS200T3X0E nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 nvme0: GET LOG PAGE (02) sqid:0 cid:0 nsid: cdw10:007f0002 cdw11: nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:0 sqid:0 cid:0 cdw0:0 Solaris: WARNING: Pool 'vm' has encountered an uncorrectable I/O failure and has been suspended. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 bibi changed: What|Removed |Added CC||gnub...@gmail.com --- Comment #9 from bibi --- Hello, I have same error on a FreeBSD 13-1 8<- Oct 8 15:47:09 donald kernel: nvme0: WRITE sqid:2 cid:0 nsid:1 lba:88256808 len:24 Oct 8 15:47:09 donald kernel: nvme0: resubmitting queued i/o Oct 8 15:47:09 donald kernel: nvme0: WRITE sqid:2 cid:0 nsid:1 lba:212904296 len:8 Oct 8 15:47:09 donald kernel: nvme0: resubmitting queued i/o Oct 8 15:47:09 donald kernel: nvme0: WRITE sqid:2 cid:0 nsid:1 lba:213705328 len:8 8<- Found also in log : 8<- Oct 7 23:49:21 donald kernel: nvme0: RECOVERY_START 6604360949574 vs 6603365139954 Oct 7 23:49:21 donald kernel: nvme0: timeout with nothing complete, resetting Oct 7 23:49:21 donald kernel: nvme0: Resetting controller due to a timeout. Oct 7 23:49:21 donald kernel: nvme0: RECOVERY_WAITING Oct 7 23:49:21 donald kernel: nvme0: resetting controller Oct 7 23:49:21 donald kernel: nvme0: aborting outstanding i/o 8<- -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 --- Comment #8 from Timothy Guo --- (In reply to Ian Brennan from comment #6) Was that a migration from TrueNAS core to Scale? Did you run into any regression specific to TrueNAS Scale? I heard Scale is less mature than Core. My SW setup used to be stable until I added that NVME disk as an upgrade. I really wish I can avoid any SW migration since the SW setup has been there working just fine for 8+ years with quite a few weird customization -- XEN virtualization, LXC container in dom0 served by TrueNAS core as a XEN guest. XEN virtualization and PCI passthrough could be a factor in this problem but so far I haven't found any evidence yet... My current top suspect is still the APST feature in FreeBSD driver... -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 --- Comment #7 from Timothy Guo --- (In reply to Timothy Guo from comment #5) I'm back reporting that the problem comes back just now (2 hours ago according to the alert mail) for no obvious reason -- I'm not actively using that server at this moment. The same nvme controller timeout reset shows up in kernel log and I loss access to the ZFS pool on it. The disk itself seems to work well physically so I'm not sure if I can ask for refund or any service. On the other hand, once the problem shows up, it appears to affect both Linux && FreeBSD running on the same physical box. Maybe firmware bug, maybe driver issue, maybe both... I used to suspect the problem is APST related, but I have no way to play with this config in FreeBSD. There is no mention of this term in the FreeBSD world. There is no user land tools that can manipulate or inspect the status of the APST related feature setting. It's kind of surprising since this feature had bad fame in the Linux world. Is there anybody who can help me at least do some diagnose on this problem? Is it feasible to manually parse the PCI config space to determine the APST status? I'll need some guide for this though... -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 --- Comment #6 from Ian Brennan --- I had the same problem basically, I gave up and installed TrueNAS Scale, Debian seemed to solve the problem. SSDs on FreeBSD just seemed way too unstable. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 --- Comment #5 from Timothy Guo --- (In reply to Timothy Guo from comment #4) The disk appears to be running in the freeNAS box just fine once again. I'll need to monitor the status for a longer while since it takes about one day for the issue to reproduce for the 2nd time, even though it repeats soon after each reboot from then on... Apologize if this turns out to be a noise. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 --- Comment #4 from Timothy Guo --- (In reply to Timothy Guo from comment #3) My SN550 was brought into a pretty bad state that it triggers AER flood on a Linux system and the BIOS was also affected that I can't even enter the BIOS setup page. As a desperate last resort, I move the disk into another machine (a NUC box). Magically the NUC box recognized the broken disk without problem, and booted just fine (Actually I lied, it landed on a prompt saying it's a FreeNAS data disk, which at least proves the disk can read just fine). Now after I moved it back to my home server box, the disk backs to live once again. I'm able to scrub the ZFS pool on this disk and find no data error at all. This time, I didn't run into any issue with APST enabled or disabled, which is really surprising. Let me switch back to the offending FreeNAS system to see how it behaves this time. Is this really a physical connection problem? How could it be -- I didn't use any adapter cable but directly hook it onto the M.2 slot on my motherboard. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 Timothy Guo changed: What|Removed |Added CC||firemeteor@users.sourceforg ||e.net --- Comment #3 from Timothy Guo --- I have an WD SN570 1TB NVME drive which suddenly run into this controller reset problem after about 2 months of active usage. Once this problem shows up it reproduces on every reboot. I'm still trying to confirm if this is an OS driver issue or disk issue. The disk appears to behave differently on different OS. When I switched to Linux, the same problematic disk appears to react to different APST config and I was able to get it pass some short read-only tests (DD, disk-wide find, grep in kernel tree etc.) with APST disabled. No matter if this observation is real or not, it encouraged me to switch back to my FreeBSD box and try ZFS scrub. Unfortunately the disk fails terribly this time and I couldn't even get it back to work in Linux as the drive appears to get stuck in some low power state... Will try dig deeper to see if there is anything I can do to get it back. BTW, the SMART log does not report any error for this drive... -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 Graham Perrin changed: What|Removed |Added CC||grahamper...@gmail.com --- Comment #2 from Graham Perrin --- (In reply to Ian Brennan from comment #0) Is there a corresponding report in the TrueNAS area? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 Tomasz "CeDeROM" CEDRO changed: What|Removed |Added CC||to...@cedro.info --- Comment #1 from Tomasz "CeDeROM" CEDRO --- Hello world :-) Just saw previous and this report. I also switched to M2.NVM Samsung SSD 980 1TB drive over ICYBOX IB-PCI224M2-ARGB PCI-E 4.0 controller on my desktop but all seems to work fine. Just a reference, may help somehow :-) nvme0: mem 0xfe60-0xfe603fff at device 0.0 on pci5 nvme0: Allocated 64MB host memory buffer nvd0: NVMe namespace nvd0: 953869MB (1953525168 512 byte sectors) FreeBSD hexagon 13.1-STABLE FreeBSD 13.1-STABLE #0 stable/13-n250096-4f69c575996: Fri Mar 25 03:50:58 CET 2022 root@hexagon:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 262969] NVMe - Resetting controller due to a timeout and possible hot unplug
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262969 Bug ID: 262969 Summary: NVMe - Resetting controller due to a timeout and possible hot unplug Product: Base System Version: 13.1-STABLE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: b...@freebsd.org Reporter: ibren...@netgrade.com I am seeing very unstable NVMe on TrueNAS 12 and 13. I'm using Western Digital Black NVMe m.2 SSD's. The exact version I'm using is WD_BLACK SN770 250GB & 2TB, firmware version 731030WD. This is on TrueNAS version: TrueNAS-12.0-U8 / FreeBSD: 12.2-RELEASE-p12 (I later upgraded to 13 Beta, see below) I set hw.nvme.per_cpu_io_queues=0, and it did not fix the problem, in fact it seems to have made it much more frequent, although I'm not 100% sure about that, need to test again. I also tried using the nvd driver with hw.nvme.use_nvd=0, which doesn't seem to make a difference, however it had slightly different results in the log when the issue happened again. See logs below, would be grateful if somebody can help with this problem. Mar 29 21:42:25 truenas nvme5: Resetting controller due to a timeout and possible hot unplug. Mar 29 21:42:25 truenas nvme5: resetting controller Mar 29 21:42:25 truenas nvme5: failing outstanding i/o Mar 29 21:42:25 truenas nvme5: READ sqid:12 cid:120 nsid:1 lba:1497544880 len:16 Mar 29 21:42:25 truenas nvme5: ABORTED - BY REQUEST (00/07) sqid:12 cid:120 cdw0:0 Mar 29 21:42:25 truenas nvme5: failing outstanding i/o Mar 29 21:42:25 truenas nvme5: READ sqid:12 cid:123 nsid:1 lba:198272936 len:16 Mar 29 21:42:25 truenas nvme5: ABORTED - BY REQUEST (00/07) sqid:12 cid:123 cdw0:0 Mar 29 21:42:25 truenas nvme5: failing outstanding i/o Mar 29 21:42:25 truenas nvme5: READ sqid:13 cid:121 nsid:1 lba:431014528 len:24 Mar 29 21:42:25 truenas nvme5: ABORTED - BY REQUEST (00/07) sqid:13 cid:121 cdw0:0 Mar 29 21:42:25 truenas nvme5: failing outstanding i/o Mar 29 21:42:25 truenas nvme5: READ sqid:15 cid:127 nsid:1 lba:864636432 len:8 Mar 29 21:42:25 truenas nvme5: ABORTED - BY REQUEST (00/07) sqid:15 cid:127 cdw0:0 Mar 29 21:42:25 truenas nvme5: failing outstanding i/o Mar 29 21:42:25 truenas nvme5: READ sqid:16 cid:126 nsid:1 lba:2445612184 len:8 Mar 29 21:42:25 truenas nvme5: ABORTED - BY REQUEST (00/07) sqid:16 cid:126 cdw0:0 Mar 29 21:42:25 truenas nvme5: failing outstanding i/o Mar 29 21:42:25 truenas nvme5: READ sqid:16 cid:120 nsid:1 lba:430503600 len:8 Mar 29 21:42:25 truenas nvme5: ABORTED - BY REQUEST (00/07) sqid:16 cid:120 cdw0:0 Mar 29 21:42:25 truenas nvme5: failing outstanding i/o Mar 29 21:42:25 truenas nvme5: READ sqid:18 cid:123 nsid:1 lba:1499051024 len:8 Mar 29 21:42:26 truenas nvme5: ABORTED - BY REQUEST (00/07) sqid:18 cid:123 cdw0:0 Mar 29 21:42:26 truenas nvme5: failing outstanding i/o Mar 29 21:42:26 truenas nvme5: WRITE sqid:18 cid:124 nsid:1 lba:1990077368 len:8 Mar 29 21:42:26 truenas nvme5: ABORTED - BY REQUEST (00/07) sqid:18 cid:124 cdw0:0 Mar 29 21:42:26 truenas nvme5: failing outstanding i/o Mar 29 21:42:26 truenas nvme5: READ sqid:19 cid:122 nsid:1 lba:1237765696 len:8 Mar 29 21:42:26 truenas nvme5: ABORTED - BY REQUEST (00/07) sqid:19 cid:122 cdw0:0 Mar 29 21:42:26 truenas nvme5: failing outstanding i/o Mar 29 21:42:26 truenas nvme5: READ sqid:19 cid:125 nsid:1 lba:180758264 len:16 Mar 29 21:42:26 truenas nvme5: ABORTED - BY REQUEST (00/07) sqid:19 cid:125 cdw0:0 Mar 29 21:42:26 truenas nvme5: failing outstanding i/o Mar 29 21:42:26 truenas nvme5: READ sqid:20 cid:121 nsid:1 lba:2445612192 len:8 Mar 29 21:42:26 truenas nvme5: ABORTED - BY REQUEST (00/07) sqid:20 cid:121 cdw0:0 Mar 29 21:42:26 truenas nvd5: detached nvme3: Resetting controller due to a timeout and possible hot unplug. nvme3: resetting controller nvme3: failing outstanding i/o nvme3: READ sqid:7 cid:127 nsid:1 lba:419546528 len:8 nvme3: ABORTED - BY REQUEST (00/07) sqid:7 cid:127 cdw0:0 nvme3: (nda3:nvme3:0:0:1): READ. NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=1901c5a0 0 7 0 0 0 failing outstanding i/o (nda3:nvme3:0:0:1): CAM status: CCB request completed with an error (nda3:nvme3:0:0:1): Error 5, Retries exhausted nvme3: READ sqid:11 cid:127 nsid:1 lba:782841288 len:8 nvme3: ABORTED - BY REQUEST (00/07) sqid:11 cid:127 cdw0:0 nvme3: (nda3:nvme3:0:0:1): READ. NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=2ea935c8 0 7 0 0 0 failing outstanding i/o nvme3: READ sqid:11 cid:123 nsid:1 lba:704576056 len:8 nvme3: ABORTED - BY REQUEST (00/07) sqid:11 cid:123 cdw0:0 nvme3: failing outstanding i/o nvme3: WRITE sqid:12 cid:127 nsid:1 lba:1016402352 len:8 nvme3: ABORTED - BY REQUEST (00/07) sqid:12 cid:127 cdw0:0 nvme3: failing outstanding i/o nvme3: READ sqid:12 cid:125 nsid:1 lba:1824854760 len:8 nvme3: ABORTED - BY REQUEST (00/07) sqid:12 cid:125 cdw0:0 nvme3: failing outstanding i/o nvme3:
