On 9/21/26 18:18, S. Ross Gohlke wrote:
On 9/15/26 07:12, S. Ross Gohlke wrote:
On 9/14/26 11:31, Kevin Bowling wrote:
On Sun, Sep 13, 2026 at 9:41 AM S. Ross Gohlke<[email protected]> wrote:
On 9/13/26 08:10, Abdelkader Boudih wrote:
Hi Again
The last round endup with 80% of the Diffs merged and 50% already MFCed.
imp@ still want to try one diff in a Netflix server...
So if there is an outage that spoil your favorite show, I'm the one to
blame. :)
Marcin Cieslak reported a wedge in Apple S3x . (I still trying to
reproduce it, but maybe this stack fix it)
New round of NVME testing :
D59637 nvme: do not touch INTMS/INTMC when configured for MSI-X
D59636 nvme: enforce the Abort Command Limit
D59634 nvme: do not complete a command when its Abort is not performed
D59633 nvme: delete the I/O queues in the system shutdown path
D59631 nvme: honor CRTO for the controller ready timeout
D59628 nvme: derive CC.CSS from CAP.CSS instead of hardcoding the NVM set
D59627 nvme: honor FLBAS Format Index Upper when selecting the LBA format
D59626 nvme: bound the AER error log byte-swap by the fetched length
D59625 nvme: reject namespaces formatted with metadata
This all extracted fromhttps://nvmexpress.org/specifications/ (Spec
2.4 , August 2026)
Link to the top of the stackhttps://reviews.freebsd.org/D59636
Mostly Spec review, previous versions of the spec was either vague, or
non-existant and nobody touched the code since then.
I request more people to test it so we add Quirks to drives that are
not spec compliant, but first we need to get the driver to spec
complianciancy first.
I tested this on my fleet for 10 days, and the machines are daily
building kernels and running as CI.
Best,
Abdelkader
I upgraded CURRENT via pkgbase last night, from 1600021 to 1600025.
* FreeBSD-src-16.snap20260912223212
* FreeBSD-utilities-16.snap20260912182422
I run a custom kernel based on MINIMAL_MMCCAM, but nvme is in the kernel.
```
% sysctl kern.conftxt | grep nvm
device nvme
```
Also, I tested with GENERIC kernel and got the same result.
System is ThinkPad L14 Gen 2 Tiger Lake.
I have everything working as expected, but intr shows unusually high
utilization, around
50% of a core, constantly, in top(1).
Did you identify _which_ intr? It sounds like it could be a22eb754c4bb.
I guess I don't know how to identify _which_ intr.
I am seeing no output in the usual places -- dmesg, /var/log/messages.
I tried enabling hw.nvme.verbose_cmd_dump: 1, but that makes no
difference.
Maybe I am wrong about nvme being the cause, I just ran out of things
to check.
All I know is, this is definitely new behavior and I have exhausted
my troubleshooting playbook.
From "top -PCS":
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
11 root 8 199 ki31 0B 128K RUN 0 60.2H
788.76% idle
12 root 29 -54 - 0B 464K WAIT 0 290:43
60.54% intr
If there is something else to check related to intr, please let me
know which manual page to start reading.
Also, I will look for a22eb754c4bb. Thanks for the pointer.
I tried everything I could think of, with no change.
* unload kernel modules
* stop all services
Then remembered the nvme upgrades.
I think this is the cause of the problem.
I don't know anything substantial about storage hardware in general or
nvme in specific.
I tried different values in loader.conf(5) for some tunables mentioned
in nvme(4) that
seemed plausibly related.
* hw.nvme.apst_enable
* hw.nvme.force_intx
They do not even show up in sysctl(8) after reboot.
```
% doas nvmecontrol devlist
nvme0: SAMSUNG MZALQ256HBJD-00BL2
nvme0ns1 (244198MB)
% doas pciconf -lv
...
nvme0@pci0:2:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d
device=0xa809 subvendor=0x144d subdevice=0xa801
vendor = 'Samsung Electronics Co Ltd'
device = 'NVMe SSD Controller 980 (DRAM-less)'
class = mass storage
subclass = NVM
...
% doas nvmecontrol identify nvme0
Controller Capabilities/Features
================================
Vendor ID: 144d
Subsystem Vendor ID: 144d
Serial Number: S65FNE1R713832
Model Number: SAMSUNG MZALQ256HBJD-00BL2
Firmware Version: 7L2QFXM7
Recommended Arb Burst: 2
IEEE OUI Identifier: 00 25 38
Multi-Path I/O Capabilities: Not Supported
Max Data Transfer Size: 2097152 bytes
Sanitize Crypto Erase: Not Supported
Sanitize Block Erase: Supported
Sanitize Overwrite: Not Supported
Sanitize NDI: Supported
Sanitize NODMMAS: No
Controller ID: 0x0005
Version: 1.4.0
Traffic Based Keep Alive: Not Supported
Controller Type: I/O Controller
Keep Alive Timer Not Supported
Maximum Outstanding Commands Not Specified
Admin Command Set Attributes
============================
Security Send/Receive: Supported
Format NVM: Supported
Firmware Activate/Download: Supported
Namespace Management: Not Supported
Device Self-test: Supported
Directives: Not Supported
NVMe-MI Send/Receive: Not Supported
Virtualization Management: Not Supported
Doorbell Buffer Config: Not Supported
Get LBA Status: Not Supported
Sanitize: block,
Abort Command Limit: 8
Async Event Request Limit: 4
Number of Firmware Slots: 3
Firmware Slot 1 Read-Only: No
Per-Namespace SMART Log: Yes
Error Log Page Entries: 64
Number of Power States: 5
Total NVM Capacity: 256060514304 bytes
Unallocated NVM Capacity: 0 bytes
Firmware Update Granularity: 04 (16384 bytes)
Host Buffer Preferred Size: 67108864 bytes
Host Buffer Minimum Size: 16777216 bytes
NVM Command Set Attributes
==========================
Submission Queue Entry Size
Max: 64
Min: 64
Completion Queue Entry Size
Max: 16
Min: 16
Number of Namespaces: 1
Compare Command: Supported
Write Uncorrectable Command: Supported
Dataset Management Command: Supported
Write Zeroes Command: Not Supported
Save Features: Supported
Reservations: Not Supported
Timestamp feature: Supported
Verify feature: Not Supported
Fused Operation Support: Not Supported
Format NVM Attributes: Per-NS Erase, Per-NS Format
Volatile Write Cache: Present, flush all
NVM Subsystem Name: nqn.1994-11.com.samsung:nvme:PM991a:M.2:S65FNE1R713832
% doas nvmecontrol identify nvme0ns1
Size: 500118192 blocks
Capacity: 500118192 blocks
Utilization: 484394000 blocks
Thin Provisioning: Not Supported
Number of LBA Formats: 1
Current LBA Format: LBA Format #00
Metadata Capabilities
Extended: Not Supported
Separate: Not Supported
Data Protection Caps: Not Supported
Data Protection Settings: Not Enabled
Multi-Path I/O Capabilities: Not Supported
Reservation Capabilities: Not Supported
Format Progress Indicator: 0% remains
Deallocate Logical Block: Read Not Reported
Optimal I/O Boundary: 0 blocks
NVM Capacity: 256060514304 bytes
Preferred Write Granularity: 32 blocks
Preferred Write Alignment: 8 blocks
Preferred Deallocate Granul: 10368 blocks
Preferred Deallocate Align: 10368 blocks
Optimal Write Size: 256 blocks
Globally Unique Identifier: 00000000000000000000000000000000
IEEE EUI64: 002538d711133242
LBA Format #00: Data Size: 512 Metadata Size: 0 Performance: Best
% sysctl hw.nvme
hw.nvme.verbose_cmd_dump: 1
hw.nvme.use_nvd: 0
% sysctl dev.nvme
dev.nvme.0.alignment_splits: 0
dev.nvme.0.ioq.3.dump_debug: 0
dev.nvme.0.ioq.3.recovery: 0
dev.nvme.0.ioq.3.num_recovery_nolock: 0
dev.nvme.0.ioq.3.num_ignored: 0
dev.nvme.0.ioq.3.num_failures: 0
dev.nvme.0.ioq.3.num_retries: 0
dev.nvme.0.ioq.3.num_intr_handler_calls: 10915
dev.nvme.0.ioq.3.num_cmds: 11049
dev.nvme.0.ioq.3.cq_head: 41
dev.nvme.0.ioq.3.sq_tail: 41
dev.nvme.0.ioq.3.sq_head: 41
dev.nvme.0.ioq.3.num_trackers: 128
dev.nvme.0.ioq.3.num_entries: 256
dev.nvme.0.ioq.2.dump_debug: 0
dev.nvme.0.ioq.2.recovery: 0
dev.nvme.0.ioq.2.num_recovery_nolock: 0
dev.nvme.0.ioq.2.num_ignored: 0
dev.nvme.0.ioq.2.num_failures: 0
dev.nvme.0.ioq.2.num_retries: 0
dev.nvme.0.ioq.2.num_intr_handler_calls: 5321
dev.nvme.0.ioq.2.num_cmds: 5410
dev.nvme.0.ioq.2.cq_head: 34
dev.nvme.0.ioq.2.sq_tail: 34
dev.nvme.0.ioq.2.sq_head: 34
dev.nvme.0.ioq.2.num_trackers: 128
dev.nvme.0.ioq.2.num_entries: 256
dev.nvme.0.ioq.1.dump_debug: 0
dev.nvme.0.ioq.1.recovery: 0
dev.nvme.0.ioq.1.num_recovery_nolock: 0
dev.nvme.0.ioq.1.num_ignored: 0
dev.nvme.0.ioq.1.num_failures: 0
dev.nvme.0.ioq.1.num_retries: 0
dev.nvme.0.ioq.1.num_intr_handler_calls: 10896
dev.nvme.0.ioq.1.num_cmds: 11014
dev.nvme.0.ioq.1.cq_head: 6
dev.nvme.0.ioq.1.sq_tail: 6
dev.nvme.0.ioq.1.sq_head: 6
dev.nvme.0.ioq.1.num_trackers: 128
dev.nvme.0.ioq.1.num_entries: 256
dev.nvme.0.ioq.0.dump_debug: 0
dev.nvme.0.ioq.0.recovery: 0
dev.nvme.0.ioq.0.num_recovery_nolock: 0
dev.nvme.0.ioq.0.num_ignored: 0
dev.nvme.0.ioq.0.num_failures: 0
dev.nvme.0.ioq.0.num_retries: 0
dev.nvme.0.ioq.0.num_intr_handler_calls: 10407
dev.nvme.0.ioq.0.num_cmds: 10503
dev.nvme.0.ioq.0.cq_head: 7
dev.nvme.0.ioq.0.sq_tail: 7
dev.nvme.0.ioq.0.sq_head: 7
dev.nvme.0.ioq.0.num_trackers: 128
dev.nvme.0.ioq.0.num_entries: 256
dev.nvme.0.adminq.dump_debug: 0
dev.nvme.0.adminq.recovery: 0
dev.nvme.0.adminq.num_recovery_nolock: 0
dev.nvme.0.adminq.num_ignored: 0
dev.nvme.0.adminq.num_failures: 0
dev.nvme.0.adminq.num_retries: 0
dev.nvme.0.adminq.num_intr_handler_calls: 225
dev.nvme.0.adminq.num_cmds: 229
dev.nvme.0.adminq.cq_head: 97
dev.nvme.0.adminq.sq_tail: 101
dev.nvme.0.adminq.sq_head: 101
dev.nvme.0.adminq.num_trackers: 16
dev.nvme.0.adminq.num_entries: 128
dev.nvme.0.fail_on_reset: 0
dev.nvme.0.cap_hi: 48
dev.nvme.0.cap_lo: 1006845951
dev.nvme.0.reset_stats: 0
dev.nvme.0.num_recovery_nolock: 0
dev.nvme.0.num_ignored: 0
dev.nvme.0.num_failures: 0
dev.nvme.0.num_retries: 0
dev.nvme.0.num_intr_handler_calls: 37764
dev.nvme.0.num_cmds: 38205
dev.nvme.0.timeout_period: 30
dev.nvme.0.admin_timeout_period: 60
dev.nvme.0.int_coal_threshold: 0
dev.nvme.0.int_coal_time: 0
dev.nvme.0.num_io_queues: 4
dev.nvme.0.wake: 0
dev.nvme.0.%iommu: rid=0x200
dev.nvme.0.%parent: pci1
dev.nvme.0.%pnpinfo: vendor=0x144d device=0xa809 subvendor=0x144d
subdevice=0xa801 class=0x010802
dev.nvme.0.%location: slot=0 function=0 dbsf=pci0:2:0:0
handle=\_SB_.PC00.PEG0.PEGP
dev.nvme.0.%driver: nvme
dev.nvme.0.%desc: Generic NVMe Device
dev.nvme.%parent:
```
If there is more information I need to provide, let me know.
I usually just wait for such things to work themselves out, and they
usually do over time, but given the call for testers I thought this
feedback might be useful... and it would be nice to have my power
sipping setup back.
Regards,
Ross
I upgraded pkgbase yesterday, from 1600025 to 1600026.
* FreeBSD-src-16.snap20260920013406
* FreeBSD-utilities-16.snap20260919182608
intr utilization is down to 3-5% from 50-60%.
Thanks.
Regards,
Ross
I forgot my tests. I am happy to run more.
doas nvmecontrol perftest -n 16 -o read -s 4096 -t 30 nvme0ns1
Threads: 16 Size: 4096 READ Time: 30 IO/s: 34572 MB/s: 135
doas nvmecontrol perftest -n 16 -o read -s 131072 -t 30 nvme0ns1
Threads: 16 Size: 131072 READ Time: 30 IO/s: 5019 MB/s: 627
doas nvmecontrol perftest -n 32 -o read -s 4096 -t 30 nvme0ns1
Threads: 32 Size: 4096 READ Time: 30 IO/s: 33295 MB/s: 130
doas nvmecontrol perftest -n 32 -o read -s 131072 -t 30 nvme0ns1
Threads: 32 Size: 131072 READ Time: 30 IO/s: 5538 MB/s: 692
doas nvmecontrol perftest -n 64 -o read -s 4096 -t 30 nvme0ns1
Threads: 64 Size: 4096 READ Time: 30 IO/s: 33066 MB/s: 129
doas nvmecontrol perftest -n 64 -o read -s 131072 -t 30 nvme0ns1
Threads: 64 Size: 131072 READ Time: 30 IO/s: 5833 MB/s: 729
