Hey Matthew, On Wed, May 1, 2024 at 2:25 PM Matthew Grooms <mgro...@shrew.net> wrote:
> On 5/1/24 14:38, mike tancsa wrote: > > Kind of struggling to check if TRIM is actually working or not with my > > SSDs on RELENG_14 in ZFS. > > > > On a pool that has almost no files on it (capacity at 0% out of 3TB), > > should not > > > > zpool -w trim <pool> be almost instant after a couple of runs ? > > Instead it seems to always take about 10min to complete. > > > > Looking at the stats, > > > > kstat.zfs.tortank1.misc.iostats.trim_bytes_failed: 0 > > kstat.zfs.tortank1.misc.iostats.trim_extents_failed: 0 > > kstat.zfs.tortank1.misc.iostats.trim_bytes_skipped: 2743435264 > > kstat.zfs.tortank1.misc.iostats.trim_extents_skipped: 253898 > > kstat.zfs.tortank1.misc.iostats.trim_bytes_written: 14835526799360 > > kstat.zfs.tortank1.misc.iostats.trim_extents_written: 1169158 > > > > what and why are bytes being skipped ? > > > > One of the drives for example > > > > sysctl -a kern.cam.ada.0 > > kern.cam.ada.0.trim_ticks: 0 > > kern.cam.ada.0.trim_goal: 0 > > kern.cam.ada.0.sort_io_queue: 0 > > kern.cam.ada.0.rotating: 0 > > kern.cam.ada.0.unmapped_io: 1 > > kern.cam.ada.0.flags: > > > 0x1be3bde<CAN_48BIT,CAN_FLUSHCACHE,CAN_NCQ,CAN_DMA,WAS_OTAG,CAN_TRIM,OPEN,SCTX_INIT,CAN_POWERMGT,CAN_DMA48,CAN_LOG,CAN_WCACHE,CAN_RAHEAD,PROBED,ANNOUNCED,DIRTY,PIM_ATA_EXT,UNMAPPEDIO> > > kern.cam.ada.0.max_seq_zones: 0 > > kern.cam.ada.0.optimal_nonseq_zones: 0 > > kern.cam.ada.0.optimal_seq_zones: 0 > > kern.cam.ada.0.zone_support: None > > kern.cam.ada.0.zone_mode: Not Zoned > > kern.cam.ada.0.write_cache: -1 > > kern.cam.ada.0.read_ahead: -1 > > kern.cam.ada.0.trim_lbas: 7771432624 > > kern.cam.ada.0.trim_ranges: 371381 > > kern.cam.ada.0.trim_count: 310842 > > kern.cam.ada.0.delete_method: DSM_TRIM > > > > If I take one of the disks out of the pool and replace it with a > > spare, and do a manual trim it seems to work > > > I had a hard time seeing evidence of this at the disk level while > fiddling with TRIM recently. It appeared that at least some counters are > driver and operation specific. For example, the da driver appears to > update counters in some paths but not others. I assume that ada is > different. There is a bug report for da, but haven't seen any feedback ... > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277673 > > You could try to run gstat with the -d flag during the time period when > the delete operations are expected to occur. That should give you an > idea of what's happening at the disk level in real time but may not > offer more info than you're already seeing. > These changes looked good. My apologies for not noticing this sooner (I think that our CC to the mailing list in bugzilla has stopped working, so I didn't see the redirect). I've committed the changes, and queued them to my stable-14 and stable-13 branches. Thank you so much for your submission. Warner
