Re: [gentoo-user] Understanding fstrim...
On Tue, Apr 14, 2020 at 10:26 AM Wols Lists wrote: > > On 14/04/20 13:51, Rich Freeman wrote: > > I believe they have > > to be PCIv3+ and typically have 4 lanes, which is a lot of bandwidth. > > My new mobo - the manual says if I put an nvme drive in - I think it's > the 2nd nvme slot - it disables the 2nd graphics card slot :-( > First, there is no such thing as an "nvme slot". You're probably describing an M.2 slot. This matters as I'll get to later... As I mentioned many motherboards share a PCIe slot with an M.2 slot. The CPU + chipset only has so many PCIe lanes. So unless they aren't already using them for expansion slots they have to double up. By doubling up they can basically stick more x2/4/8 PCIe slots into the motherboard than they could if they completely dedicated them. Or they could let that second GPU talk directly to the CPU vs having to go through the chipset (I think - I'm not really an expert on PCIe), and let the NVMe talk directly to the CPU if you aren't using that second GPU. > > But using the 1st nvme slot disables a sata slot, which buggers my raid > up ... :-( > While that might be an M.2 slot, it probably isn't an "nvme slot". M.2 can be used for either SATA or PCIe. Some motherboards have one, the other, or both. And M.2 drives can be either, so you need to be sure you're using the right one. If you get the wrong kind of drive it might not work, or it might end up being a SATA drive when you intended to use an NVMe. A SATA drive will have none of the benefits of NVMe and will be functionally no different from just a regular 2.5" SSD that plugs into a SATA cable - it is just a different form factor. It sounds like they doubled up a PCIe port on the one M.2 connector, and they doubled up a SATA port on the other M.2 connector. It isn't necessarily a bad thing, but obviously you need to make tradeoffs. If you want a motherboard with a dozen x16 PCIe's, 5 M.2's,14 SATA ports, and 10 USB3's on it there is no reason that shouldn't be possible, but don't expect to find it in the $60 bargain bin, and don't expect all those lanes to talk directly to the CPU unless you're using EPYC or something else high-end. :) -- Rich
Re: [gentoo-user] Understanding fstrim...
On 14/04/20 13:51, Rich Freeman wrote: > I believe they have > to be PCIv3+ and typically have 4 lanes, which is a lot of bandwidth. My new mobo - the manual says if I put an nvme drive in - I think it's the 2nd nvme slot - it disables the 2nd graphics card slot :-( Seeing as I need two graphics cards to double-head my system, that means I can't use two nvmes :-( But using the 1st nvme slot disables a sata slot, which buggers my raid up ... :-( Oh well. That's life :-( Cheers, Wol
Re: [gentoo-user] Understanding fstrim...
On Mon, Apr 13, 2020 at 11:32 PM wrote: > > Since I have a NVMe drive on a M.2 socket I would > be interested at what level/stage (?word? ...sorry...) > the data go a different path as with the classical sata > SSDs. > > Is this just "protocol" or there is something different? NVMe involves both hardware, protocol, and of course software changes driven by these. First, a disclaimer, I am by no means an expert in storage transport protocols/etc and obviously there are a ton of standards so that any random drive works with any random motherboard/etc. If I missed something or have any details wrong please let me know. >From the days of IDE to pre-NVMe on PC the basic model was that the CPU would talk to a host bus adapter (HBA) which would in turn talk to the drive controller. The HBA was usually on the motherboard but of course it could be in an expansion slot. The CPU talked to the HBA using the bus standards of the day (ISA/PCI/PCIe/etc) and this was of course a high-speed bus designed to work with all kinds of stuff. AHCI is the latest generation of protocols for communication between the CPU and the HBA so that any HBA can work with any OS/etc using generic drivers. The protocol was designed with spinning hard drives in mind and has some limitations with SSD. The HBA would talk to the drive over SCSI/SAS/SATA/PATA and there are a bunch of protocols designed for this. Again, they were designed in the era of hard drives and have some limitations. The concept of NVMe is to ditch the HBA and stick the drive directly on the PCIe bus which is much faster, and streamline the protocols. This is a little analogous to the shift to IDE from the old days of separate drive controllers - cutting out a layer of interface hardware. Early NVMes just had their own protocols, but a standard was created so that just as with AHCI the OS can use a single driver for any drive vendor. At the hardware level the big change is that NVMe just uses the PCIe bus. That M.2 adapter has a different form factor than a regular PCIe slot, and I didn't look at the whole pinout, but you probably could just make a dumb adapter to plug a drive right into a PCIe slot as electronically I think they're just PCIe cards. I believe they have to be PCIv3+ and typically have 4 lanes, which is a lot of bandwidth. And of course it is the same interface used for NICs/graphics/etc so it is pretty low latency and can support hardware interrupts and all that stuff. It is pretty common for motherboards to share an M.2 slot with a PCIe slot so that you can use one or the other but not both for this reason - the same lanes are in use for both. The wikipedia article has a good comparison of the protocol-level changes. For the most part it mainly involves making things far more parallel. With ATA/AHCI you'd have one queue of commands that was only a few instructions deep. With NVMe you can have thousands of queues with thousands of commands in each and 2048 different hardware interrupts for the drive to be able to signal back when one command vs another has completed (although I'm really curious if anybody takes any kind of advantage of that - unless you have different drivers trying to use the same drive in parallel it seems like MSI-X isn't saving much here - maybe if you had a really well-trusted VM or something or the command set has some way to segment the drive virtually...). This basically allows billions of operations to be in-progress at any given time so there is much less of a bottleneck in the protocol/interface itself. NVMe is all about IOPS. I don't know all the gory details but I wouldn't be surprised that once you get past this if many of the commands themselves are the same just to keep things simple. Or maybe they just rewrote it all from scratch - I didn't look into it and would be curious to hear from somebody who has. Obviously the concepts of read/write/trim/etc are going to apply regardless of interface. -- Rich
Re: [gentoo-user] Understanding fstrim...
On 04/13 04:58, Rich Freeman wrote: > On Mon, Apr 13, 2020 at 4:34 PM antlists wrote: > > > > aiui, the spec says you can send a command "trim 1GB starting at block > > X". Snag is, the linux block size of 4KB means that it gets split into > > loads of trim commands, which then clogs up all the buffers ... > > > > Hmm, found the ATA spec at: > http://nevar.pl/pliki/ATA8-ACS-3.pdf > > In particular page 76 which outlines the LBA addressing for TRIM. It > looks like up to 64k ranges can be sent per trim, and each range can > cover up to 64k blocks. So that is 2^32 blocks per trim command, or > 2TiB of data per TRIM if the blocks are 512 bytes (which I'm guessing > is the case for ATA but I didn't check). The command itself would be > half a megabyte since each range is 64 bits. > > But if the kernel chops them up as you say that will certainly add > overhead. The drive controller itself is probably the bigger > bottleneck unless it is designed to do fast TRIMs. > > -- > Rich > Hi all, thanks **a lot** for all this great information! :) Since I have a NVMe drive on a M.2 socket I would be interested at what level/stage (?word? ...sorry...) the data go a different path as with the classical sata SSDs. Is this just "protocol" or there is something different? On the internet I read, that the io-scheduler is choosen differentlu by the kernel, if there is a NVMe driven detected, for example. (I think, the io-scheduler has nothing to do with the fstrim operation itsself (I think...) -- it is there only as an example...) I think, due to the corona lockdown I have to fstrim my hair myself :) 8) Cheers! Meino
Re: [gentoo-user] Understanding fstrim...
On Mon, Apr 13, 2020 at 4:34 PM antlists wrote: > > aiui, the spec says you can send a command "trim 1GB starting at block > X". Snag is, the linux block size of 4KB means that it gets split into > loads of trim commands, which then clogs up all the buffers ... > Hmm, found the ATA spec at: http://nevar.pl/pliki/ATA8-ACS-3.pdf In particular page 76 which outlines the LBA addressing for TRIM. It looks like up to 64k ranges can be sent per trim, and each range can cover up to 64k blocks. So that is 2^32 blocks per trim command, or 2TiB of data per TRIM if the blocks are 512 bytes (which I'm guessing is the case for ATA but I didn't check). The command itself would be half a megabyte since each range is 64 bits. But if the kernel chops them up as you say that will certainly add overhead. The drive controller itself is probably the bigger bottleneck unless it is designed to do fast TRIMs. -- Rich
Re: [gentoo-user] Understanding fstrim...
On 13/04/2020 17:05, Rich Freeman wrote: And what takes time when doing a "large" TRIM is transmitting a _large_ list of blocks to the SSD via the TRIM command. That's why e.g. those ~6-7GiB trims I did just before (see my other mail) took a couple of seconds for 13GiB ~ 25M LBAs ~ a whole effin bunch of TRIM commands (no idea... wait, 1-4kB per TRIM and 4B/LBA is max. 1k LBAs/TRIM and for 25M LBAs you'll need minimum 25-100k TRIM commands... go figure;) no wonder it takes a second or few;) There is no reason that 100k TRIM commands need to take much time. Transmitting the commands is happening at SATA speeds at least. I'm not sure what the length of the data in a trim instruction is, but even if it were 10-20 bytes you could send 100k of those in 1MB, which takes <10ms to transfer depending on the SATA generation. Dare I say it ... buffer bloat? poor implementation? aiui, the spec says you can send a command "trim 1GB starting at block X". Snag is, the linux block size of 4KB means that it gets split into loads of trim commands, which then clogs up all the buffers ... Plus all too often the trim command is synchronous, so although it is pretty quick, the drive won't accept the next command until the previous one has completed. Cheers, Wol
Re: [gentoo-user] Understanding fstrim...
On Mon, Apr 13, 2020 at 11:41 AM David Haller wrote: > > First of all: "physical write blocks" in the physical flash are 128kB > or something in that size range, not 4kB or even 512B Yup, though I never claimed otherwise. I just made the generic statement that the erase blocks are much larger than the write blocks, even moreso than on a 4k hard drive. (The only time I mentioned 4k was in the context of hard drives, not SSDs.) > Anyway, a write to a single (used) logical 512B block > involves: > > 1. read existing data of the phy-block-group (e.g. 128KB) > 2. write data of logical block to the right spot of in-mem block-group > 3. write in-mem block-group to (a different, unused) phy-block-group > 4. update all logical block pointers to new phy-block-group as needed > 5. mark old phy-block-group as unused Yup. Hence my statement that my description was a simplification and that a real implementation would probably use extents to save memory. You're describing 128kB extents. However, there is no reason that the drive has to keep all the blocks in an erase group together, other than to save memory in the mapping layer. If it doesn't then it can modify a logical block without having to re-read adjacent logical blocks. > And what takes time when doing a "large" TRIM is transmitting a > _large_ list of blocks to the SSD via the TRIM command. That's why > e.g. those ~6-7GiB trims I did just before (see my other mail) took a > couple of seconds for 13GiB ~ 25M LBAs ~ a whole effin bunch of TRIM > commands (no idea... wait, 1-4kB per TRIM and 4B/LBA is max. 1k > LBAs/TRIM and for 25M LBAs you'll need minimum 25-100k TRIM > commands... go figure ;) no wonder it takes a second or few ;) There is no reason that 100k TRIM commands need to take much time. Transmitting the commands is happening at SATA speeds at least. I'm not sure what the length of the data in a trim instruction is, but even if it were 10-20 bytes you could send 100k of those in 1MB, which takes <10ms to transfer depending on the SATA generation. Now, the problem is the implementation on the drive. If the drive takes a long time to retire each command then that is what backs up the queue, and hence that is why the behavior depends a lot on firmware/etc. The drive mapping is like a filesystem and as we all know some filesystems are faster than others for various operations. Also as we know hardware designers often aren't optimizing for performance in these matters. > Oh, and yes, on rotating rust, all that does not matter. You'd just > let the data rot and write at 512B (or now 4kB) granularity. Well, > those 4k-but-512Bemulated drives (which is about all new ones by now I > think) have to do something like SSDs. But only on the 4kB level. Plus > the SMR shingling stuff of course. When will those implement TRIM? And that would be why I used 4k hard drives and SMR drives as an analogy. 4k hard drives do not support TRIM but as you (and I) pointed out, they're only dealing with 4k at a time. SMR drives sometimes do support TRIM. -- Rich
Re: [gentoo-user] Understanding fstrim...
Hello, On Mon, 13 Apr 2020, Rich Freeman wrote: >So, "trimming" isn't something a drive does really. It is a logical >command issued to the drive. > >The fundamental operations the drive does at the physical layer are: >1. Read a block >2. Write a block that is empty >3. Erase a large group of blocks to make them empty [..] >Now, in this hypothetical case here is how the drive handles a TRIM >command. If it gets the logical instruction "TRIM block 1" what it >does is: > >1. Look at the mapping table to determine that logical block 1 is at >physical block 123001. >2. Mark physical block 123001 as unused-but-dirty in the mapping table. > >That's all it does. There are four ways that a drive can get marked >as unused on an SSD: >1. Out of the box all blocks are unused-but-clean. (costs no >operations that you care about) >2. The trim command marks a block as unused-but-dirty. (costs no operations) >3. Block overwrites mark the old block as unused-but-dirty. (costs a >write operation, but you were writing data anyway) >4. Task 2 can mark blocks as unused-but-dirty. (costs a bunch of reads >and writes) > >Basically the goal of TRIM is to do more of #2 and less of #4 above, >which is an expensive read-write defragmentation process. Plus #4 >also increases drive wear since it involves copying data. Beautifully summarized Rich! But I'd like to add two little aspects: First of all: "physical write blocks" in the physical flash are 128kB or something in that size range, not 4kB or even 512B ... Haven't read, but looking enticing neither https://en.wikipedia.org/wiki/Write_amplification nor https://en.wikipedia.org/wiki/Trim_(computing) I hope they cover it ;) Anyway, a write to a single (used) logical 512B block involves: 1. read existing data of the phy-block-group (e.g. 128KB) 2. write data of logical block to the right spot of in-mem block-group 3. write in-mem block-group to (a different, unused) phy-block-group 4. update all logical block pointers to new phy-block-group as needed 5. mark old phy-block-group as unused And whatnot. And second: fstrim just makes the OS (via the Filesystem driver via the SATA/NVME/SCSI driver through some hoops), or the Filesystem when mounted with 'discard' via the drivers, tell the SSD one simple thing about logical blocks that a deleted file used to use (in the TRIM ATA/SCSI/SATA/NVME command, wikipedite for where TRIM is specced ;): "Hey, SSD, here's a list of LBAs (logical blocks) I no longer need. You may hencewith treat them as empty/unused." Without it, the SSD has no idea about those blocks being unneeded and treats blocks, once written to, as used blocks, doing the _tedious_ Copy-on-Write when a write hits one of those logical blocks, even if those were deleted on the filesystem level years ago... see above WP articles. Without TRIM, the SSD only gets to know the fact, when the driver (the FS) writes again to the same logical block ... With TRIM, the SSD-Controller knows what logical blocks it can treat as unused, and do much better wear-leveling. So, it's sort of a "trickle down 'unlink()' to the SSD"-feature. On the logical-block level, mind you. But for the SSD, that can be quite a "relief" regarding space for wear-leveling. And what takes time when doing a "large" TRIM is transmitting a _large_ list of blocks to the SSD via the TRIM command. That's why e.g. those ~6-7GiB trims I did just before (see my other mail) took a couple of seconds for 13GiB ~ 25M LBAs ~ a whole effin bunch of TRIM commands (no idea... wait, 1-4kB per TRIM and 4B/LBA is max. 1k LBAs/TRIM and for 25M LBAs you'll need minimum 25-100k TRIM commands... go figure ;) no wonder it takes a second or few ;) Oh, and yes, on rotating rust, all that does not matter. You'd just let the data rot and write at 512B (or now 4kB) granularity. Well, those 4k-but-512Bemulated drives (which is about all new ones by now I think) have to do something like SSDs. But only on the 4kB level. Plus the SMR shingling stuff of course. When will those implement TRIM? HTH, -dnh -- All Hardware Sucks and I do not consider myself to actually have any data until there's an offsite backup of it. -- Anthony de Boer
Re: [gentoo-user] Understanding fstrim...
On Mon, Apr 13, 2020 at 9:18 AM wrote: > > One quesion -- not to express any doubt of what you wrote Rich, but onlu > to check, whether I understand that detail or not: > > Fstrim "allows" the drive to trim ittself. The actual "trimming" is > done by the drive ittself without any interaction from the outside > of the SSD. > ... > On the one hand, the SSD drive keeps track of the information, what > blocks are used and unused. And trimming is done by the drive in > itsself. On the other hand trimming is "just giving the drive more > information about" > > What kind of information does the commandline tool fstrim transfers to > the SSD beside the command "fstrim yourself" (an ioctl, I think?), > which is needed to fstrim the blocks and what kind of information is > the SDD collecting itsself for this purpose? > So, "trimming" isn't something a drive does really. It is a logical command issued to the drive. The fundamental operations the drive does at the physical layer are: 1. Read a block 2. Write a block that is empty 3. Erase a large group of blocks to make them empty The key attribute of flash is that the physical unit of blocks that can be erased is much larger than the individual blocks that can be read/written. This is what leads to the need for TRIM. If SSDs worked like traditional hard drives where an individual block could be rewritten freely then there would be no need for trimming. Actually, drives with 4k sectors is a bit of an analogy though the cost for partial rewrites on a hard drive is purely a matter of performance and not longevity and the 4k sector issue can be solved via alignment. It is an analogous problem though, and SMR hard drives is a much closer analogy (really no different from 4k sectors except now the magnitude of the problem is MUCH bigger). Whether you use TRIM or not an SSD has to translate any logical operation into the three physical operations above. It also has to balance wear into this. So, if you don't use trim a logical instruction like "write this data to block 1" turns into: 1. Look at the mapping table to determine that logical block 1 is at physical block 123001. 2. Write the new contents of block 1 to physical block 823125 which is unused-but-clean. 3. Update the mapping table so that block 1 is at physical block 823125, and mark 123001 as unused-but-dirty. Then the controller would have two background tasks that it would run periodically: Task 1 - look for contiguous regions marked as unused-but-dirty. When one of the right size/alignment is identified erase it physically and mark it as unused-but-clean. Task 2 - if the amount of unused-but-clean space gets too low, then find areas that have fragmented unused-but-dirty space. The used space in those regions would get copied to new unused-but-clean blocks and remapped, and then task 1 will erase them and mark them as clean. This deals with fragmented unused space. Every SSD will do some variation on this whether you ever use the trim command as it is necessary for wear-leveling and dealing with the physical erase limitations. Now, in this hypothetical case here is how the drive handles a TRIM command. If it gets the logical instruction "TRIM block 1" what it does is: 1. Look at the mapping table to determine that logical block 1 is at physical block 123001. 2. Mark physical block 123001 as unused-but-dirty in the mapping table. That's all it does. There are four ways that a drive can get marked as unused on an SSD: 1. Out of the box all blocks are unused-but-clean. (costs no operations that you care about) 2. The trim command marks a block as unused-but-dirty. (costs no operations) 3. Block overwrites mark the old block as unused-but-dirty. (costs a write operation, but you were writing data anyway) 4. Task 2 can mark blocks as unused-but-dirty. (costs a bunch of reads and writes) Basically the goal of TRIM is to do more of #2 and less of #4 above, which is an expensive read-write defragmentation process. Plus #4 also increases drive wear since it involves copying data. Now this is all a bit oversimplified but I believe it is accurate as far as it illustrates the concept. A real drive probably groups logical blocks a bit more so that it doesn't need to maintain a 1:1 block mapping which seems like it would use a lot of space. Again, it is a bit like a filesystem so all the optimizations filesystems use like extents/etc would apply. At the physical level the principle is that the drive has to deal with the issue that reads/writes are more granular than erases, and everything else flows from that. The same issue applies to SMR hard drives, which were discussed on this list a while ago. -- Rich
Re: [gentoo-user] Understanding fstrim...
Hello,
On Mon, 13 Apr 2020, tu...@posteo.de wrote:
>On 04/13 11:06, Michael wrote:
>> On Monday, 13 April 2020 06:32:37 BST tu...@posteo.de wrote:
[..]
>My question are more driven by curiousty than by anxiety...
[..]
>For example [the fstrim manpage] says:
>"For most desktop and server systems a sufficient trimming frequency is
>once a week."
>
>...but why is this ok to do so? Are all PCs made equal? Are all
>use cases are equal? It even does not distinguish between SSD/Sata
>and SSD/NVMe(M2 in my case).
Observe your use pattern a bit and use 'fstrim -v' when you think it's
worth it, as it basically boils down to how much you delete *when*. If
you e.g.:
- constantly use the drive as a fast cache for video-editing etc.,
writing large files to the drive and later deleting them again
-> run fstrim daily or even mount with 'discard'-option
- write/delete somewhat regularly, e.g. as a system-drive and running
updates or emerge @world (esp. if you build on the SSD) e.g. weekly
they effectively are a write operation and a bunch of deletions. Or
if you do whatever other deletions somewhat regularly
-> run fstrim after each one or three such deletions, e.g. via
a weekly cronjob
- mostly write (if anything), rarely delete anything
-> run fstrim manually a few days after $some deletions have
accumulated or any other convenient time you can remember to
and are sure all deleted files can be gone, be it bi-weekly,
monthly, tri-montly, yearly, completly irregularly, whenever ;)
Choose anything in the range that fits _your_ use-pattern best,
considering capacity, free-space (no matter if on a partition or
unallocated) and what size was trimmed when running 'fstrim -v'...
Running that weekly (I'd suggest bi-weekly) 'fstrim' cronjob is not a
bad suggestion as a default I guess, but observe your use and choose
to deviate or not :)
My gut says to run fstrim if:
- it'd trim more than 5% (-ish) capacity
- it'd trim more than 20% (-ish) of the remaining "free" space
(including unallocated)
- it'd trim more than $n GiB (where $n may be anything ;)
whichever comes first (and the two latter can only be determined by
observation). No need to run fstrim after deleting just 1kiB. Or 1MiB.
Not that me lazybag adheres to that, read on if you will... ;)
FWIW:
I run fstrim a few times a year when I think of it and guesstimate I
did delete quite a bit in the meantime (much like I run fsck ;) ...
This usually trims a few GiB on my 128G drive:
# fdisk -u -l /dev/sda
Disk /dev/sda: 119.2 GiB, 128035676160 bytes, 250069680 sectors
Disk model: SAMSUNG SSD 830
[..]
Device Boot Start End Sectors Size Id Type
/dev/sda12048 109053951 109051904 52G 83 Linux
/dev/sda2 109053952 218105855 109051904 52G 83 Linux
(I did leave ~15GiB unpartitioned, and was too lazy to rectify that
yet, at the time I partitioned in 2012, for many (cheaper?) SSDs that
overprovisioning was still a good thing and 'duh intarweb' was quite
worse than today regarding the problem)...
So while I'm about it, I guess it's time to run fstrim (for the first
time this year IIRC) ...
# fstrim -v /sda1 ; fstrim -v /sda2 ## mountpoints mangled
/sda1: 7563407360 bytes were trimmed
/sda2: 6842478592 bytes were trimmed
# calc 'x=config("display",1); 7563407360/2^30; 6842478592/2^30'
~7.0
~6.4
So, my typical few GiB or about 12.8% disk capacity (summed) were
trimmed (oddly enough, it's always been in this 4-8GiB/partition
range). I probably should run fstrim a bit more often though, but then
again I've still got those unallocated 15G, so I guess I'm fine. And
that's with quite a large Gentoo system on /dev/sda2 and all its at
times large (like libreoffice, firefox, seamonkey, icedtea, etc.)
updates:
# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda252G 45G 3.8G 93% /
PORTAGE_TMPDIR, PORTDIR, (and distfiles and packages) are on other
HDDs though, so building stuff does not affect the SSD, only the
actual install (merge) and whatever else. But I've got /var/log/ on
the SSD on both systems (sda1/sda2).
While I'm at it:
# smartctl -A /dev/sda
[..]
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH [..] RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 010[..] 0
9 Power_On_Hours 0x0032 091 091 000[..] 43261
12 Power_Cycle_Count 0x0032 097 097 000[..] 2617
177 Wear_Leveling_Count 0x0013 093 093 000[..] 247
179 Used_Rsvd_Blk_Cnt_Tot 0x0013 100 100 010[..] 0
181 Program_Fail_Cnt_Total 0x0032 100 100 010[..] 0
182 Erase_Fail_Count_Total 0x0032 100 100 010[..] 0
183 Runtime_Bad_Block 0x0013 100 100 010[..] 0
187 Uncorrectable_Error_Cnt 0x0032 100 100 000[..] 0
190 Airflow_Temperature_Cel 0x0032 067 050 000[..] 33
195 ECC_Error_Rate 0x001a 200 200 000[..] 0
199
Re: [gentoo-user] Understanding fstrim...
On 04/13 08:18, Rich Freeman wrote: > On Mon, Apr 13, 2020 at 7:55 AM Michael wrote: > > > > I have noticed when prolonged fstrim takes place on an old SSD drive of mine > > it becomes unresponsive. As Rich said this is not because data is being > > physically deleted, only a flag is switched from 1 to 0 to indicate its > > availability for further writes. > > That, and it is going about it in a brain-dead manner. > > A modern drive is basically a filesystem of sorts. It just uses block > numbers instead of filenames, but it can be just as complex > underneath. > > And just as with filesystems there are designs that are really lousy > and designs that are really good. And since nobody sees the source > code or pays much attention to the hidden implementation details, it > is often not designed with your requirements in mind. > > I suspect a lot of SSDs are using the SSD-equivalent of FAT32 to > manage their block remapping. Some simple algorithm that gets the job > done but which doesn't perform well/etc. > > This makes me wonder if there would be a benefit from coming up with a > flash block layer of some sort that re-implements this stuff properly. > We have stuff like f2fs which does this at the filesystem level. > However, this might be going too far as this inevitably competes with > the filesystem layer on features/etc. > > Maybe what we need is something more like lvm for flash. It doesn't > try to be a filesystem. It just implements block-level storage > mapping one block device to a new block device. It might very well > implement a log-based storage layer. It would accept TRIM commands > and any other related features. It would then have a physical device > translation layer. Maybe it would be aware of different drive models > and their idiosyncrasies, so on some drives it might just be a NOOP > passthrough and on other drives it implements its own log-based > storage with batched trims on large contiguous regions, and so on. > Since it isn't a full POSIX filesystem it could be much simpler and > just focus on the problem it needs to solve - dealing with brain-dead > SSD controllers. > > -- > Rich > Hi Rich, hi Michael, THAT is information I like...now I start(!) to understand the "inner mechanics" of fstrim...thank you very much!!! :::))) One quesion -- not to express any doubt of what you wrote Rich, but onlu to check, whether I understand that detail or not: Fstrim "allows" the drive to trim ittself. The actual "trimming" is done by the drive ittself without any interaction from the outside of the SSD. You wrote: > > Now, the drive controller needs to keep track of which blocks are in > > use (which it does whether you use fstrim or not), and that data is > > probably stored in some kind of flash so that it is persistent, but > > presumably that is managed in such a manner that it is unlikely to > > fail before the rest of the drive fails. and: > > Fundamentally trimming is just giving the drive more information about > > the importance of the data it is storing. Just about any filesystem For me (due my beginners level of knowing the "behind the scene" things) this is kinda contradictionous. On the one hand, the SSD drive keeps track of the information, what blocks are used and unused. And trimming is done by the drive in itsself. On the other hand trimming is "just giving the drive more information about" What kind of information does the commandline tool fstrim transfers to the SSD beside the command "fstrim yourself" (an ioctl, I think?), which is needed to fstrim the blocks and what kind of information is the SDD collecting itsself for this purpose? Cheers! Meino >
Re: [gentoo-user] Understanding fstrim...
On Mon, Apr 13, 2020 at 7:55 AM Michael wrote: > > I have noticed when prolonged fstrim takes place on an old SSD drive of mine > it becomes unresponsive. As Rich said this is not because data is being > physically deleted, only a flag is switched from 1 to 0 to indicate its > availability for further writes. That, and it is going about it in a brain-dead manner. A modern drive is basically a filesystem of sorts. It just uses block numbers instead of filenames, but it can be just as complex underneath. And just as with filesystems there are designs that are really lousy and designs that are really good. And since nobody sees the source code or pays much attention to the hidden implementation details, it is often not designed with your requirements in mind. I suspect a lot of SSDs are using the SSD-equivalent of FAT32 to manage their block remapping. Some simple algorithm that gets the job done but which doesn't perform well/etc. This makes me wonder if there would be a benefit from coming up with a flash block layer of some sort that re-implements this stuff properly. We have stuff like f2fs which does this at the filesystem level. However, this might be going too far as this inevitably competes with the filesystem layer on features/etc. Maybe what we need is something more like lvm for flash. It doesn't try to be a filesystem. It just implements block-level storage mapping one block device to a new block device. It might very well implement a log-based storage layer. It would accept TRIM commands and any other related features. It would then have a physical device translation layer. Maybe it would be aware of different drive models and their idiosyncrasies, so on some drives it might just be a NOOP passthrough and on other drives it implements its own log-based storage with batched trims on large contiguous regions, and so on. Since it isn't a full POSIX filesystem it could be much simpler and just focus on the problem it needs to solve - dealing with brain-dead SSD controllers. -- Rich
Re: [gentoo-user] Understanding fstrim...
On Monday, 13 April 2020 12:39:11 BST Rich Freeman wrote: > On Mon, Apr 13, 2020 at 1:32 AM wrote: > > fstrim clears blocks, which currently are not in use and which > > contents is != 0. > > > >... > > > > BUT: Clearing blocks is an action, which includes writes to the cells of > > the SSD. > > I see a whole bunch of discussion, but it seems like many here don't > actually understand what fstrim actually does. > > It doesn't "clear" anything, and it doesn't care what the contents of > a block are. It doesn't write to the cells of the SSD per se. > > It issues the TRIM command to the drive for any unused blocks (or a > subset of them if you use non-default options). It doesn't care what > the contents of the blocks are when it does so - it shouldn't even try > to read the blocks to know what their content is. > > Trimming a block won't clear it at all. It doesn't write to the cells > of the SSD either - at least not the ones being trimmed. It just > tells the drive controller that the blocks are no longer in use. > > Now, the drive controller needs to keep track of which blocks are in > use (which it does whether you use fstrim or not), and that data is > probably stored in some kind of flash so that it is persistent, but > presumably that is managed in such a manner that it is unlikely to > fail before the rest of the drive fails. > > On a well-implemented drive trimming actually REDUCES writes. When > you trim a block the drive controller will stop trying to preserve its > contents. If you don't trim it, then the controller will preserve its > contents. Preserving the content of unused blocks necessarily > involves more writing to the drive than just treating as > zero-fillled/etc. > > Now, where you're probably getting at the concept of clearing and > zeroing data is that if you try to read a trimmed block the drive > controller probably won't even bother to read the block from the ssd > and will just return zeros. Those zeros were never written to flash - > they're just like a zero-filled file in a filesystem. If you write a > bazillion zeros to a file on ext4 it will just record in the > filesystem data that you have a bunch of blocks of zero and it won't > allocate any actual space on disk - reading that file requires no > reading of the actual disk beyond the metadata because they're not > stored in actual extents. Indeed blocks are more of a virtual thing > on an SSD (or even hard drive these days), so if a logical block isn't > mapped to a physical storage area there isn't anything to read in the > first place. > > However, when you trimmed the file the drive didn't go find some area > of flash and fill it with zeros. It just marked it as unused or > removed its logical mapping to physical storage. > > In theory you should be able to use discard or trim the filesystem > every 5 minutes with no negative effects at all. In theory. However, > many controllers (especially old ones) aren't well-implemented and may > not handle this efficiently. A trim operation is still an operation > the controller has to deal with, and so deferring it to a time when > the drive is idle could improve performance, especially for drives > that don't do a lot of writes. If a drive has a really lousy > controller then trims might cause its stupid firmware to do stupid > things. However, this isn't really anything intrinsic to the concept > of trimming. > > Fundamentally trimming is just giving the drive more information about > the importance of the data it is storing. Just about any filesystem > benefits from having more information about what it is storing if it > is well-implemented. In a perfect world we'd just enable discard on > our mounts and be done with it. > > I'd probably just look up the recommendations for your particular > drive around trimming and follow those. Somebody may have benchmarked > it to determine how brain-dead it is. If you bought a more name-brand > SSD you're probably more likely to benefit from more frequent > trimming. > > I'm personally using zfs which didn't support trim/discard until very > recently, and I'm not on 0.8 yet, so for me it is a bit of a moot > point. I plan to enable it once I can do so. What Rich said, plus: I have noticed when prolonged fstrim takes place on an old SSD drive of mine it becomes unresponsive. As Rich said this is not because data is being physically deleted, only a flag is switched from 1 to 0 to indicate its availability for further writes. As I understand the firmware performs wear-leveling when it needs to in the HPA allocated blocks, rather than waiting for the user/OS to run fstrim to obtain some more 'free' space. signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Understanding fstrim...
On Mon, Apr 13, 2020 at 1:32 AM wrote: > > fstrim clears blocks, which currently are not in use and which > contents is != 0. >... > BUT: Clearing blocks is an action, which includes writes to the cells of > the SSD. I see a whole bunch of discussion, but it seems like many here don't actually understand what fstrim actually does. It doesn't "clear" anything, and it doesn't care what the contents of a block are. It doesn't write to the cells of the SSD per se. It issues the TRIM command to the drive for any unused blocks (or a subset of them if you use non-default options). It doesn't care what the contents of the blocks are when it does so - it shouldn't even try to read the blocks to know what their content is. Trimming a block won't clear it at all. It doesn't write to the cells of the SSD either - at least not the ones being trimmed. It just tells the drive controller that the blocks are no longer in use. Now, the drive controller needs to keep track of which blocks are in use (which it does whether you use fstrim or not), and that data is probably stored in some kind of flash so that it is persistent, but presumably that is managed in such a manner that it is unlikely to fail before the rest of the drive fails. On a well-implemented drive trimming actually REDUCES writes. When you trim a block the drive controller will stop trying to preserve its contents. If you don't trim it, then the controller will preserve its contents. Preserving the content of unused blocks necessarily involves more writing to the drive than just treating as zero-fillled/etc. Now, where you're probably getting at the concept of clearing and zeroing data is that if you try to read a trimmed block the drive controller probably won't even bother to read the block from the ssd and will just return zeros. Those zeros were never written to flash - they're just like a zero-filled file in a filesystem. If you write a bazillion zeros to a file on ext4 it will just record in the filesystem data that you have a bunch of blocks of zero and it won't allocate any actual space on disk - reading that file requires no reading of the actual disk beyond the metadata because they're not stored in actual extents. Indeed blocks are more of a virtual thing on an SSD (or even hard drive these days), so if a logical block isn't mapped to a physical storage area there isn't anything to read in the first place. However, when you trimmed the file the drive didn't go find some area of flash and fill it with zeros. It just marked it as unused or removed its logical mapping to physical storage. In theory you should be able to use discard or trim the filesystem every 5 minutes with no negative effects at all. In theory. However, many controllers (especially old ones) aren't well-implemented and may not handle this efficiently. A trim operation is still an operation the controller has to deal with, and so deferring it to a time when the drive is idle could improve performance, especially for drives that don't do a lot of writes. If a drive has a really lousy controller then trims might cause its stupid firmware to do stupid things. However, this isn't really anything intrinsic to the concept of trimming. Fundamentally trimming is just giving the drive more information about the importance of the data it is storing. Just about any filesystem benefits from having more information about what it is storing if it is well-implemented. In a perfect world we'd just enable discard on our mounts and be done with it. I'd probably just look up the recommendations for your particular drive around trimming and follow those. Somebody may have benchmarked it to determine how brain-dead it is. If you bought a more name-brand SSD you're probably more likely to benefit from more frequent trimming. I'm personally using zfs which didn't support trim/discard until very recently, and I'm not on 0.8 yet, so for me it is a bit of a moot point. I plan to enable it once I can do so. -- Rich
Re: [gentoo-user] Understanding fstrim...
Have your backup cron job call fstrim once everything is safely backed up? Well, yes, but that's beside the point. What I really wanted to stress was that mounting an SSD-backed filesystem with "discard" has effects on the ability to recover deleted data. Normally it's not a problem, but shit happens -- and when it happens on such a filesystem don't waste time with recovery tools, as all you'll get back are files full of 0xFFs. andrea
Re: [gentoo-user] Understanding fstrim...
Hi Michael, thank you for replying to my questions! :) On 04/13 11:06, Michael wrote: > On Monday, 13 April 2020 06:32:37 BST tu...@posteo.de wrote: > > Hi, > > > > From the list I already have learned, that most of my concerns regarding > > the lifetime and maintainance to prolong it are without a > > reason. > > Probably your concerns about SSD longevity are without a reason, but keep up > to date backups just in case. ;-) ...of course! :) My question are more driven by curiousty than by anxiety... > > Nonetheless I am interested in the technique as such. > > > > My SSD (NVme/M2) is ext4 formatted and I found articles on the > > internet, that it is neither a good idea to activate the "discard" > > option at mount time nor to do a fstrim either at each file deletion > > no triggered by a cron job. > > Beside what the interwebs say about fstrim, the man page provides good > advice. > They recommend running a cron job once a week, for most desktop and server > implementations. ...but it neither explains why to do so nor does it explain the technical background. For example it saus: "For most desktop and server systems a sufficient trimming frequency is once a week." ...but why is this ok to do so? Are all PCs made equal? Are all use cases are equal? It even does not distinguish between SSD/Sata and SSD/NVMe(M2 in my case). These are the points, where my curiousty kicks in and I am starting to ask questions. :) > > Since there seems to be a "not so good point in time", when to do a > > fstrim, I think there must also be a point in time, when it is quite > > right to fstrim the mu SSD. > > > > fstrim clears blocks, which currently are not in use and which > > contents is != 0. > > > > The more unused blocks there are, which has a contents != 0, the > > lesser the count of blocks is, which the wear leveling algorithm can > > use for its purpose. > > The wear levelling mechanism is using the HPA as far as I know, although you > can always overprovision it.[1] For example: Take an SSD with 300 GB user-useable space. To over-overprovisioning the device the user decides to partitioning only the half of the disk and format it. The rest is left untouched in "nowhere land". Now the controller has a lot of space to shuffle data around. Fstrim only works on the mounted part of the SSD. So the used blocks in "nowhere land" remain...unfstrimmed? To not to use all available space for the partitions is a hint I found online...and then I asked me the question above... If what I read online is wrong my assumptions are wrong...which isn't reassuring either. > > > That leads to the conclusion: to fstrim as often as possible, to keep the > > count of empty blocks as high as possible. > > Not really. Why would you need the count of empty blocks as high as > possible, Unused blocks with data cannot be used for wearleveling. Suppose you have a total amount of 100 block, 50 blocks are used, 25 are unused and empty, 25 are unused and filled with former data. In this case only 25 blocks are available to spread the next write operation. After fstrim 50 blocks would be available again and the same amount of writes could now be spread over 50 sectors. At least that is what I read online... > unless you are about to right some mammoth file and *need* to use up every > available space possible on this disk/partition? > > > > BUT: Clearing blocks is an action, which includes writes to the cells of > > the SSD. > > > > Which is not that nice. > > It's OK, as long as you are not over-writing cells which do not need to be > overwritten. Cells with deleted data will be overwritten as some point. > > > > Then, do a fstrim just in the moment, when there is no useable block > > left. > > Why leave it at the last moment and incur a performance penalty while waiting > for fstrim to complete? Performance is not my concern (at least in the moment ;) ). I try to fully understand the mechanisms here, since what I read online is not without contradictions... > > Then the wear-leveling algorithm is already at its limits. > > > > Which is not that nice either. > > > > The truth - as so often - is somewhere in between. > > > > Is it possible to get an information from the SSD, how many blocks are > > in the state of "has contents" and "is unused" and how many blocks are > > in the state of "has *no* contents" and "is unused"? > > > > Assuming this information is available: Is it possible to find the > > sweat spot, when to fstrim SSD? > > I humbly suggest you may be over-thinking something a cron job running fstrim > once a week, or once a month, or twice a month would take care of without you > knowing or worrying about. To technically overthink problems is a vital part of my profession and exactly what I am asked for. I cannot put this behaviour down so easily. :) >From my experience there aren't too manu questions, Michael, there is often only a lack of related answers. >
Re: [gentoo-user] Understanding fstrim...
On Monday, 13 April 2020 06:32:37 BST tu...@posteo.de wrote: > Assuming this information is available: Is it possible to find the > sweat spot, when to fstrim SSD? This crontab entry is my compromise: 15 3 */2 * * /sbin/fstrim -a It does assume I'll be elsewhere at 03:15, of course. -- Regards, Peter.
Re: [gentoo-user] Understanding fstrim...
On Monday, 13 April 2020 06:32:37 BST tu...@posteo.de wrote: > Hi, > > From the list I already have learned, that most of my concerns regarding > the lifetime and maintainance to prolong it are without a > reason. Probably your concerns about SSD longevity are without a reason, but keep up to date backups just in case. ;-) > Nonetheless I am interested in the technique as such. > > My SSD (NVme/M2) is ext4 formatted and I found articles on the > internet, that it is neither a good idea to activate the "discard" > option at mount time nor to do a fstrim either at each file deletion > no triggered by a cron job. Beside what the interwebs say about fstrim, the man page provides good advice. They recommend running a cron job once a week, for most desktop and server implementations. > Since there seems to be a "not so good point in time", when to do a > fstrim, I think there must also be a point in time, when it is quite > right to fstrim the mu SSD. > > fstrim clears blocks, which currently are not in use and which > contents is != 0. > > The more unused blocks there are, which has a contents != 0, the > lesser the count of blocks is, which the wear leveling algorithm can > use for its purpose. The wear levelling mechanism is using the HPA as far as I know, although you can always overprovision it.[1] > That leads to the conclusion: to fstrim as often as possible, to keep the > count of empty blocks as high as possible. Not really. Why would you need the count of empty blocks as high as possible, unless you are about to right some mammoth file and *need* to use up every available space possible on this disk/partition? > BUT: Clearing blocks is an action, which includes writes to the cells of > the SSD. > > Which is not that nice. It's OK, as long as you are not over-writing cells which do not need to be overwritten. Cells with deleted data will be overwritten as some point. > Then, do a fstrim just in the moment, when there is no useable block > left. Why leave it at the last moment and incur a performance penalty while waiting for fstrim to complete? > Then the wear-leveling algorithm is already at its limits. > > Which is not that nice either. > > The truth - as so often - is somewhere in between. > > Is it possible to get an information from the SSD, how many blocks are > in the state of "has contents" and "is unused" and how many blocks are > in the state of "has *no* contents" and "is unused"? > > Assuming this information is available: Is it possible to find the > sweat spot, when to fstrim SSD? I humbly suggest you may be over-thinking something a cron job running fstrim once a week, or once a month, or twice a month would take care of without you knowing or worrying about. Nevertheless, if the usage of your disk/partitions is variable and one week you may fill it up with deleted data, while for the rest of the month you won't even touch it, there's SSDcronTRIM, a script I've been using for a while.[2] [1] https://www.thomas-krenn.com/en/wiki/SSD_Over-provisioning_using_hdparm [2] https://github.com/chmatse/SSDcronTRIM signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Understanding fstrim...
On Mon, 13 Apr 2020 11:22:47 +0200, Andrea Conti wrote: > I have no desire to enter the whole performance/lifetime debate; I'd > just like to point out that one very real consequence of using fstrim > (or mounting with the discard option) that I haven't seen mentioned > often is that it makes the contents of any removed files Truly Gone(tm). > > No more extundelete to save your back when you mistakenly rm something > that you haven't backed up for a while... Have your backup cron job call fstrim once everything is safely backed up? -- Neil Bothwick Life's a cache, and then you flush... pgpQgbcxpCa66.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Understanding fstrim...
> My SSD (NVme/M2) is ext4 formatted and I found articles on the > internet, that it is neither a good idea to activate the "discard" > option at mount time nor to do a fstrim either at each file deletion > no triggered by a cron job. I have no desire to enter the whole performance/lifetime debate; I'd just like to point out that one very real consequence of using fstrim (or mounting with the discard option) that I haven't seen mentioned often is that it makes the contents of any removed files Truly Gone(tm). No more extundelete to save your back when you mistakenly rm something that you haven't backed up for a while... andrea
[gentoo-user] Understanding fstrim...
Hi, >From the list I already have learned, that most of my concerns regarding the lifetime and maintainance to prolong it are without a reason. Nonetheless I am interested in the technique as such. My SSD (NVme/M2) is ext4 formatted and I found articles on the internet, that it is neither a good idea to activate the "discard" option at mount time nor to do a fstrim either at each file deletion no triggered by a cron job. Since there seems to be a "not so good point in time", when to do a fstrim, I think there must also be a point in time, when it is quite right to fstrim the mu SSD. fstrim clears blocks, which currently are not in use and which contents is != 0. The more unused blocks there are, which has a contents != 0, the lesser the count of blocks is, which the wear leveling algorithm can use for its purpose. That leads to the conclusion: to fstrim as often as possible, to keep the count of empty blocks as high as possible. BUT: Clearing blocks is an action, which includes writes to the cells of the SSD. Which is not that nice. Then, do a fstrim just in the moment, when there is no useable block left. Then the wear-leveling algorithm is already at its limits. Which is not that nice either. The truth - as so often - is somewhere in between. Is it possible to get an information from the SSD, how many blocks are in the state of "has contents" and "is unused" and how many blocks are in the state of "has *no* contents" and "is unused"? Assuming this information is available: Is it possible to find the sweat spot, when to fstrim SSD? Cheers! Meino
