> Le 10/04/2026 15:19 CEST, Anthony D'Atri <[email protected]> a écrit : > > > > > On that we agree: WAL+DB dead is a dead OSD. > > I've seen a speculative outline on recovering the OSD but have not seen > someone try.
Which means, for all intents and purposes, no block.db = stopped osd = no more write to block = no possibility of data loss. > > > And it's indeed a fair > > tradeoff: I'd expect the bad backend drives to die long before > > Spinners often have a lower TBW and AFR than even low-endurance SSDs. > > > > the enterprise write-intensive drives > > Do you find that you strictly need high endurance SSDs? > 99% of enterprise SSDs never use more than 15% of their rated endurance. > These days write-intensive usually means Optane-replacement SLC. > More common here is mixed-use, which is typically the very same hardware as a > "read intensive" SSD just with more overprovisioning. > If you really need more endurance than comes out of the box with an RI > device, you can often adjust the OP yourself at much lower cost. Oh I don't need the endurance. Even the IOPS I don't need. But given the shitshow the drives I have are, I wanted to make super duper sure the drives I got for wal+db could never ever be blamed. Even at the current market price, It ended up costing ~€700 for 8x400GB of enterprise write-intensive drives, which was marginally more expensive than the MU or RI drives (like less than €100 in total), so I'd say it's worth it. It's now the price brokers ask for a *single* enterprise 1.92TB drive. At least I know I cannot blame the hardware. And since I'm buying refurb, I reasoned WI drives would have more life left in them from the previous owner(s), as the overcommit is stupidly high on those (combined with SLC). > > I have for wal+db. (or, in the case of > > spinning rust, I'd expect a mechanical failure long before a flash > > failure). > > Most SSD failures are firmware and can be fixed in-situ. I've had to replace > 4000 HDDs with a design flaw - scattered around the world. Nightmare > > > Thus, in the end, it's no different from a dedicated wal+db iff wal+db > > stays dedicated: I've seen people suggest to keep wal+db colocated if > > there's a fast flash in front of the drive, as wal+db would essentially > > always be hot in the cache. > > Indeed I've read studies that claim a superset of benefits from a transparent > cache compared to just WAL+DB. I can definitely see why. The point however, was that having wal+db dedicated meant osd would stop using the drive if the fast one died. In terms of performance I'd expect no worse than with a cached colocated wal+db. However, we've now established that, for wal+db to be transparently cached, we'd need bcache to avoid data loss. dm-cache needs a way to block osd from using the drive if the cache is missing. > > (note: I know that the OSD thinks it directly talks to the crap drives, > > Reads are always served from the slow device, and writes over a certain size > go there directly. My understanding from dm-cache docs was that no, in plain dm-cache, reads hit the cache drive. For the behavior you describe (only writes use the cache), you'd need dm-writecache, which is another mode for dm-cache. As for the write size, I guess it's configurable, but I don't think it'll be an issue for my case: what I'm after is preventing the constant chatter or small random writes that hit the drive with a sustained workload. The occasional big write should not matter too much. > I've seen use-cases where the hassle of WAL+DB offload doesn't actually make > an appreciable improvement. I guess it really depends on the drive you have. For HDDs it's a no brainer as far as I understand it, but for SSDs you'd hit diminishing returns, even with NVMe WAL+DB. It's a case where transparent caching of block with an NVMe would make an improvement where WAL+DB does not. Thanks for all the info! Alexis _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
