On Mon, Aug 3, 2020 at 8:24 AM Dale <[email protected]> wrote: > > In the past, I've never seen the drive on the larger files be that slow even > toward the end. Generally, it stays pretty close to 180MBs/sec or so which > is what I usually get with PMR drives.
Yeah, just hard to be certain without ditching the filesystem layer or doing some kind of comparison. The difference in write speed across the drive on recent drives I've gotten is more pronounced than I've seen on other drives, but these drives tend to be around 12TB. Definitely the thing to watch out for is a big drop in transfer rate once a large number of blocks have been transferred continuously, and then performance returns after you let the drive thrash for a while. I've seen complaints of zfs rebuilds going from hours/days to weeks/months in length, so it isn't just a 50% drop when you're doing worst-case access patterns. On the other hand I hear that mdadm isn't so bad, so if the writes are sequential the drive might be better at skipping the cache, and maybe zfs just does its rebuild non-sequentially (which isn't really ideal anyway). I haven't really dug into the guts of how zfs metadata works, but with btrfs I believe the chunks are basically their own layer, and the filesystem can scrub them without really any care about what files are stored in them. That allows them to be easily scrubbed sequentially. When I did rebuilds on btrfs they tended to run at about the max throughput of the drives as long as there wasn't any other access going on. It can also do read-only scrubs to check data integrity sequentially across the disk, which suggests the checksums are stored at a lower layer and so the data can be verified without worrying about file fragmentation and so on. This layering also lets btrfs switch "RAID modes" on the fly with half of the disk being RAID1 and half the disk being RAID5 and so on - each region of a disk is independent from the others and so mode changes only impact new regions until you do a full rebalance to rewrite everything. Of course, zfs has its own advantages. -- Rich
