Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Mon, 2022-11-14 at 15:08 -0500, Michael Stone wrote: > On Mon, Nov 14, 2022 at 08:40:47PM +0100, hw wrote: > > Not really, it was just an SSD. Two of them were used as cache and they > > failed > > was not surprising. It's really unfortunate that SSDs fail particulary fast > > when used for purposes they can be particularly useful for. > > If you buy hard drives and use them in the wrong application, they also > fail quickly. And? > And, again, you weren't using the right SSD so it *wasn't* > particularly useful. Sure it was. An SSD that is readily available and relatively inexpensive can be very useful and be the right one for the purpose. Another one that isn't readily available and relatively expensive may last longer, and that doesn't mean that it's the right one for the purpose. > But at this point you seem to just want to argue > in circles for no reason, so like others I'm done with this waste of > time. IIRC, you're the one trying to argue against experience, claiming that the experience wasn't how it was while it was as it was, and that conclusions drawn from the experience are invalid while they aren't.
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
hw writes: On Fri, 2022-11-11 at 21:26 +0100, Linux-Fan wrote: > hw writes: > > > On Thu, 2022-11-10 at 23:05 -0500, Michael Stone wrote: > > > On Thu, Nov 10, 2022 at 06:55:27PM +0100, hw wrote: > > > > On Thu, 2022-11-10 at 11:57 -0500, Michael Stone wrote: > > > > > On Thu, Nov 10, 2022 at 05:34:32PM +0100, hw wrote: > > > > > > And mind you, SSDs are *designed to fail* the sooner the more > > > > > > data you write to > > > > > > them. They have their uses, maybe even for storage if you're so > > > > > > desperate, but not for backup storage. > > [...] > > > Why would anyone use SSDs for backups? They're way too expensive for > > that. > > I actually do this for offsite/portable backups because SSDs are shock > resistant (dont lose data when being dropped etc.). I'd make offsite backups over internet. If you can afford SSDs for backups, well, why not. Yes, I do offsite backup over Internet too. Still, an attacker could possibly delete those from my running system. Not so much for the detached separate portable SSD. > The most critical thing to acknowledge about using SSDs for backups is > that the data retention time of SSDs (when not powered) is decreasing with each > generation. Do you mean each generation of SSDs or of backups? What do manufacturers say how long you can store an SSD on a shelf before the data on it has degraded? Generations of SSDs. In the early days, a shelf life in the magnitude of years was claimed. Later on, most datasheets I have seen have been lacking in this regard. [...] > The small (drive size about 240GB) ones I use for backup are much less > durable. You must have quite a lot of them. That gets really expensive. Two: I write the new backup to the one I have here, then carry it to the offsite location and take back the old copy from there. I do not these SSD's prices but it weren't expensive units at the time. > For one of them, the manufacturer claims 306TBW, the other has > 360 TBW specified. I do not currently know how much data I have written to > them already. As you can see from the sizes, I backup only a tiny subset > of the data to SSDs i.e. the parts of my data that I consider most critical > (VM images not being among them...). Is that because you have them around anyway because they were replaced with larger ones, or did you actually buy them to put backups on them? I still had them around: I started my backups with 4 GiB CF cards for backup, then quickly upgraded to 16 GiB cards all bought specifically for the backup purposes. Then I upgraded to 32 GB cards. Somewhere around that time I equipped my main system with dual 240 GB SSDs. Later, I upgraded to 2 TB SSDs with the intention to not only run the OS off the SSDs, but also the VMs. By this, I got the small SSDs out, ready to serve the increased need for backup storage since 32 GB CF cards were no longer feasible... Nowdays, my “important data backup” is still close to the historical 32 GiB limit although slightly above it (it ranges between 36 and 46 GiB or such). There are large amounts of free space on the backup SSDs filled by (1) a live system to facilitate easy restore and (2) additional, less important data (50 GiB or so). [...] > > There was no misdiagnosis. Have you ever had a failed SSD? They > > usually just disappear. I've had one exception in which the SDD at [...] > Just for the record I recall having observed this once in a very similar > fashion. It was back when a typical SSD size was 60 GiB. By now we should > mostly be past this “SSD fails early with controller fault” issues. It can > still happen and I still expect SSDs to fail with less notice compared to > HDDs. Why did they put bad controllers into the SSDs? Maybe because the good ones were to expensive at the time? Maybe because the manufacturers were yet to acquire good experience on how to produce them reliably. I can only speculate. [...] YMMV Linux-Fan öö pgpOoGLUZ_I5l.pgp Description: PGP signature
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Fri, 2022-11-11 at 21:26 +0100, Linux-Fan wrote: > hw writes: > > > On Thu, 2022-11-10 at 23:05 -0500, Michael Stone wrote: > > > On Thu, Nov 10, 2022 at 06:55:27PM +0100, hw wrote: > > > > On Thu, 2022-11-10 at 11:57 -0500, Michael Stone wrote: > > > > > On Thu, Nov 10, 2022 at 05:34:32PM +0100, hw wrote: > > > > > > And mind you, SSDs are *designed to fail* the sooner the more data > > you > > > > > > write > > > > > > to > > > > > > them. They have their uses, maybe even for storage if you're so > > > > > > desperate, > > > > > > but > > > > > > not for backup storage. > > [...] > > > Why would anyone use SSDs for backups? They're way too expensive for that. > > I actually do this for offsite/portable backups because SSDs are shock > resistant (dont lose data when being dropped etc.). I'd make offsite backups over internet. If you can afford SSDs for backups, well, why not. > The most critical thing to acknowledge about using SSDs for backups is that > the data retention time of SSDs (when not powered) is decreasing with each > generation. Do you mean each generation of SSDs or of backups? What do manufacturers say how long you can store an SSD on a shelf before the data on it has degraded? > Write endurance has not become critical in any of my SSD uses so far. > Increasing workloads have also resulted in me upgrading the SSDs. So far I > always upgraded faster than running into the write endurance limits. I do > not use the SSDs as caches but as full-blown file system drives, though. I use them as system disks because they don't mind being switched off and on and partly because they don't need as much electricity as hard disks. If it wasn't for that, I'd use hard disks for system disks. I don't use them for storage, they're way too small and expensive for that. Fortunately, system disks can be small; the data is on the server anyway. There are exceptions, like hard drives suck for laptops and SSDs are much better for them, and things that greatly benefit from low latencies, lots of IOOPs, high transfer rates. > On the current system, the SSDs report having written about 14 TB and are > specified by the manufacturer for an endurance of 6300 TBW (drive size is 4 > TB). Wow you have expensive drives. > The small (drive size about 240GB) ones I use for backup are much less > durable. You must have quite a lot of them. That gets really expensive. > For one of them, the manufacturer claims 306TBW, the other has > 360 TBW specified. I do not currently know how much data I have written to > them already. As you can see from the sizes, I backup only a tiny subset of > the data to SSDs i.e. the parts of my data that I consider most critical (VM > images not being among them...). Is that because you have them around anyway because they were replaced with larger ones, or did you actually buy them to put backups on them? > [...] > > > There was no misdiagnosis. Have you ever had a failed SSD? They usually > > just > > disappear. I've had one exception in which the SDD at first only sometimes > > disappeared and came back, until it disappeared and didn't come back. > > [...] > > Just for the record I recall having observed this once in a very similar > fashion. It was back when a typical SSD size was 60 GiB. By now we should > mostly be past this “SSD fails early with controller fault” issues. It can > still happen and I still expect SSDs to fail with less notice compared to > HDDs. Why did they put bad controllers into the SSDs? > When I had my first (and so far only) disk failure (on said 60G SSD) I > decided to: > > * Retain important data on HDDs (spinning rust) for the time being > > * and also implement RAID1 for all important drives > > Although in theory running two disks instead of one should increase the > overall chance of having one fail, no disks failed after this change so > far. I don't have any disks that aren't important. Even when the data can be recovered, it's not worth the trouble not to use redundancy. I consider redundacy a requirement, there is no storing anything on a single disk. I'd only tolerate it for backups when there are multiple backups when it can't be avoided. Sooner or later, a disk will fail.
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Mon, Nov 14, 2022 at 08:40:47PM +0100, hw wrote: Not really, it was just an SSD. Two of them were used as cache and they failed was not surprising. It's really unfortunate that SSDs fail particulary fast when used for purposes they can be particularly useful for. If you buy hard drives and use them in the wrong application, they also fail quickly. And, again, you weren't using the right SSD so it *wasn't* particularly useful. But at this point you seem to just want to argue in circles for no reason, so like others I'm done with this waste of time.
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Fri, 2022-11-11 at 14:48 -0500, Michael Stone wrote: > On Fri, Nov 11, 2022 at 07:15:07AM +0100, hw wrote: > > There was no misdiagnosis. Have you ever had a failed SSD? They usually > > just > > disappear. > > Actually, they don't; that's a somewhat unusual failure mode. What else happens? All the ones I have seen failing had disappeared. > [...] > I've had way more dead hard drives, which is typical. Because there were more hard drives than SSDs? > > There was no "not normal" territory, either, unless maybe you consider ZFS > > cache > > as "not normal". In that case, I would argue that SSDs are well suited for > > such > > applications because they allow for lots of IOOPs and high data transfer > > rates, > > and a hard disk probably wouldn't have failed in place of the SSD because > > they > > don't wear out so quickly. Since SSDs are so well suited for such purposes, > > that can't be "not normal" territory for them. Perhaps they just need to be > > more resilient than they are. > > You probably bought the wrong SSD. Not really, it was just an SSD. Two of them were used as cache and they failed was not surprising. It's really unfortunate that SSDs fail particulary fast when used for purposes they can be particularly useful for. > SSDs write in erase-block units, > which are on the order of 1-4MB. If you're writing many many small > blocks (as you would with a ZFS ZIL cache) there's significant write > amplification. For that application you really need a fairly expensive > write-optimized SSD, not a commodity (read-optimized) SSD. If you can get one you can use one. The question is if it's worthwhile to spend the extra money for special SSDs which aren't readily available or if it's better to just replace common ones which are readily available from time to time. > (And in fact, > SSD is *not* ideal for this because the data is written sequentially and > basically never read so low seek times aren't much benefit; NVRAM is > better suited.) if you can get that > If you were using it for L2ARC cache then mostly that > makes no sense for a backup server. Without more details it's really > hard to say any more. Honestly, even with the known issues of using > commidity SSD for SLOG I find it really hard to believe that your > backups were doing enough async transactions for that to matter--far > more likely is still that you simply got a bad copy, just like you can > get a bad hd. Sometimes you get a bad part, that's life. Certainly not > something to base a religion on. > That can happen one way or another. The last SSD that failed was used as a system disk in a Linux server in a btrfs mirror. Nothing much was written to it. > > Considering that, SSDs generally must be of really bad quality for that to > > happen, don't you think? > > No, I think you're making unsubstantiated statements, and I'm mostly > trying to get better information on the record for others who might be > reading. I didn't keep detailed records and don't remember all the details, so the better information you're looking for is not available. I can say that SSDs failed about the same as HDDs because that is my experience, and that's enough for me. You need to understand what experience is and what it means.
Re: Sorry for the misattribution [was: ZFS performance (was: Re: deduplicating file systems: VDO] withDebian?)
On Sat, 2022-11-12 at 07:27 +0100, to...@tuxteam.de wrote: > On Fri, Nov 11, 2022 at 07:22:19PM +0100, to...@tuxteam.de wrote: > > [...] > > > I think what hede was hinting at was that early SSDs had a (pretty) > > limited number of write cycles [...] > > As was pointed out to me, the OP wasn't hede. It was hw. Sorry for the > mis-attribution. > > Cheers What I was saying is something like that the number of failures out of some number of disks doesn't show the numbers of storage space failing (or however you want to call it). For example, when you have 100 SSDs with 500GB each and 2 of them failed, then 49TB of storage survived. With hard disks, you may have a 100 of them with 16TB each, and when 2 of them failed, 1568TB of storage have survived. That would mean that the "failure rate" of SSDs is 32 times higher than the one of hard disks. I don't know what the actual numbers are. Just citing some report saying 2/1518 vs. 44/1669 failures of SSDs vs. hard disks is meaningless, especially when these disks were used for different things. If the average SDD size was 1TB and the average HDD size was 16TB, then that would mean that the actual survival rate of storage on hard disks is over 17 times higher than the survival rate of storage on SSDs, ignoring what the disks were used for.
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Fri, 2022-11-11 at 17:05 +, Curt wrote: > On 2022-11-11, wrote: > > > > I just contested that their failure rate is higher than that of HDDs. > > This is something which was true in early days, but nowadays it seems > > to be just a prejudice. > > If he prefers extrapolating his anecdotal personal experience to a > general rule rather than applying a verifiable general rule to his > personal experience, then there's just nothing to be done for him! > There is no "verifyable general rule" here. You can argue all you want and claim that SSDs fail less than hard disks, and it won't change the fact that, as I've said, I've seen them failing about the same. It's like you argue that the sun didn't go up today. It won't change the fact that I've seen that the sun has gone up today.
Sorry for the misattribution [was: ZFS performance (was: Re: deduplicating file systems: VDO] withDebian?)
On Fri, Nov 11, 2022 at 07:22:19PM +0100, to...@tuxteam.de wrote: [...] > I think what hede was hinting at was that early SSDs had a (pretty) > limited number of write cycles [...] As was pointed out to me, the OP wasn't hede. It was hw. Sorry for the mis-attribution. Cheers -- t signature.asc Description: PGP signature
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
hw writes: On Thu, 2022-11-10 at 23:05 -0500, Michael Stone wrote: > On Thu, Nov 10, 2022 at 06:55:27PM +0100, hw wrote: > > On Thu, 2022-11-10 at 11:57 -0500, Michael Stone wrote: > > > On Thu, Nov 10, 2022 at 05:34:32PM +0100, hw wrote: > > > > And mind you, SSDs are *designed to fail* the sooner the more data you > > > > write > > > > to > > > > them. They have their uses, maybe even for storage if you're so > > > > desperate, > > > > but > > > > not for backup storage. [...] Why would anyone use SSDs for backups? They're way too expensive for that. I actually do this for offsite/portable backups because SSDs are shock resistant (dont lose data when being dropped etc.). The most critical thing to acknowledge about using SSDs for backups is that the data retention time of SSDs (when not powered) is decreasing with each generation. Write endurance has not become critical in any of my SSD uses so far. Increasing workloads have also resulted in me upgrading the SSDs. So far I always upgraded faster than running into the write endurance limits. I do not use the SSDs as caches but as full-blown file system drives, though. On the current system, the SSDs report having written about 14 TB and are specified by the manufacturer for an endurance of 6300 TBW (drive size is 4 TB). The small (drive size about 240GB) ones I use for backup are much less durable. For one of them, the manufacturer claims 306TBW, the other has 360 TBW specified. I do not currently know how much data I have written to them already. As you can see from the sizes, I backup only a tiny subset of the data to SSDs i.e. the parts of my data that I consider most critical (VM images not being among them...). [...] There was no misdiagnosis. Have you ever had a failed SSD? They usually just disappear. I've had one exception in which the SDD at first only sometimes disappeared and came back, until it disappeared and didn't come back. [...] Just for the record I recall having observed this once in a very similar fashion. It was back when a typical SSD size was 60 GiB. By now we should mostly be past this “SSD fails early with controller fault” issues. It can still happen and I still expect SSDs to fail with less notice compared to HDDs. When I had my first (and so far only) disk failure (on said 60G SSD) I decided to: * Retain important data on HDDs (spinning rust) for the time being * and also implement RAID1 for all important drives Although in theory running two disks instead of one should increase the overall chance of having one fail, no disks failed after this change so far. YMMV Linux-Fan öö pgp7fF7ksy0yP.pgp Description: PGP signature
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Fri, Nov 11, 2022 at 02:05:33PM -0500, Dan Ritter wrote: 300TB/year. That's a little bizarre: it's 9.51 MB/s. Modern high end spinners also claim 200MB/s or more when feeding them continuous writes. Apparently WD thinks that can't be sustained more than 5% of the time. Which makes sense for most workloads. Very rarely do people write continuously to disks *and never keep the data there to read it later*. There are exceptions (mostly of the transaction log type for otherwise memory-resident data), and you can get write optimized storage, but you'll pay more. For most people that's a bad deal, because it would mean paying for a level of write endurance that they'll never use.
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Fri, Nov 11, 2022 at 07:15:07AM +0100, hw wrote: There was no misdiagnosis. Have you ever had a failed SSD? They usually just disappear. Actually, they don't; that's a somewhat unusual failure mode. I have had a couple of ssd failures, out of hundreds. (And I think mostly from a specific known-bad SSD design; I haven't had any at all in the past few years.) I've had way more dead hard drives, which is typical. There was no "not normal" territory, either, unless maybe you consider ZFS cache as "not normal". In that case, I would argue that SSDs are well suited for such applications because they allow for lots of IOOPs and high data transfer rates, and a hard disk probably wouldn't have failed in place of the SSD because they don't wear out so quickly. Since SSDs are so well suited for such purposes, that can't be "not normal" territory for them. Perhaps they just need to be more resilient than they are. You probably bought the wrong SSD. SSDs write in erase-block units, which are on the order of 1-4MB. If you're writing many many small blocks (as you would with a ZFS ZIL cache) there's significant write amplification. For that application you really need a fairly expensive write-optimized SSD, not a commodity (read-optimized) SSD. (And in fact, SSD is *not* ideal for this because the data is written sequentially and basically never read so low seek times aren't much benefit; NVRAM is better suited.) If you were using it for L2ARC cache then mostly that makes no sense for a backup server. Without more details it's really hard to say any more. Honestly, even with the known issues of using commidity SSD for SLOG I find it really hard to believe that your backups were doing enough async transactions for that to matter--far more likely is still that you simply got a bad copy, just like you can get a bad hd. Sometimes you get a bad part, that's life. Certainly not something to base a religion on. Considering that, SSDs generally must be of really bad quality for that to happen, don't you think? No, I think you're making unsubstantiated statements, and I'm mostly trying to get better information on the record for others who might be reading.
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
to...@tuxteam.de wrote: > > I think what hede was hinting at was that early SSDs had a (pretty) > limited number of write cycles per "block" [1] before failure; they had > (and have) extra blocks to substitute broken ones and do a fair amount > of "wear leveling behind the scenes. So it made more sense to measure > failures along the "TB written" axis than along the time axis. They still do, and in fact each generation gets worse in terms of durability while getting better in price/capacity. Here's Western Digital's cheap line of NVMe SSDs: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-blue-nvme-ssd/product-brief-wd-blue-sn570-nvme-ssd.pdf MTBF is listed as 1.5 million hours... 160 years. Lifetime endurance is listed as 150TB for the 250GB version, and 300TB for the 500GB version. 600 full writes expected. Here's the more expensive Red line: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-red-ssd/product-brief-western-digital-wd-red-sn700-nvme-ssd.pdf MTTF: 1.7 million hours. Snicker. Endurance: 1000TB for the 500GB version, 2000TB for the 1TB version. A nice upgrade from 600 writes to 2000 writes. Unrelated, but cool: the 4TB version weighs 10 grams. -dsr-
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
Jeffrey Walton wrote: > On Fri, Nov 11, 2022 at 2:01 AM wrote: > > > > On Fri, Nov 11, 2022 at 07:15:07AM +0100, hw wrote: > > > On Thu, 2022-11-10 at 23:05 -0500, Michael Stone wrote: > >... Here's a report > > by folks who do lots of HDDs and SDDs: > > > > https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1-2021/ > > > > The gist, for disks playing similar roles (they don't use yet SSDs for bulk > > storage, because of the costs): 2/1518 failures for SSDs, 44/1669 for HDDs. > > Forgive my ignorance... Isn't Mean Time Before Failure (MTFB) the > interesting statistic? > > When selecting hardware, like HDD vs SSD, you don't know if and when a > failure is going to occur. You can only estimate failures using MTBF. > > After the installation and with failure data in hand, you can check if > the MTBF estimate is accurate. I expect most of the HDD and SSD > failures to fall within 1 standard deviation of the reported MTBF. And > you will have some data points that show failure before MTBF, and some > data points that show failure after MTBF. You'd like that to be true. I'd like that to be true. What do we actually see? Here's Western Digital's new 22TB 7200RPm NAS disk: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-red-pro-hdd/product-brief-western-digital-wd-red-pro-hdd.pdf Claimed MTBF: 1 million hours. Believe it or not, this is par for the course for high-end disks. 24 hours a day, 365 days a year: 8760 hours per year. 100/8760 = 114 years. So, no: MTBF numbers must be presumed to be malicious lies. More reasonable: this drive comes with a 5 year warranty. Treat that as the expected lifetime, and you'll be somewhat closer to the truth maybe. Here's a new number that used to be just for SSDs, but is now for spinners as well: expected workload (per year) or Total TB Written (lifetime). For the above disk family, all of them claim 300TB/year. That's a little bizarre: it's 9.51 MB/s. Modern high end spinners also claim 200MB/s or more when feeding them continuous writes. Apparently WD thinks that can't be sustained more than 5% of the time. -dsr-
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Fri, Nov 11, 2022 at 12:53:21PM -0500, Jeffrey Walton wrote: > On Fri, Nov 11, 2022 at 2:01 AM wrote: > > > > On Fri, Nov 11, 2022 at 07:15:07AM +0100, hw wrote: > > > On Thu, 2022-11-10 at 23:05 -0500, Michael Stone wrote: > >... Here's a report > > by folks who do lots of HDDs and SDDs: > > > > https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1-2021/ > > > > The gist, for disks playing similar roles (they don't use yet SSDs for bulk > > storage, because of the costs): 2/1518 failures for SSDs, 44/1669 for HDDs. > > Forgive my ignorance... Isn't Mean Time Before Failure (MTFB) the > interesting statistic? I think what hede was hinting at was that early SSDs had a (pretty) limited number of write cycles per "block" [1] before failure; they had (and have) extra blocks to substitute broken ones and do a fair amount of "wear leveling behind the scenes. So it made more sense to measure failures along the "TB written" axis than along the time axis. Cheers [1] In a very sloppy sense: those beasts have big write units, 256K and up. -- t signature.asc Description: PGP signature
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Fri, Nov 11, 2022 at 2:01 AM wrote: > > On Fri, Nov 11, 2022 at 07:15:07AM +0100, hw wrote: > > On Thu, 2022-11-10 at 23:05 -0500, Michael Stone wrote: >... Here's a report > by folks who do lots of HDDs and SDDs: > > https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1-2021/ > > The gist, for disks playing similar roles (they don't use yet SSDs for bulk > storage, because of the costs): 2/1518 failures for SSDs, 44/1669 for HDDs. Forgive my ignorance... Isn't Mean Time Before Failure (MTFB) the interesting statistic? When selecting hardware, like HDD vs SSD, you don't know if and when a failure is going to occur. You can only estimate failures using MTBF. After the installation and with failure data in hand, you can check if the MTBF estimate is accurate. I expect most of the HDD and SSD failures to fall within 1 standard deviation of the reported MTBF. And you will have some data points that show failure before MTBF, and some data points that show failure after MTBF. Jeff
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Fri, Nov 11, 2022 at 05:05:51PM -, Curt wrote: > On 2022-11-11, wrote: > > > > I just contested that their failure rate is higher than that of HDDs. [...] > If he prefers extrapolating his anecdotal personal experience to a > general rule rather than applying a verifiable general rule to his > personal experience, then there's just nothing to be done for him! > > > > I'm out of this thread. So you lured me in again, you baddy :) Everyone has right to their prejudices. I cherish mine too. My beef was rather with being misrepresented myself. Cheers -- t signature.asc Description: PGP signature
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On 2022-11-11, wrote: > > I just contested that their failure rate is higher than that of HDDs. > This is something which was true in early days, but nowadays it seems > to be just a prejudice. If he prefers extrapolating his anecdotal personal experience to a general rule rather than applying a verifiable general rule to his personal experience, then there's just nothing to be done for him! > I'm out of this thread. >
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Fri, Nov 11, 2022 at 09:12:36AM +0100, hw wrote: > Backblaze does all kinds of things. whatever. > > The gist, for disks playing similar roles (they don't use yet SSDs for bulk > > storage, because of the costs): 2/1518 failures for SSDs, 44/1669 for HDDs. > > > > I'll leave the maths as an exercise to the reader. > > Numbers never show you the real picture, especially not statistical ones. Your gut feeling might be more relevant for you. But it's not for me, so that's why I'll bow out of this thread :-) > You > even say it yourself that the different types of disks were used for different > purposes. That makes your numbers meaningless. Please, re-read what I wrote (you even quoted it above). It's nearly the opposite of what you say here. I said "similar roles". I won't read the blog entry for you aloud here. > Out of cruiosity, what do these numbers look like in something like survivours > per TB? Those numbers will probably show a very different picture, won't > they. DidI say similar roles? The devices compared are doing the same job, So TB per unit time are, for all we know, comparable. > And when even Backblaze doesn't use SSDs for backup storage because they're > expensive, then why would you suggest or assume that anyone do or does that? There again. Please read what others write before paraphrasing them wrongly. I never said you should use SSDs for bulk data storage. They are too expensive for that. That's something you, backblaze and me all agreed from the start. Why on earth do you bring that up here, then? I just contested that their failure rate is higher than that of HDDs. This is something which was true in early days, but nowadays it seems to be just a prejudice. I'm out of this thread. Cheers -- t signature.asc Description: PGP signature
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Fri, 2022-11-11 at 08:01 +0100, to...@tuxteam.de wrote: > On Fri, Nov 11, 2022 at 07:15:07AM +0100, hw wrote: > > On Thu, 2022-11-10 at 23:05 -0500, Michael Stone wrote: > > [...] > > > Why would anyone use SSDs for backups? They're way too expensive for that. > > Possibly. > > > So far, the failure rate with SSDs has been not any better than the failure > > rate > > of hard disks. Considering that SSDs are supposed to fail less, the > > experience > > with them is pretty bad. > > You keep pulling things out of whatever (thin air, it seems). That's more like what you're doing. In my case, it's my own experience. > Here's a report > by folks who do lots of HDDs and SDDs: > > https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1-2021/ Backblaze does all kinds of things. > The gist, for disks playing similar roles (they don't use yet SSDs for bulk > storage, because of the costs): 2/1518 failures for SSDs, 44/1669 for HDDs. > > I'll leave the maths as an exercise to the reader. Numbers never show you the real picture, especially not statistical ones. You even say it yourself that the different types of disks were used for different purposes. That makes your numbers meaningless. Out of cruiosity, what do these numbers look like in something like survivours per TB? Those numbers will probably show a very different picture, won't they. And when even Backblaze doesn't use SSDs for backup storage because they're expensive, then why would you suggest or assume that anyone do or does that?
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Fri, Nov 11, 2022 at 07:15:07AM +0100, hw wrote: > On Thu, 2022-11-10 at 23:05 -0500, Michael Stone wrote: [...] > Why would anyone use SSDs for backups? They're way too expensive for that. Possibly. > So far, the failure rate with SSDs has been not any better than the failure > rate > of hard disks. Considering that SSDs are supposed to fail less, the > experience > with them is pretty bad. You keep pulling things out of whatever (thin air, it seems). Here's a report by folks who do lots of HDDs and SDDs: https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1-2021/ The gist, for disks playing similar roles (they don't use yet SSDs for bulk storage, because of the costs): 2/1518 failures for SSDs, 44/1669 for HDDs. I'll leave the maths as an exercise to the reader. Cheers -- t signature.asc Description: PGP signature
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Thu, 2022-11-10 at 23:05 -0500, Michael Stone wrote: > On Thu, Nov 10, 2022 at 06:55:27PM +0100, hw wrote: > > On Thu, 2022-11-10 at 11:57 -0500, Michael Stone wrote: > > > On Thu, Nov 10, 2022 at 05:34:32PM +0100, hw wrote: > > > > And mind you, SSDs are *designed to fail* the sooner the more data you > > > > write > > > > to > > > > them. They have their uses, maybe even for storage if you're so > > > > desperate, > > > > but > > > > not for backup storage. > > > > > > It's unlikely you'll "wear out" your SSDs faster than you wear out your > > > HDs. > > > > > > > I have already done that. > > Then you're either well into "not normal" territory and need to buy an > SSD with better write longevity (which I seriously doubt for a backup > drive) or you just got unlucky and got a bad copy (happens with > anything) or you've misdiagnosed some other issue. > Why would anyone use SSDs for backups? They're way too expensive for that. So far, the failure rate with SSDs has been not any better than the failure rate of hard disks. Considering that SSDs are supposed to fail less, the experience with them is pretty bad. There was no misdiagnosis. Have you ever had a failed SSD? They usually just disappear. I've had one exception in which the SDD at first only sometimes disappeared and came back, until it disappeared and didn't come back. There was no "not normal" territory, either, unless maybe you consider ZFS cache as "not normal". In that case, I would argue that SSDs are well suited for such applications because they allow for lots of IOOPs and high data transfer rates, and a hard disk probably wouldn't have failed in place of the SSD because they don't wear out so quickly. Since SSDs are so well suited for such purposes, that can't be "not normal" territory for them. Perhaps they just need to be more resilient than they are. You could argue that the SSDs didn't fail because they were worn out but for other reasons. I'd answer that it's irrelevant for the user why exactly a disk failed, especially when it just disappears, and that hard disks don't fail because the storage media wears out like SSDs do but for other reasons. Perhaps you could buy an SSD that withstands being written to it better. The question then is if that's economical. It would also be based on the assumption that SSDs don't so much fail for other reasons than the storage media being worn out. Since all the failed SSDs have disappeared, I have to assume that they didn't fail because the storage media was worn out but for other reasons. Considering that, SSDs generally must be of really bad quality for that to happen, don't you think?
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Thu, Nov 10, 2022 at 06:55:27PM +0100, hw wrote: On Thu, 2022-11-10 at 11:57 -0500, Michael Stone wrote: On Thu, Nov 10, 2022 at 05:34:32PM +0100, hw wrote: > And mind you, SSDs are *designed to fail* the sooner the more data you write > to > them. They have their uses, maybe even for storage if you're so desperate, > but > not for backup storage. It's unlikely you'll "wear out" your SSDs faster than you wear out your HDs. I have already done that. Then you're either well into "not normal" territory and need to buy an SSD with better write longevity (which I seriously doubt for a backup drive) or you just got unlucky and got a bad copy (happens with anything) or you've misdiagnosed some other issue.
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Thu, 2022-11-10 at 11:57 -0500, Michael Stone wrote: > On Thu, Nov 10, 2022 at 05:34:32PM +0100, hw wrote: > > And mind you, SSDs are *designed to fail* the sooner the more data you write > > to > > them. They have their uses, maybe even for storage if you're so desperate, > > but > > not for backup storage. > > It's unlikely you'll "wear out" your SSDs faster than you wear out your > HDs. > I have already done that.
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Thu, Nov 10, 2022 at 05:34:32PM +0100, hw wrote: And mind you, SSDs are *designed to fail* the sooner the more data you write to them. They have their uses, maybe even for storage if you're so desperate, but not for backup storage. It's unlikely you'll "wear out" your SSDs faster than you wear out your HDs.
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On Thu, 2022-11-10 at 02:19 -0500, gene heskett wrote: > On 11/10/22 00:37, David Christensen wrote: > > On 11/9/22 00:24, hw wrote: > > > On Tue, 2022-11-08 at 17:30 -0800, David Christensen wrote: > > [...] > Which brings up another suggestion in two parts: > > 1: use amanda, with tar and compression to reduce the size of the > backups. And use a backup cycle of a week or 2 because amanda will if > advancing a level, only backup that which has been changed since the > last backup. On a quiet system, a level 3 backup for a 50gb network of > several machines can be under 100 megs. More on a busy system of course. > Amanda keeps track of all that automatically. Amanda is nice, yet quite unwieldy (try to get a file out of the backups ...). I used it long time ago (with tapes) and I'd have to remember or re-learn how to use amanda to back up particular directories and such ... I think I might be better off learning more about snapshots. > 2: As disks fail, replace them with SSD's which use much less power than > spinning rust. And they are typically 5x faster than commodity spinning > rust. Is this a joke? https://www.dell.com/en-us/shop/visiontek-16tb-class-qlc-7mm-25-ssd/apd/ab329068/storage-drives-media Cool, 30% discount on black friday saves you $2280 for every pair of disks, and it even starts right now. (Do they really mean that? What if I had a datacenter and ordered 512 or so of them? I'd save almost $1.2 million, what a great deal!) And mind you, SSDs are *designed to fail* the sooner the more data you write to them. They have their uses, maybe even for storage if you're so desperate, but not for backup storage. > Here, and historically with spinning rust, backing up 5 machines, at 3am > every morning is around 10gb total and under 45 minutes. This includes > the level 0's it does by self adjusting the schedule to spread the level > 0's, AKA the fulls, out over the backup cycle so the amount of storage > used for any one backup run is fairly consistent. That's almost half a month for 4TB. Why does it take so long?
Re: ZFS performance (was: Re: deduplicating file systems: VDO withDebian?)
On 11/10/22 00:37, David Christensen wrote: On 11/9/22 00:24, hw wrote: > On Tue, 2022-11-08 at 17:30 -0800, David Christensen wrote: > Hmm, when you can backup like 3.5TB with that, maybe I should put FreeBSD on my > server and give ZFS a try. Worst thing that can happen is that it crashes and > I'd have made an experiment that wasn't successful. Best thing, I guess, could > be that it works and backups are way faster because the server doesn't have to > actually write so much data because it gets deduplicated and reading from the > clients is faster than writing to the server. Be careful that you do not confuse a ~33 GiB full backup set, and 78 snapshots over six months of that same full backup set, with a full backup of 3.5 TiB of data. I would suggest a 10 GiB pool to backup the latter. Writing to a ZFS filesystem with deduplication is much slower than simply writing to, say, an ext4 filesystem -- because ZFS has to hash every incoming block and see if it matches the hash of any existing block in the destination pool. Storing the existing block hashes in a dedicated dedup virtual device will expedite this process. >> I run my backup script each night. It uses rsync to copy files and > > Aww, I can't really do that because my servers eats like 200-300W because it has > so many disks in it. Electricity is outrageously expensive here. Which brings up another suggestion in two parts: 1: use amanda, with tar and compression to reduce the size of the backups. And use a backup cycle of a week or 2 because amanda will if advancing a level, only backup that which has been changed since the last backup. On a quiet system, a level 3 backup for a 50gb network of several machines can be under 100 megs. More on a busy system of course. Amanda keeps track of all that automatically. 2: As disks fail, replace them with SSD's which use much less power than spinning rust. And they are typically 5x faster than commodity spinning rust. Here, and historically with spinning rust, backing up 5 machines, at 3am every morning is around 10gb total and under 45 minutes. This includes the level 0's it does by self adjusting the schedule to spread the level 0's, AKA the fulls, out over the backup cycle so the amount of storage used for any one backup run is fairly consistent. Perhaps platinum rated power supplies? Energy efficient HDD's/ SSD's? >> directories from various LAN machines into ZFS filesystems named after >> each host -- e.g. pool/backup/hostname (ZFS namespace) and >> /var/local/backup/hostname (Unix filesystem namespace). I have a >> cron(8) that runs zfs-auto-snapshot once each day and once each month >> that takes a recursive snapshot of the pool/backup filesystems. Their >> contents are then available via Unix namespace at >> /var/local/backup/hostname/.zfs/snapshot/snapshotname. If I want to >> restore a file from, say, two months ago, I use Unix filesystem tools to >> get it. > > Sounds like a nice setup. Does that mean you use snapshots to keep multiple > generations of backups and make backups by overwriting everything after you made > a snapshot? Yes. > In that case, is deduplication that important/worthwhile? You're not > duplicating it all by writing another generation of the backup but store only > what's different through making use of the snapshots. Without deduplication or compression, my backup set and 78 snapshots would require 3.5 TiB of storage. With deduplication and compression, they require 86 GiB of storage. > ... I only never got around to figure [ZFS snapshots] out because I didn't have the need. I accidentally trash files on occasion. Being able to restore them quickly and easily with a cp(1), scp(1), etc., is a killer feature. Users can recover their own files without needing help from a system administrator. > But it could also be useful for "little" things like taking a snapshot of the > root volume before updating or changing some configuration and being able to > easily to undo that. FreeBSD with ZFS-on-root has a killer feature called "Boot Environments" that has taken that idea to the next level: https://klarasystems.com/articles/managing-boot-environments/ >> I have 3.5 TiB of backups. It is useful to group files with similar characteristics (size, workload, compressibility, duplicates, backup strategy, etc.) into specific ZFS filesystems (or filesystem trees). You can then adjust ZFS properties and backup strategies to match. For compressed and/or encrypted archives, image, etc., I do not use compression or de-duplication >>> >>> Yeah, they wouldn't compress. Why no deduplication? >> >> >> Because I very much doubt that there will be duplicate blocks in such files. > > Hm, would it hurt? Yes. ZFS deduplication is resource intensive. > Oh it's not about performance when degraded, but about performance. IIRC when >
