Re: [gentoo-user] Hard drive pricing and the near future
On Wednesday, 6 October 2021 21:24:07 BST Frank Steinmetzger wrote: > Well I’m on ye Olde Continent and here I use a price search engine called > geizhals (German for scrooge). But they have an English equivalent ;-) and > of course use European pricing schemes (meaning including taxes, which you > do different). > https://skinflint.co.uk/?m=1 What a useful site! Thank you Frank. -- Regards, Peter.
Re: [gentoo-user] Hard drive pricing and the near future
Frank Steinmetzger wrote: > Am Wed, Oct 06, 2021 at 10:24:07PM +0200 schrieb Frank Steinmetzger: >> Am Tue, Oct 05, 2021 at 06:32:24PM -0500 schrieb Dale: >>> Howdy all, >>> […] >>> If anyone reading this does track the pricing of drives, are they on the >>> rise, stable, dropping or what? Is this a good time to expand while it >>> is more cost effective? >> Well I’m on ye Olde Continent and here I use a price search engine called >> geizhals (German for scrooge). But they have an English equivalent ;-) and >> of course use European pricing schemes (meaning including taxes, which you >> do different). >> https://skinflint.co.uk/?m=1 >> >> The big difference here: it allows you to filter and sort by all sorts of >> criteria, such as being (non-)SMR or price/TB. And it shows you the prices >> of lots of different shops, including Amazon. So while it may not give you >> an idea of the US market, it gives you one of available drive models. > Ah, and another reason why I wanted to suggest this site: it also remembers > the price development of an item. If you view an item’s details, you have > its price history on the top right of the screen. > I used to use pricewatch.com but it seems to be gone now. I also ran up on a site once that shows a graph of prices going back to when a product was first introduced. Since my brain has no elephant genes, I have no idea what the name of the site was nor can I find it now either. :-/ Neat sites tho. Dale :-) :-)
Re: [gentoo-user] Hard drive pricing and the near future
On 06/10/2021 21:20, Laurence Perkins wrote: But currently the WD Purples and the Seagate Skyhawks should all be CMR. Seagate have said the Iromwolf range will remain CMR. BarraCuda is now all SMR (renamed from Barracuda, presumably to say it's still the budget range, but the old drives are pre-SMR). I get the impression that FireCuda are the "budget CMR" range, but don't quote me on that. Obviously I have no power to hold them to that, so do not take this as any kind of guarantee. But if they change it without warning they're likely to have a lot of unhappy customers. AS WD found out when loads of "marketed for NAS" WD Reds got returned "not fit for purpose" when people put them in NASs. Cheers, Wol
Re: [gentoo-user] Hard drive pricing and the near future
Am Wed, Oct 06, 2021 at 10:24:07PM +0200 schrieb Frank Steinmetzger: > Am Tue, Oct 05, 2021 at 06:32:24PM -0500 schrieb Dale: > > Howdy all, > > […] > > If anyone reading this does track the pricing of drives, are they on the > > rise, stable, dropping or what? Is this a good time to expand while it > > is more cost effective? > > Well I’m on ye Olde Continent and here I use a price search engine called > geizhals (German for scrooge). But they have an English equivalent ;-) and > of course use European pricing schemes (meaning including taxes, which you > do different). > https://skinflint.co.uk/?m=1 > > The big difference here: it allows you to filter and sort by all sorts of > criteria, such as being (non-)SMR or price/TB. And it shows you the prices > of lots of different shops, including Amazon. So while it may not give you > an idea of the US market, it gives you one of available drive models. Ah, and another reason why I wanted to suggest this site: it also remembers the price development of an item. If you view an item’s details, you have its price history on the top right of the screen. -- Grüße | Greetings | Qapla’ Please do not share anything from, with or about me on any social network. The only thing that makes some people bearable is their absence. signature.asc Description: PGP signature
Re: [gentoo-user] Hard drive pricing and the near future
Am Tue, Oct 05, 2021 at 06:32:24PM -0500 schrieb Dale: > Howdy all, > […] > If anyone reading this does track the pricing of drives, are they on the > rise, stable, dropping or what? Is this a good time to expand while it > is more cost effective? Well I’m on ye Olde Continent and here I use a price search engine called geizhals (German for scrooge). But they have an English equivalent ;-) and of course use European pricing schemes (meaning including taxes, which you do different). https://skinflint.co.uk/?m=1 The big difference here: it allows you to filter and sort by all sorts of criteria, such as being (non-)SMR or price/TB. And it shows you the prices of lots of different shops, including Amazon. So while it may not give you an idea of the US market, it gives you one of available drive models. For instance, here are all internal non-SMR 3.5″ drives of 6 TB and up: https://skinflint.co.uk/?cat=hde7s=1080_SATA+1.5Gb%2Fs%7E1080_SATA+3Gb%2Fs%7E1080_SATA+6Gb%2Fs%7E13810_6000%7E3772_3.5%7E8457_non-SMR I like to buy naked drives, because I don’t like “cable salad”, i.e. having multiple external cases, each needing its own special power supply. Instead, I’d opt for a hard drive dock which lets you insert a hard disk as if it were an SD card. Actually, I use this internal bay for my PC case: https://skinflint.co.uk/sharkoon-sata-quickport-internal-multi-a862667.html It eats standard SATA drives and is connected via standard connectors to my PSU and mobo. > However, in a year or so, I'm getting fiber internet, dang fast at that. > It's starts at 200Mb but still over a 100 times faster than current > connection. It goes all the way up to 1Gb. God help us all. ROFL But when are you ever going to watch it all? -- Grüße | Greetings | Qapla’ Please do not share anything from, with or about me on any social network. Filling your car becomes much cheaper if you install a smaller tank. signature.asc Description: PGP signature
RE: [gentoo-user] Hard drive pricing and the near future
> On Wed, Oct 6, 2021 at 3:37 PM Laurence Perkins wrote: > > I think what to look for there would be if there's a way to align the BTRFS > > chunks to the SMR blocks. > > There are definitely ways to implement filesystems that are more compatible > with SMR. You basically want something like a log-based filesystem. A COW > filesystem is actually a really good candidate as they don't do in-place > writes ever, and all you need to do is defer block frees and garbage collect > and so on to make it more log-based. COW is a good compromise. Log-based filesystems like NILFS2 generally have terrible performance. Their only advantage in this situation is that they don't suffer any detectable performance loss from SMR. EXT4 has new formatting options that tell it to sequentialize the metadata writes as much as possible. Supposedly it helps quite a bit, but I haven't tested it myself yet. > > One of the issues though is that these drives obfuscate how they work. > The filesystem has no way to intentionally write to the CMR vs SMR regions. It depends on the drive. There are ATA commands you can use and at least some drives will tell you what their zones are and can be told to give the host more control over what goes where. https://zonedstorage.io/getting-started/smr-disk/ has a good description. Definitely avoid the Drive Managed Interface ones except maybe for cheap, bulk storage. But the Host Aware and Host Managed ones can perform pretty well on some IO loads with proper FS tuning. > > This is why drive-managed SMR really shouldn't be a thing. > Host-managed SMR makes a LOT more sense, because then the filesystem can > mitigate most of the issues and not end up fighting the drive firmware, whose > behavior isn't even standardized. Yup. Only reason for drive-managed is for drop-in replacement of low-throughput systems that couldn't otherwise handle it. > > > But the manufacturers decided to continue manufacturing CMR disks for the > > surveillance industry, so I haven't had to worry about it just yet. > > There are lots of CMR drives out there. It is just that you have to be > careful as nothing is well-documented and it is all subject to change. It is > like trying to figure out how many channels a DIMM has or what its timing > capabilities are. > > I saw a good price on an Exos drive and I believe those are all CMR. > This is why I use slickdeals - you can set up searches and get alerts when a > price drops. It also picks up stuff like Best Buy who often has some of the > best prices on USB enclosures for whatever reason when they go on sale. Almost everything marketed as being for the surveillance industry is going to be CMR. We often run the drives at 80% or better throughput and a lot of it is non-sequential. SMRs just can't keep up. So there was a lot of pushback from the surveillance industry about not changing the underlying storage technology without clearly labeling it as such. Prior to that the announcement from Seagate and I think WD was that everything larger than 12TB was going to be SMR. (I work on the software side of things, I get hardware stuff second hand, so my understanding may be fuzzy.) But currently the WD Purples and the Seagate Skyhawks should all be CMR. Obviously I have no power to hold them to that, so do not take this as any kind of guarantee. But if they change it without warning they're likely to have a lot of unhappy customers. > > -- > Rich > > LMP
Re: [gentoo-user] Hard drive pricing and the near future
On Wed, Oct 6, 2021 at 3:37 PM Laurence Perkins wrote: > I think what to look for there would be if there's a way to align the BTRFS > chunks to the SMR blocks. There are definitely ways to implement filesystems that are more compatible with SMR. You basically want something like a log-based filesystem. A COW filesystem is actually a really good candidate as they don't do in-place writes ever, and all you need to do is defer block frees and garbage collect and so on to make it more log-based. One of the issues though is that these drives obfuscate how they work. The filesystem has no way to intentionally write to the CMR vs SMR regions. This is why drive-managed SMR really shouldn't be a thing. Host-managed SMR makes a LOT more sense, because then the filesystem can mitigate most of the issues and not end up fighting the drive firmware, whose behavior isn't even standardized. > But the manufacturers decided to continue manufacturing CBR disks for the > surveillance industry, so I haven't had to worry about it just yet. There are lots of CMR drives out there. It is just that you have to be careful as nothing is well-documented and it is all subject to change. It is like trying to figure out how many channels a DIMM has or what its timing capabilities are. I saw a good price on an Exos drive and I believe those are all CMR. This is why I use slickdeals - you can set up searches and get alerts when a price drops. It also picks up stuff like Best Buy who often has some of the best prices on USB enclosures for whatever reason when they go on sale. -- Rich
RE: [gentoo-user] Hard drive pricing and the near future
> On Wed, Oct 6, 2021 at 12:16 PM Laurence Perkins wrote: > > > > Other option, depending on exactly what your use case is would be to look > > into your choice of filesystem. SMR doesn't like random writes into one of > > its chunks unless it has enough idle time to go back and straighten it out > > later is all. There are now format options for ext4 to align its metadata > > to the SMR sections and to make it avoid random writes as much as it can. > > Additionally BTRFS, ZFS, and NILFS2 are all structured such that they tend > > to write from one end of the disk to the other and then jump back to the > > beginning, so they see little if any degradation from SMR. > > Unless something has changed, it was ZFS rebuilds that caused a lot of the > initial fuss on Linux. Drives were getting dropped from pools due to > timeouts/etc during rebuilds. I'm not sure how sequential the IO is for ZFS > rebuilds. I think btrfs seems a bit smarter about scrubs in general. > > -- > Rich > Good to know. I don't use ZFS myself, and I suspect the benchmarks I looked at when I was checking on how best to deal with SMR drives didn't try a pool rebuild for ZFS. Scrub on BTRFS is read-only unless there are errors that need correcting, which should be rare. BTRFS balance operations might be affected, but those aren't needed as often as they used to be and should still operate linearly on whole chunks. I think what to look for there would be if there's a way to align the BTRFS chunks to the SMR blocks. But the manufacturers decided to continue manufacturing CBR disks for the surveillance industry, so I haven't had to worry about it just yet. LMP
Re: [gentoo-user] Hard drive pricing and the near future
On Wed, Oct 6, 2021 at 12:16 PM Laurence Perkins wrote: > > Other option, depending on exactly what your use case is would be to look > into your choice of filesystem. SMR doesn't like random writes into one of > its chunks unless it has enough idle time to go back and straighten it out > later is all. There are now format options for ext4 to align its metadata to > the SMR sections and to make it avoid random writes as much as it can. > Additionally BTRFS, ZFS, and NILFS2 are all structured such that they tend to > write from one end of the disk to the other and then jump back to the > beginning, so they see little if any degradation from SMR. Unless something has changed, it was ZFS rebuilds that caused a lot of the initial fuss on Linux. Drives were getting dropped from pools due to timeouts/etc during rebuilds. I'm not sure how sequential the IO is for ZFS rebuilds. I think btrfs seems a bit smarter about scrubs in general. -- Rich
RE: [gentoo-user] Hard drive pricing and the near future
> -Original Message- > From: Dale > Sent: Tuesday, October 5, 2021 4:32 PM > To: gentoo-user@lists.gentoo.org > Subject: [gentoo-user] Hard drive pricing and the near future > > > Howdy all, > > I still have quite a bit of drive storage but I've read that prices on drives > are on the rise. Thing is, I don't track them much. I'm looking at buying a > 8TB drive and I've researched to make sure I'm getting a PMR/CMR drive. I'm > avoiding a SMR since it doesn't perform as well in my use case. I tend to > stick with Seagate, WD and other major makers. > > If anyone reading this does track the pricing of drives, are they on the > rise, stable, dropping or what? Is this a good time to expand while it is > more cost effective? I shop around on ebay, Amazon and others before buying. > I'm not opposed to buying used since I can sometimes find one that was > pulled and sometimes has only a few hours of use. I found one once that only > had like 10 hours on it. Still got it too. > > One reason I'm wanting to do this now is price. However, in a year or so, > I'm getting fiber internet, dang fast at that. It's starts at 200Mb but > still over a 100 times faster than current connection. It goes all the way > up to 1Gb. God help us all. ROFL > > Thoughts?? > > Dale > > :-) :-) > If you want to avoid SMR, look for the drives built for video surveillance systems. Most manufacturers have a line for that purpose. For WD it's the "purple" ones. Other option, depending on exactly what your use case is would be to look into your choice of filesystem. SMR doesn't like random writes into one of its chunks unless it has enough idle time to go back and straighten it out later is all. There are now format options for ext4 to align its metadata to the SMR sections and to make it avoid random writes as much as it can. Additionally BTRFS, ZFS, and NILFS2 are all structured such that they tend to write from one end of the disk to the other and then jump back to the beginning, so they see little if any degradation from SMR. LMP
Re: [gentoo-user] Hard drive pricing and the near future
SMR drive prices are hopefully dropping. A couple of months ago I bought a couple of used Hitachi 4 GB drives, for my raid setup. Being raid I got recertified drives on ebay. They were about $65, up from about $50 for the same drives a couple of years ago. Larger new drives are hopefully coming down (I'd still like to have proper backup when I can afford it) RE: the SATA power connection change, it's stupid but easy to fix. Just cut the orange wire going to the drive power connector. I don't think anyone uses 3.3V on hard drives (the orange wire), and letting it float works with the rather damaged change to the spec. Of course, only do this on the drive connectors. Doesn't cause a problem on older drives. Alternately, you can use a "Molex" to sata power connector, since the 4 pin Molex connectors don't have a 3.3V line the adapters always leave it unconnected. Might be preferable if your' system has some of the older connectors and you are nervous about cutting wires though in the case of the drives it's absolutely OK. As these drive power connector are usually daisy chained it's best to cut the orange wire on all the SATA power connectors. --"Fascism begins the moment a ruling class, fearing the people may use their political democracy to gain economic democracy, begins to destroy political democracy in order to retain its power of exploitation and special privilege." Tommy Douglas Oct 5, 2021, 17:32 by rdalek1...@gmail.com: > Howdy all, > > I still have quite a bit of drive storage but I've read that prices on > drives are on the rise. Thing is, I don't track them much. I'm looking > at buying a 8TB drive and I've researched to make sure I'm getting a > PMR/CMR drive. I'm avoiding a SMR since it doesn't perform as well in > my use case. I tend to stick with Seagate, WD and other major makers. > > If anyone reading this does track the pricing of drives, are they on the > rise, stable, dropping or what? Is this a good time to expand while it > is more cost effective? I shop around on ebay, Amazon and others before > buying. I'm not opposed to buying used since I can sometimes find one > that was pulled and sometimes has only a few hours of use. I found one > once that only had like 10 hours on it. Still got it too. > > One reason I'm wanting to do this now is price. However, in a year or > so, I'm getting fiber internet, dang fast at that. It's starts at 200Mb > but still over a 100 times faster than current connection. It goes all > the way up to 1Gb. God help us all. ROFL > > Thoughts?? > > Dale > > :-) :-) >
Re: [gentoo-user] Hard drive pricing and the near future
Matt Connell wrote: > On Tue, 2021-10-05 at 18:32 -0500, Dale wrote: >> If anyone reading this does track the pricing of drives, are they on >> the rise, stable, dropping or what? > I can't speak to trends, but I've used this site in the past to keep an > eye out for a deal when it comes up. It only indexes Amazon prices, > but its usually a good bellweather for how things look in general. > > https://diskprices.com/ > > > > That's a nifty site. Wish it would allow me to select brands but this is a really good starting point. Thanks much. Dale :-) :-)
Re: [gentoo-user] Hard drive pricing and the near future
On Tue, 2021-10-05 at 18:32 -0500, Dale wrote: > If anyone reading this does track the pricing of drives, are they on > the rise, stable, dropping or what? I can't speak to trends, but I've used this site in the past to keep an eye out for a deal when it comes up. It only indexes Amazon prices, but its usually a good bellweather for how things look in general. https://diskprices.com/
Re: [gentoo-user] Hard drive pricing and the near future
Rich Freeman wrote: > On Tue, Oct 5, 2021 at 7:32 PM Dale wrote: >> If anyone reading this does track the pricing of drives, are they on the >> rise, stable, dropping or what? Is this a good time to expand while it >> is more cost effective? I shop around on ebay, Amazon and others before >> buying. I'm not opposed to buying used since I can sometimes find one >> that was pulled and sometimes has only a few hours of use. I found one >> once that only had like 10 hours on it. Still got it too. > Dropping I would say. For a while the supply was interrupted, most > likely due to Chia. Fortunately the price of Chia dropped and it > became the network had gotten so large that payback was going to be > very slow except for a few weeks in the beginning. I suspect that > people with a lot of storage might be farming Chia with their spare > storage, but I doubt anybody is buying pallets of hard drives just to > farm it. > > If you aren't in a hurry or picky about the model I suggest setting up > searches on slickdeals. Then be sure to check online to see if the > drive is known to be SMR. When I buy a drive I do a bit of > benchmarking just to make sure - I think just running more than one > pass on badblocks would probably catch it (granted the access is all > sequential, but the drive has no way of knowing that and so on each > pass it would have to do two passes to consolidate writes). > > Usually the best prices are on USB3 10+TB hard drives. The good 3.5" > drives tend to be more expensive since they're targeted at commercial > use. You can generally shuck the drive out of a USB3 enclosure if you > want to, but if your PSU isn't compatible you have to do a bit of > workarounds because they use the latest SATA power standard and some > genius decided not to make that backwards-compatible with the SATA > power found all over the place. Usually that is only used in > enterprise drives and the USB3 enclosures often use surplus enterprise > disks (so you're getting a really good value with them). If you keep > it in the enclosure you don't have to worry about it. I've found > about half my PSUs work fine them, and half require polyamide tape > games to work. > I may give it a bit and see what they do then. My /home is at 65% so I got time, especially while on this whimpy DSL. I recently discovered torrents and its advantages and now my DSL stays busy. I only pause it when I need the internet for something else. When fiber gets here, oh dear. You the one who introduced me to SMR. I bought one and started a thread about why my external drive had this bumpy feel long after my backups were done. You posted about SMR and how they work. For the backup drive, I don't mind much but if I had known before I bought it, I would have avoided it. I let the drive sit until the bumpy feel goes away after I do my backups, which at times takes a while. Now I try to avoid them and research before hitting the buy button. I've looked into buying external drives and removing them for internal use. It seems to be a little risky given the power problem. At one point I thought I found a adapter, maybe a China made thing, that plugs into the drive and then regular power supply cables plug into the adapter. I never bought one since I think it may be best to just buy drives made for going in my puter case and hooking directly to my cables. I've read where you can save quite a bit of money doing that tho. May give this a bit more time. See what the prices do. Thanks for the info. Dale :-) :-)
Re: [gentoo-user] Hard drive pricing and the near future
On Tue, Oct 5, 2021 at 7:32 PM Dale wrote: > > If anyone reading this does track the pricing of drives, are they on the > rise, stable, dropping or what? Is this a good time to expand while it > is more cost effective? I shop around on ebay, Amazon and others before > buying. I'm not opposed to buying used since I can sometimes find one > that was pulled and sometimes has only a few hours of use. I found one > once that only had like 10 hours on it. Still got it too. Dropping I would say. For a while the supply was interrupted, most likely due to Chia. Fortunately the price of Chia dropped and it became the network had gotten so large that payback was going to be very slow except for a few weeks in the beginning. I suspect that people with a lot of storage might be farming Chia with their spare storage, but I doubt anybody is buying pallets of hard drives just to farm it. If you aren't in a hurry or picky about the model I suggest setting up searches on slickdeals. Then be sure to check online to see if the drive is known to be SMR. When I buy a drive I do a bit of benchmarking just to make sure - I think just running more than one pass on badblocks would probably catch it (granted the access is all sequential, but the drive has no way of knowing that and so on each pass it would have to do two passes to consolidate writes). Usually the best prices are on USB3 10+TB hard drives. The good 3.5" drives tend to be more expensive since they're targeted at commercial use. You can generally shuck the drive out of a USB3 enclosure if you want to, but if your PSU isn't compatible you have to do a bit of workarounds because they use the latest SATA power standard and some genius decided not to make that backwards-compatible with the SATA power found all over the place. Usually that is only used in enterprise drives and the USB3 enclosures often use surplus enterprise disks (so you're getting a really good value with them). If you keep it in the enclosure you don't have to worry about it. I've found about half my PSUs work fine them, and half require polyamide tape games to work. -- Rich
