Re: [ceph-users] Suggestions/experiences with mixed disk sizes and models from 4TB - 14TB
On 16/01/2019 18:33, Götz Reinicke wrote: My question is: How are your experiences with the current >=8TB SATA disks are some very bad models out there which I should avoid? Be careful with Seagate consumer SATA drives. They are now shipping SMR drives without mentioning that fact anywhere in the documentation. One example of such a model is the 4TB ST4000DM004 (previous models like the ST4000DM000 were not SMR). I expect this to cause catastrophically slow performance under heavy write volumes, e.g. when rebuilding or rebalancing PGs. I assume enterprise models are fine (if you read the fine print), but I would avoid any current generation Seagate consumer models unless you're happy buying a sample first and benchmarking it to confirm what kind of drive it is, or you can find someone who has done so. SMR drives have a telltale sign of unreasonably fast random write performance for a brief time (well beyond practical IOPS for any normal HDD), which then craters to nearly zero once the internal journal fills up. Personally I'm using MD05ACA800 (8TB toshiba, spec unknown, seems to be a B2B model but they're available for cheap) and they seem to work well so far in my home cluster, but I haven't finished setting things up yet. Those are definitely not SMR. -- Hector Martin (hec...@marcansoft.com) Public Key: https://marcan.st/marcan.asc ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Suggestions/experiences with mixed disk sizes and models from 4TB - 14TB
I've run my home cluster with drives ranging in size from 500GB to 8TB before and the biggest issue you run into is that the bigger drives will get a proportional more number of PGs which will increase the memory requirements on them. Typically you want around 100 PGs/OSD, but if you mix 4TB and 14TB drives in a cluster the 14TB drives will have 3.5 times the number of PGs. So if the 4TB drives have 100 PGs, the 14TB drives will have 350. Or if the 14TB drives have 100 PGs, the 4TB drives will only have just 28 PGs on them. Using the balancer plugin in the mgr will pretty much be required. Also since you're using EC you'll need to make sure the math works with these nodes receiving 2-3.5 times the data. Bryan From: ceph-users on behalf of Götz Reinicke Date: Wednesday, January 16, 2019 at 2:33 AM To: ceph-users Subject: [ceph-users] Suggestions/experiences with mixed disk sizes and models from 4TB - 14TB Dear Ceph users, I’d like to get some feedback for the following thought: Currently I run some 24*4TB bluestore OSD nodes. The main focus is on storage space over IOPS. We use erasure code and cephfs, and things look good right now. The „but“ is, I do need more disk space and don’t have so much more rack space available, so I was thinking of adding some 8TB or even 12TB OSDs and/or exchange over time 4TB OSDs with bigger disks. My question is: How are your experiences with the current >=8TB SATA disks are some very bad models out there which I should avoid? The current OSD nodes are connected by 4*10Gb bonds, so for replication/recovery speed is a 24 Chassis with bigger disks useful, or should I go with smaller chassis? Or dose the chassi sice does not matter at all that much in my setup. I know, EC is quit computing intense, so may be bigger disks hav also there an impact? Lot’s of questions, may be you can help answering some. Best regards and Thanks a lot for feedback . Götz ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Suggestions/experiences with mixed disk sizes and models from 4TB - 14TB
Have had some good experiences with ST1NM0156-2AA111 Also running with EC but using RBD for slow storage for VM's, have had good performance for what id expect for 10TB drive's and EC. I would definitely say getting helium drives helps VS standard air filled when you get to the 8TB+ drives. On Wed, Jan 16, 2019 at 5:34 PM Götz Reinicke < goetz.reini...@filmakademie.de> wrote: > Dear Ceph users, > > I’d like to get some feedback for the following thought: > > Currently I run some 24*4TB bluestore OSD nodes. The main focus is on > storage space over IOPS. > > We use erasure code and cephfs, and things look good right now. > > The „but“ is, I do need more disk space and don’t have so much more rack > space available, so I was thinking of adding some 8TB or even 12TB OSDs > and/or exchange over time 4TB OSDs with bigger disks. > > My question is: How are your experiences with the current >=8TB SATA disks > are some very bad models out there which I should avoid? > > The current OSD nodes are connected by 4*10Gb bonds, so for > replication/recovery speed is a 24 Chassis with bigger disks useful, or > should I go with smaller chassis? Or dose the chassi sice does not matter > at all that much in my setup. > > I know, EC is quit computing intense, so may be bigger disks hav also > there an impact? > > Lot’s of questions, may be you can help answering some. > > Best regards and Thanks a lot for feedback . Götz > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Suggestions/experiences with mixed disk sizes and models from 4TB - 14TB
Dear Ceph users, I’d like to get some feedback for the following thought: Currently I run some 24*4TB bluestore OSD nodes. The main focus is on storage space over IOPS. We use erasure code and cephfs, and things look good right now. The „but“ is, I do need more disk space and don’t have so much more rack space available, so I was thinking of adding some 8TB or even 12TB OSDs and/or exchange over time 4TB OSDs with bigger disks. My question is: How are your experiences with the current >=8TB SATA disks are some very bad models out there which I should avoid? The current OSD nodes are connected by 4*10Gb bonds, so for replication/recovery speed is a 24 Chassis with bigger disks useful, or should I go with smaller chassis? Or dose the chassi sice does not matter at all that much in my setup. I know, EC is quit computing intense, so may be bigger disks hav also there an impact? Lot’s of questions, may be you can help answering some. Best regards and Thanks a lot for feedback . Götz smime.p7s Description: S/MIME cryptographic signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
