A classic raid5 system takes a looong time to rebullid the raid, so i would say NO, but how long does it take for ceph to rebullid the placement group?
J > El 2 oct 2015, a las 12:01, Simon Hallam <[email protected]> escribió: > > The way I look at it is: > > Would you normally put 18*2TB disks in a single RAID5 volume? If the answer > is no, then a replication factor of 2 is not enough. > > Cheers, > > Simon > > From: ceph-users [mailto:[email protected]] On Behalf Of > Javier C.A. > Sent: 02 October 2015 09:58 > To: [email protected] > Subject: Re: [ceph-users] Predict performance > > Christian > > thank you so much for your answer. > > You're right, when I say Performance, I actually mean the "classic FIO > test"..... > > Regarding the CPU, you meant 2Ghz per OSD and per CPU CORE, isn't? > > One last question, with a total number of 18xOSD (2TB/OSD), and a replica > factor of 2, is it really risky? This won't be a critical cluster, but > neither is a lab/test cluster, you know.... > > Thanks again. > J > > > Date: Fri, 2 Oct 2015 17:16:21 +0900 > > From: [email protected] > > To: [email protected] > > CC: [email protected] > > Subject: Re: [ceph-users] Predict performance > > > > > > Hello, > > > > More line breaks, formatting. > > A wall of text makes people less likely to read things. > > > > On Fri, 2 Oct 2015 07:08:29 +0000 Javier C.A. wrote: > > > > > Hello > > > Before posting this message, I've been reading older posts in the > > > mailing list, but I didn't get any clear answer..... > > > > Define performance. > > Many people seem to be fascinated by the speed of sequential (more or less) > > writes and reads, while their use case would actually be better served by > > an increased small IOPS performance. > > > > >I happen to have > > > three servers available to test Ceph, and I would like to know if there > > > is any kind of "performance prediction formula". > > > > If there is such a thing (that actually works with less than a 10% error > > margin), I'm sure RedHat would like to charge you for it. ^_- > > > > >-My OSD servers are: > > > - 1 x Intel E5-2603v3 1.6Ghz (6 cores) > > Slightly underpowered, especially when it comes to small write IOPS. > > My personal formula is at least 2GHz per OSD with SSD journal. > > > > >- 32G RAM D4 > > OK, more is better (for read performance, see below). > > > > >- 10Gb ethernet network, jumbo frames enabled - > > > > Slight overkill given the rest of your setup, I guess you saw all the fun > > people keep having with jumbo frames in the ML archives. > > > > >SSOO: 2 x 500GB RAID 1 > > >- OSD (6 OSD): - 2TB 7200 SATA4 6Gbps > > >- 1 x SSD Intel SC3700 200GB for > > > journaling of all 6 OSDs. - > > This means that the most throughput you'll ever be able to write to those > > nodes is the speed of that SSD, 365MB/s, lets make that 350MB/s. > > Thus the slight overkill comment earlier. > > OTOH the HDDs get to use most of the IOPS (after discounting FS journals, > > overhead, the OSD leveldb, etc). > > So lets say slightly less than 100 IOPS per OSD. > > > > >Replication factor = 2. > > see below. > > > > >- XFS > > I find Ext4 faster, but that's me. > > > > >-MON nodes > > > will be running in other servers. With this OSD setup, how could I > > > predict the cpeh cluster performace (IOPS, R/W BW, latency...)? > > > > Of these, latency is the trickiest one, as so many things factor into it > > aside from the network. > > A test case where you're hitting basically just one OSD will look a lot > > worse than what an evenly spread out (more threads over a sufficiently > > large data set) test would. > > > > Userspace (librbd) results can/will vastly differ from kernel RBD clients. > > > > IOPS is a totally worthless data point w/o clearly defining what you're > > measuring how. > > Lets assume the "standard" of 4KB blocks and 32threads, random writes. > > Also lets assume a replication factor of 3, see below. > > > > Sustained sync'ed (direct=1 option in fio) IOPS with your setup will be in > > the 500 to 600 range (given a quiescent cluster). > > This of course can change dramatically with non-direct writes and caching > > (kernel page cache and/or RBD client caches). > > > > The same is true for reads, if your data set fits into the page caches of > > your storage nodes, it will be fast, if everything needs to be read from > > the HDDs, you're back to what these devices can do (~100 IOPS per HDD). > > > > To give you a concrete example, on my test cluster I have 5 nodes, 4 > > HDDs/OSDs each and no journal SSDs. > > So that's in theory 100 IOPS per HDD, divided by 2 for the on-disk journal, > > divided by 3 for replication: > > 20*100/2/3=333 > > Which amazingly is what I get with rados bench and 4K blocks, fio from a > > kernel client and direct I/O is around 200. > > > > BW, as in throughput is easier, about 350MB/s max for sustained sequential > > writes (the limit of the journal SSD) and lets say 750MB/s for sustained > > reads. > > Again, if you're reading just 8GB in your tests and that fits nicely in > > the page caches of the OSDs, it will be wire speed. > > > > >Should I configure a replica factor of 3? > > > > > If you value your data, which you will on a production server, then yes. > > This will of course cost you 1/3 of your performance compared to replica 2. > > > > Regards, > > > > Christian > > -- > > Christian Balzer Network/Systems Engineer > > [email protected] Global OnLine Japan/Fusion Communications > > http://www.gol.com/ > Please visit our new website at www.pml.ac.uk and follow us on Twitter > @PlymouthMarine > > Winner of the Environment & Conservation category, the Charity Awards 2014. > > Plymouth Marine Laboratory (PML) is a company limited by guarantee registered > in England & Wales, company number 4178503. Registered Charity No. 1091222. > Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK. > > This message is private and confidential. If you have received this message > in error, please notify the sender and remove it from your system. You are > reminded that e-mail communications are not secure and may contain viruses; > PML accepts no liability for any loss or damage which may be caused by > viruses.
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
