> SSD journals will really help to get the full IOPS out of each disk. > Make sure the SSD has enough write speed to match the OSDs using it. > ie, if your SSDs can write 400MB/s, and the OSDs can write 100MB/s, then you > only want 4 OSDs sharing an SSD for journals. >
I don't think looking at raw write speeds is a good indication for SSD / disk ratio: The chances that you are only doing sequential IO across all those disks at full speed to potentially reach those speeds are practically zero. However, reducing the wear on those SSDs and minimizing impact when one does fail are good reasons to keep the ratio low. Make sure you have SSDs which can handle lots of writes before the fail (eg Intel dc3700) > Make sure you have enough network bandwidth to handle all of the OSDs. > 10x disks at 100 MB/s is 1 GB/s. You'll need 10GigE to handle that. For "regular" VM operations you will probably be maxed out on IOPS bandwidth before the network comes in to play. Note that for each node you add the total bandwidth of the cluster is increased. 10Gb is very nice for replication traffic though, e.g. when a node fails. Especially if you intend to create hosts with lots of large disks. > To really get the best performance, you need more money and a lot of testing. > :-) Yep, sure do. >From my experience (40 OSDs, SSD journals + spinners, no client write >caching): * Getting enough IOPS for a lot OpenStack instances doing some IO is pretty easy since the writes are nicely spread out. * A lot of sequential IO from a few machines is also no problem. * Doing lots of small random write IO from a single instance (e.g. a Mysql server trying to do 500+ IOPS) didn't work out so well. If you really need consistent fast IO for database like loads I think you should probably go with SSD only (pools). (I'd be happy to hear the numbers about running high random write loads on it :) And another nice hardware scaling PDF from dreamhost... https://objects.dreamhost.com/inktankweb/Inktank_Hardware_Configuration_Guide.pdf Cheers, Robert van Leeuwen
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
