If you're at all concerned with performance and the budget is set, drop a storage node and replace some osds with SSDs in the other nodes. We use 32x 4TB + 4x SSDs storage nodes and have 192GB of memory; 128GB wasn't enough. If you try to do this setup without SSD journals then you are going to scream at yourself in the future for not doing it.
We have 4x 200GB SSDs but only use 8x 10GB journals on each one. You want to pay attention to the speed of the SSD, not the size, when judging how many OSDs to assign their journals to it. Every write to each OSD is first written on the journal and then onto the drive. ________________________________ From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of Sergio A. de Carvalho Jr. [scarvalh...@gmail.com] Sent: Thursday, April 07, 2016 12:18 PM To: Alan Johnson Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Ceph performance expectations Thanks, Alan. Unfortunately, we currently don't have much flexibility in terms of the hardware we can get so adding SSDs might not be possible in the near future. What is the best practice here, allocating, for each OSD, one disk just for data and one disk just for thd journal? Since the journals are rather small (in our setup a 5GB partition is created on every disk), wouldn't this a bit of a waste of disk space? I was wondering if it would make sense to give each OSD one full 4TB disk and use one of the 900 GB disks for all journals (12 journals in this case). Would that cause even more contention since then different OSDs would then be trying to write their journals to the same disk? On Thu, Apr 7, 2016 at 4:13 PM, Alan Johnson <al...@supermicro.com<mailto:al...@supermicro.com>> wrote: I would strongly consider your journaling setup, (you do mention that you will revisit this) but we have found that co-locating journals does impact performance and usually separating them on flash is a good idea. Also not sure of your networking setup which can also have significant impact. From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com<mailto:ceph-users-boun...@lists.ceph.com>] On Behalf Of Sergio A. de Carvalho Jr. Sent: Thursday, April 07, 2016 5:01 AM To: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: [ceph-users] Ceph performance expectations Hi all, I've setup a testing/development Ceph cluster consisting of 5 Dell PowerEdge R720xd servers (256GB RAM, 2x 8-core Xeon E5-2650 @ 2.60 GHz, dual-port 10Gb Ethernet, 2x 900GB + 12x 4TB disks) running CentOS 6.5 and Ceph Hammer 0.94.6. All servers use one 900GB disk for the root partition and the other 13 disks are assigned to OSDs, so we have 5 x 13 = 65 OSDs in total. We also run 1 monitor on every host. Journals are 5GB partitions on each disk (this is something we obviously will need to revisit later). The purpose of this cluster will be to serve as a backend storage for Cinder volumes and Glance images in an OpenStack cloud. With this setup, I'm getting what I'm considering an "okay" performance: # rados -p images bench 5 write Maintaining 16 concurrent writes of 4194304 bytes for up to 5 seconds or 0 objects Total writes made: 394 Write size: 4194304 Bandwidth (MB/sec): 299.968 Stddev Bandwidth: 127.334 Max bandwidth (MB/sec): 348 Min bandwidth (MB/sec): 0 Average Latency: 0.212524 Stddev Latency: 0.13317 Max latency: 0.828946 Min latency: 0.0707341 Does that look acceptable? How much more can I expect to achieve by fine-tunning and perhaps using a more efficient setup? I do understand the bandwidth above is a product of running 16 concurrent writes, and rather small object sizes (4MB). Bandwidth lowers significantly with 64MB and 1 thread: # rados -p images bench 5 write -b 67108864 -t 1 Maintaining 1 concurrent writes of 67108864 bytes for up to 5 seconds or 0 objects Total writes made: 7 Write size: 67108864 Bandwidth (MB/sec): 71.520 Stddev Bandwidth: 24.1897 Max bandwidth (MB/sec): 64 Min bandwidth (MB/sec): 0 Average Latency: 0.894792 Stddev Latency: 0.0547502 Max latency: 0.99311 Min latency: 0.832765 Is such a drop expected? Now, what I'm really concerned is about upload times. Uploading a randomly-generated 1GB file takes a bit too long: # time rados -p images put random_1GB /tmp/random_1GB real 0m35.328s user 0m0.560s sys 0m3.665s Is this normal? If so, if I setup this cluster as a backend for Glance, does that mean uploading a 1GB image will require 35 seconds (plus whatever time Glance requires to do its own thing)? Thanks, Sergio
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
