Hello,
I'm evaluate ceph cluster, to see if you can use it for our
virtualization solution (proxmox). I'm using 3 nodes, running Ubuntu
16.04 with stock ceph (10.2.6), every OSD uses separate 8 TB spinning
drive (XFS), MONITORs are installed on the same nodes, all nodes are
connected via 10G switch.
The problem is, on client I have only ~25-30 MB/s with seq. write. (dd
with "oflag=direct"). Proxmox uses Firefly, which is old, I know. But
I have the same performance on my desktop running the same version as
ceph nodes using rbd mount, iperf shows full speed (1GB or 10GB up to
client).
I know that this setup is not optimal and for production I will use
separate MON nodes and ssd for OSDs, but was wondering is this
performance still normal. This is my cluster status.
cluster 3ea55c7e-5829-46d0-b83a-92c6798bde55
health HEALTH_OK
monmap e5: 3 mons at
{ceph01=10.1.8.31:6789/0,ceph02=10.1.8.32:6789/0,ceph03=10.1.8.33:6789/0}
election epoch 60, quorum 0,1,2 ceph01,ceph02,ceph03
osdmap e570: 42 osds: 42 up, 42 in
flags sortbitwise,require_jewel_osds
pgmap v14784: 1024 pgs, 1 pools, 23964 MB data, 6047 objects
74743 MB used, 305 TB / 305 TB avail
1024 active+clean
btw, bench on nodes itself looks good as far I see.
ceph01:~# rados bench -p rbd 10 write
....
Total time run: 10.159667
Total writes made: 1018
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 400.801
Stddev Bandwidth: 38.2018
Max bandwidth (MB/sec): 472
Min bandwidth (MB/sec): 344
Average IOPS: 100
Stddev IOPS: 9
Max IOPS: 118
Min IOPS: 86
Average Latency(s): 0.159395
Stddev Latency(s): 0.110994
Max latency(s): 1.1069
Min latency(s): 0.0432668
Thanks,
Stan
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
