Thanks for helping
I thought that with CEPH, the more servers you have the better the
performance
- that is why I am so confused
Also, I tried to add the 4th server ( still no luck - in fact the rado
bench output I included was from 4 servers, one OSD on each, bluestore,
replication 2 )
Here is the same rados bench but with only 2 servers/ 2 OSDs
rados bench -p rbd 120 write --no-cleanup && rados bench -p rbd 120 seq
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size
4194304 for up to 120 seconds or 0 objects
Object prefix: benchmark_data_osd01_384454
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0 0 0 0 -
0
1 16 101 85 339.981 340 0.244679
0.160362
2 16 149 133 265.972 192 0.303997
0.226268
3 16 197 181 241.307 192 0.179013
0.255609
4 16 241 225 224.975 176 0.353464
0.272623
5 16 289 273 218.376 192 0.303821
0.282425
6 16 338 322 214.643 196 0.326009
0.29105
7 16 387 371 211.977 196 0.27048
0.296497
8 16 436 420 209.977 196 0.287188
0.299224
9 16 479 463 205.755 172 0.380512
0.302272
10 16 527 511 204.378 192 0.289532
0.306163
11 16 576 560 203.614 196 0.406783
0.309271
12 16 624 608 202.645 192 0.282266
0.312167
13 16 667 651 200.286 172 0.377555
0.313733
14 16 716 700 199.978 196 0.350938
0.315445
15 16 764 748 199.445 192 0.183931
0.317474
Here is my iperf3 results
0.00-1.00 sec 1.15 GBytes 9.92 Gbits/sec 0 850 KBytes
[ 4] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 850 KBytes
On Thu, 19 Apr 2018 at 08:28, Marc Roos <[email protected]> wrote:
>
> If I may guess, because with 3 it reads from 3 and with 2 it reads only
> from 2. You should be able to verify this with something like dstat -d
> -D sda,sdb,sdc,sdd,sde,sdf,sdg not?
>
> With replication of 2, objects are still being stored among the 3 nodes.
>
> I am getting with iperf3 on 10Gbit
> [ ID] Interval Transfer Bandwidth Retr Cwnd
> [ 4] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec 0 1.31 MBytes
> [ 4] 10.00-20.00 sec 11.5 GBytes 9.89 Gbits/sec 0 1.79 MBytes
>
>
>
> -----Original Message-----
> From: Steven Vacaroaia [mailto:[email protected]]
> Sent: donderdag 19 april 2018 14:11
> To: ceph-users
> Subject: [ceph-users] ceph luminous 12.2.4 - 2 servers better than 3 ?
>
> Hi,
>
> Any idea why 2 servers with one OSD each will provide better performance
> than 3 ?
>
> Servers are identical
> Performance is impacted irrespective if I used SSD for WAL/DB or not
> Basically, I am getting lots of cur MB/s zero
>
> Network is separate 10 GB for public and private I tested it with iperf
> and I am getting 9.3 Gbs
>
> I have tried replication by 2 and 3 with same results ( much better for
> 2 servers than 3 )
>
> reinstalled CEPH multiple times
> ceph.conf very simple - no major customization ( see below) I am out of
> ideas - any hint will be TRULY appreciated
>
> Steven
>
>
>
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
>
>
> public_network = 10.10.30.0/24
> cluster_network = 192.168.0.0/24
>
>
> osd_pool_default_size = 2
> osd_pool_default_min_size = 1 # Allow writing 1 copy in a degraded state
> osd_crush_chooseleaf_type = 1
>
>
> [mon]
> mon_allow_pool_delete = true
> mon_osd_min_down_reporters = 1
>
> [osd]
> osd_mkfs_type = xfs
> osd_mount_options_xfs =
> "rw,noatime,nodiratime,attr2,logbufs=8,logbsize=256k,largeio,inode64,swa
> lloc,allocsize=4M"
> osd_mkfs_options_xfs = "-f -i size=2048"
> bluestore_block_db_size = 32212254720
> bluestore_block_wal_size = 1073741824
>
> rados bench -p rbd 120 write --no-cleanup && rados bench -p rbd 120 seq
> hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects
> of size 4194304 for up to 120 seconds or 0 objects Object prefix:
> benchmark_data_osd01_383626
> sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
> lat(s)
> 0 0 0 0 0 0 -
> 0
> 1 16 57 41 163.991 164 0.197929
> 0.065543
> 2 16 57 41 81.992 0 -
> 0.065543
> 3 16 67 51 67.9936 20 0.0164632
> 0.249939
> 4 16 67 51 50.9951 0 -
> 0.249939
> 5 16 71 55 43.9958 8 0.0171439
> 0.319973
> 6 16 181 165 109.989 440 0.0159057
> 0.563746
> 7 16 182 166 94.8476 4 0.221421
> 0.561684
> 8 16 182 166 82.9917 0 -
> 0.561684
> 9 16 240 224 99.5458 116 0.0232989
> 0.638292
> 10 16 264 248 99.1901 96 0.0222669
> 0.583336
> 11 16 264 248 90.1729 0 -
> 0.583336
> 12 16 285 269 89.6579 42 0.0165706
> 0.600606
> 13 16 285 269 82.7611 0 -
> 0.600606
> 14 16 310 294 83.9918 50 0.0254241
> 0.756351
>
>
>
>
>
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
