Can you provide more detail regarding the infrastructure backing this environment? What hard drive, ssd, and processor are you using? Also, what is providing networking?
I'm seeing 4k blocksize tests here. Latency is going to destroy you. On Jan 3, 2018 8:11 AM, "Steven Vacaroaia" <[email protected]> wrote: > Hi, > > I am doing a PoC with 3 DELL R620 and 12 OSD , 3 SSD drives ( one on each > server), bluestore > > I configured the OSD using the following ( /dev/sda is my SSD drive) > ceph-disk prepare --zap-disk --cluster ceph --bluestore /dev/sde > --block.wal /dev/sda --block.db /dev/sda > > Unfortunately both fio and bench tests show much worse performance for the > pools than for the individual disks > > Example: > DISKS > fio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4k > --numjobs=14 --iodepth=1 --runtime=60 --time_based --group_reporting > --name=journal-test > > SSD drive > Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/465.2MB/0KB /s] [0/119K/0 > iops] [eta 00m:00s] > > HD drive > Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/179.2MB/0KB /s] [0/45.9K/0 > iops] [eta 00m:00s] > > POOL > > fio write.fio > Jobs: 1 (f=0): [w(1)] [100.0% done] [0KB/51428KB/0KB /s] [0/12.9K/0 iops] > > cat write.fio > [write-4M] > description="write test with 4k block" > ioengine=rbd > clientname=admin > pool=scbench > rbdname=image01 > iodepth=32 > runtime=120 > rw=randwrite > bs=4k > > > rados bench -p scbench 12 write > > Max bandwidth (MB/sec): 224 > Min bandwidth (MB/sec): 0 > Average IOPS: 26 > Stddev IOPS: 24 > Max IOPS: 56 > Min IOPS: 0 > Average Latency(s): 0.59819 > Stddev Latency(s): 1.64017 > Max latency(s): 10.8335 > Min latency(s): 0.00475139 > > > > > I must be missing something - any help/suggestions will be greatly > appreciated > > Here are some specific info > > ceph -s > cluster: > id: 91118dde-f231-4e54-a5f0-a1037f3d5142 > health: HEALTH_OK > > services: > mon: 1 daemons, quorum mon01 > mgr: mon01(active) > osd: 12 osds: 12 up, 12 in > > data: > pools: 4 pools, 484 pgs > objects: 70082 objects, 273 GB > usage: 570 GB used, 6138 GB / 6708 GB avail > pgs: 484 active+clean > > io: > client: 2558 B/s rd, 2 op/s rd, 0 op/s wr > > ceph osd pool ls detail > pool 1 'test-replicated' replicated size 2 min_size 1 crush_rule 0 > object_hash rjenkins pg_num 128 pgp_num 128 last_change 157 flags > hashpspool stripe_width 0 application rbd > removed_snaps [1~3] > pool 2 'test-erasure' erasure size 3 min_size 3 crush_rule 1 object_hash > rjenkins pg_num 128 pgp_num 128 last_change 334 flags hashpspool > stripe_width 8192 application rbd > removed_snaps [1~5] > pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash > rjenkins pg_num 128 pgp_num 128 last_change 200 flags hashpspool > stripe_width 0 application rbd > removed_snaps [1~3] > pool 4 'scbench' replicated size 2 min_size 1 crush_rule 0 object_hash > rjenkins pg_num 100 pgp_num 100 last_change 330 flags hashpspool > stripe_width 0 > removed_snaps [1~3] > > [cephuser@ceph ceph-config]$ ceph osd df tree > ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME > -1 6.55128 - 2237G 198G 2038G 0 0 - root default > -7 0 - 0 0 0 0 0 - host ods03 > -3 2.18475 - 2237G 181G 2055G 8.12 0.96 - host osd01 > 3 hdd 0.54619 1.00000 559G 53890M 506G 9.41 1.11 90 osd.3 > 4 hdd 0.54619 1.00000 559G 30567M 529G 5.34 0.63 89 osd.4 > 5 hdd 0.54619 1.00000 559G 59385M 501G 10.37 1.22 93 osd.5 > 6 hdd 0.54619 1.00000 559G 42156M 518G 7.36 0.87 93 osd.6 > -5 2.18178 - 2234G 189G 2044G 8.50 1.00 - host osd02 > 0 hdd 0.54520 1.00000 558G 32460M 526G 5.68 0.67 90 osd.0 > 1 hdd 0.54520 1.00000 558G 54578M 504G 9.55 1.12 89 osd.1 > 2 hdd 0.54520 1.00000 558G 47761M 511G 8.35 0.98 93 osd.2 > 7 hdd 0.54619 1.00000 559G 59584M 501G 10.40 1.22 92 osd.7 > -9 2.18475 - 2237G 198G 2038G 8.88 1.04 - host osd03 > 8 hdd 0.54619 1.00000 559G 52462M 508G 9.16 1.08 99 osd.8 > 10 hdd 0.54619 1.00000 559G 35284M 524G 6.16 0.73 88 osd.10 > 11 hdd 0.54619 1.00000 559G 71739M 489G 12.53 1.47 87 osd.11 > 12 hdd 0.54619 1.00000 559G 43832M 516G 7.65 0.90 93 osd.12 > TOTAL 6708G 570G 6138G 8.50 > MIN/MAX VAR: 0.63/1.47 STDDEV: 2.06 > > > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
