Thanks for your willingness to help DELL R620, 1 CPU, 8 cores, 64 GB RAM cluster network is using 2 bonded 10 GB NICs ( mode=4), MTU=9000
SSD drives are Enterprise grade - 400 GB SSD Toshiba PX04SHB040 HDD drives are - 10k RPM, 600 GB Toshiba AL13SEB600 Steven On 3 January 2018 at 09:41, Brady Deetz <bde...@gmail.com> wrote: > Can you provide more detail regarding the infrastructure backing this > environment? What hard drive, ssd, and processor are you using? Also, what > is providing networking? > > I'm seeing 4k blocksize tests here. Latency is going to destroy you. > > On Jan 3, 2018 8:11 AM, "Steven Vacaroaia" <ste...@gmail.com> wrote: > >> Hi, >> >> I am doing a PoC with 3 DELL R620 and 12 OSD , 3 SSD drives ( one on each >> server), bluestore >> >> I configured the OSD using the following ( /dev/sda is my SSD drive) >> ceph-disk prepare --zap-disk --cluster ceph --bluestore /dev/sde >> --block.wal /dev/sda --block.db /dev/sda >> >> Unfortunately both fio and bench tests show much worse performance for >> the pools than for the individual disks >> >> Example: >> DISKS >> fio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4k >> --numjobs=14 --iodepth=1 --runtime=60 --time_based --group_reporting >> --name=journal-test >> >> SSD drive >> Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/465.2MB/0KB /s] [0/119K/0 >> iops] [eta 00m:00s] >> >> HD drive >> Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/179.2MB/0KB /s] [0/45.9K/0 >> iops] [eta 00m:00s] >> >> POOL >> >> fio write.fio >> Jobs: 1 (f=0): [w(1)] [100.0% done] [0KB/51428KB/0KB /s] [0/12.9K/0 iops] >> >> cat write.fio >> [write-4M] >> description="write test with 4k block" >> ioengine=rbd >> clientname=admin >> pool=scbench >> rbdname=image01 >> iodepth=32 >> runtime=120 >> rw=randwrite >> bs=4k >> >> >> rados bench -p scbench 12 write >> >> Max bandwidth (MB/sec): 224 >> Min bandwidth (MB/sec): 0 >> Average IOPS: 26 >> Stddev IOPS: 24 >> Max IOPS: 56 >> Min IOPS: 0 >> Average Latency(s): 0.59819 >> Stddev Latency(s): 1.64017 >> Max latency(s): 10.8335 >> Min latency(s): 0.00475139 >> >> >> >> >> I must be missing something - any help/suggestions will be greatly >> appreciated >> >> Here are some specific info >> >> ceph -s >> cluster: >> id: 91118dde-f231-4e54-a5f0-a1037f3d5142 >> health: HEALTH_OK >> >> services: >> mon: 1 daemons, quorum mon01 >> mgr: mon01(active) >> osd: 12 osds: 12 up, 12 in >> >> data: >> pools: 4 pools, 484 pgs >> objects: 70082 objects, 273 GB >> usage: 570 GB used, 6138 GB / 6708 GB avail >> pgs: 484 active+clean >> >> io: >> client: 2558 B/s rd, 2 op/s rd, 0 op/s wr >> >> ceph osd pool ls detail >> pool 1 'test-replicated' replicated size 2 min_size 1 crush_rule 0 >> object_hash rjenkins pg_num 128 pgp_num 128 last_change 157 flags >> hashpspool stripe_width 0 application rbd >> removed_snaps [1~3] >> pool 2 'test-erasure' erasure size 3 min_size 3 crush_rule 1 object_hash >> rjenkins pg_num 128 pgp_num 128 last_change 334 flags hashpspool >> stripe_width 8192 application rbd >> removed_snaps [1~5] >> pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash >> rjenkins pg_num 128 pgp_num 128 last_change 200 flags hashpspool >> stripe_width 0 application rbd >> removed_snaps [1~3] >> pool 4 'scbench' replicated size 2 min_size 1 crush_rule 0 object_hash >> rjenkins pg_num 100 pgp_num 100 last_change 330 flags hashpspool >> stripe_width 0 >> removed_snaps [1~3] >> >> [cephuser@ceph ceph-config]$ ceph osd df tree >> ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME >> -1 6.55128 - 2237G 198G 2038G 0 0 - root default >> -7 0 - 0 0 0 0 0 - host ods03 >> -3 2.18475 - 2237G 181G 2055G 8.12 0.96 - host osd01 >> 3 hdd 0.54619 1.00000 559G 53890M 506G 9.41 1.11 90 osd.3 >> 4 hdd 0.54619 1.00000 559G 30567M 529G 5.34 0.63 89 osd.4 >> 5 hdd 0.54619 1.00000 559G 59385M 501G 10.37 1.22 93 osd.5 >> 6 hdd 0.54619 1.00000 559G 42156M 518G 7.36 0.87 93 osd.6 >> -5 2.18178 - 2234G 189G 2044G 8.50 1.00 - host osd02 >> 0 hdd 0.54520 1.00000 558G 32460M 526G 5.68 0.67 90 osd.0 >> 1 hdd 0.54520 1.00000 558G 54578M 504G 9.55 1.12 89 osd.1 >> 2 hdd 0.54520 1.00000 558G 47761M 511G 8.35 0.98 93 osd.2 >> 7 hdd 0.54619 1.00000 559G 59584M 501G 10.40 1.22 92 osd.7 >> -9 2.18475 - 2237G 198G 2038G 8.88 1.04 - host osd03 >> 8 hdd 0.54619 1.00000 559G 52462M 508G 9.16 1.08 99 osd.8 >> 10 hdd 0.54619 1.00000 559G 35284M 524G 6.16 0.73 88 osd.10 >> 11 hdd 0.54619 1.00000 559G 71739M 489G 12.53 1.47 87 osd.11 >> 12 hdd 0.54619 1.00000 559G 43832M 516G 7.65 0.90 93 osd.12 >> TOTAL 6708G 570G 6138G 8.50 >> MIN/MAX VAR: 0.63/1.47 STDDEV: 2.06 >> >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >>
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
