Sorry for the delay Here are the results when using bs=16k and rw=write ( Note: I am running the command directly on a OSD host as root)
fio /home/cephuser/write.fio write-4M: (g=0): rw=write, bs=16K-16K/16K-16K/16K-16K, ioengine=rbd, iodepth=32 fio-2.2.8 Starting 1 process rbd engine: RBD version: 1.12.0 Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/172.6MB/0KB /s] [0/11.5K/0 iops] [eta 00m:00s] Here are the results when runnnig with bs=4k and rw=randwrite [root@osd03 ~]# fio /home/cephuser/write.fio write-4M: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=32 fio-2.2.8 Starting 1 process rbd engine: RBD version: 1.12.0 Jobs: 1 (f=0): [w(1)] [100.0% done] [0KB/54056KB/0KB /s] [0/13.6K/0 iops] [eta 00m:00s] On 3 January 2018 at 15:28, <ceph.nov...@habmalnefrage.de> wrote: > Hi Steven. > > interesting... 'm quite curious after your post now. > > I've migrated our prod. CEPH cluster to 12.2.2 and Bluestore just today > and haven't heard back anything "bad" from the applications/users so far. > performance tests on our test cluster were good before, but we use S3/RGW > only anyhow ;) > > there are two things I would like to know/learn... could you try/test and > feed back?! > > - change all your tests to use >=16k block size, see also BStore comments > here (https://www.mail-archive.com/ceph-users@lists.ceph.com/msg43023.html > ) > - change your "write.fio" file profile from "rw=randwrite" to "rw=write" > (or something similar :O ) to compare apples with apples ;) > > thanks for your efforts and looking forward for those results ;) > > best regards > Notna > > ------------------------------------------------------------ > ---------------------- > > Gesendet: Mittwoch, 03. Januar 2018 um 16:20 Uhr > Von: "Steven Vacaroaia" <ste...@gmail.com> > An: "Brady Deetz" <bde...@gmail.com> > Cc: ceph-users <ceph-users@lists.ceph.com> > Betreff: Re: [ceph-users] ceph luminous - performance issue > > Thanks for your willingness to help > > DELL R620, 1 CPU, 8 cores, 64 GB RAM > cluster network is using 2 bonded 10 GB NICs ( mode=4), MTU=9000 > > SSD drives are Enterprise grade - 400 GB SSD Toshiba PX04SHB040 > HDD drives are - 10k RPM, 600 GB Toshiba AL13SEB600 > > Steven > > > On 3 January 2018 at 09:41, Brady Deetz <bde...@gmail.com[mailto:bdeet > z...@gmail.com]> wrote: > Can you provide more detail regarding the infrastructure backing this > environment? What hard drive, ssd, and processor are you using? Also, what > is providing networking? > > I'm seeing 4k blocksize tests here. Latency is going to destroy you. > > > On Jan 3, 2018 8:11 AM, "Steven Vacaroaia" <ste...@gmail.com[mailto:stef9 > 7...@gmail.com]> wrote: > > Hi, > > I am doing a PoC with 3 DELL R620 and 12 OSD , 3 SSD drives ( one on each > server), bluestore > > I configured the OSD using the following ( /dev/sda is my SSD drive) > ceph-disk prepare --zap-disk --cluster ceph --bluestore /dev/sde > --block.wal /dev/sda --block.db /dev/sda > > Unfortunately both fio and bench tests show much worse performance for the > pools than for the individual disks > > Example: > DISKS > fio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4k > --numjobs=14 --iodepth=1 --runtime=60 --time_based --group_reporting > --name=journal-test > > SSD drive > Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/465.2MB/0KB /s] [0/119K/0 > iops] [eta 00m:00s] > > HD drive > Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/179.2MB/0KB /s] [0/45.9K/0 > iops] [eta 00m:00s] > > POOL > > fio write.fio > Jobs: 1 (f=0): [w(1)] [100.0% done] [0KB/51428KB/0KB /s] [0/12.9K/0 iops] > > > cat write.fio > [write-4M] > description="write test with 4k block" > ioengine=rbd > clientname=admin > pool=scbench > rbdname=image01 > iodepth=32 > runtime=120 > rw=randwrite > bs=4k > > > rados bench -p scbench 12 write > > > Max bandwidth (MB/sec): 224 > Min bandwidth (MB/sec): 0 > Average IOPS: 26 > Stddev IOPS: 24 > Max IOPS: 56 > Min IOPS: 0 > Average Latency(s): 0.59819 > Stddev Latency(s): 1.64017 > Max latency(s): 10.8335 > Min latency(s): 0.00475139 > > > > > I must be missing something - any help/suggestions will be greatly > appreciated > > Here are some specific info > > > ceph -s > cluster: > id: 91118dde-f231-4e54-a5f0-a1037f3d5142 > health: HEALTH_OK > > services: > mon: 1 daemons, quorum mon01 > mgr: mon01(active) > osd: 12 osds: 12 up, 12 in > > data: > pools: 4 pools, 484 pgs > objects: 70082 objects, 273 GB > usage: 570 GB used, 6138 GB / 6708 GB avail > pgs: 484 active+clean > > io: > client: 2558 B/s rd, 2 op/s rd, 0 op/s wr > > > ceph osd pool ls detail > pool 1 'test-replicated' replicated size 2 min_size 1 crush_rule 0 > object_hash rjenkins pg_num 128 pgp_num 128 last_change 157 flags > hashpspool stripe_width 0 application rbd > removed_snaps [1~3] > pool 2 'test-erasure' erasure size 3 min_size 3 crush_rule 1 object_hash > rjenkins pg_num 128 pgp_num 128 last_change 334 flags hashpspool > stripe_width 8192 application rbd > removed_snaps [1~5] > pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash > rjenkins pg_num 128 pgp_num 128 last_change 200 flags hashpspool > stripe_width 0 application rbd > removed_snaps [1~3] > pool 4 'scbench' replicated size 2 min_size 1 crush_rule 0 object_hash > rjenkins pg_num 100 pgp_num 100 last_change 330 flags hashpspool > stripe_width 0 > removed_snaps [1~3] > > [cephuser@ceph ceph-config]$ ceph osd df tree > ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME > -1 6.55128 - 2237G 198G 2038G 0 0 - root default > -7 0 - 0 0 0 0 0 - host ods03 > -3 2.18475 - 2237G 181G 2055G 8.12 0.96 - host osd01 > 3 hdd 0.54619 1.00000 559G 53890M 506G 9.41 1.11 90 osd.3 > 4 hdd 0.54619 1.00000 559G 30567M 529G 5.34 0.63 89 osd.4 > 5 hdd 0.54619 1.00000 559G 59385M 501G 10.37 1.22 93 osd.5 > 6 hdd 0.54619 1.00000 559G 42156M 518G 7.36 0.87 93 osd.6 > -5 2.18178 - 2234G 189G 2044G 8.50 1.00 - host osd02 > 0 hdd 0.54520 1.00000 558G 32460M 526G 5.68 0.67 90 osd.0 > 1 hdd 0.54520 1.00000 558G 54578M 504G 9.55 1.12 89 osd.1 > 2 hdd 0.54520 1.00000 558G 47761M 511G 8.35 0.98 93 osd.2 > 7 hdd 0.54619 1.00000 559G 59584M 501G 10.40 1.22 92 osd.7 > -9 2.18475 - 2237G 198G 2038G 8.88 1.04 - host osd03 > 8 hdd 0.54619 1.00000 559G 52462M 508G 9.16 1.08 99 osd.8 > 10 hdd 0.54619 1.00000 559G 35284M 524G 6.16 0.73 88 osd.10 > 11 hdd 0.54619 1.00000 559G 71739M 489G 12.53 1.47 87 osd.11 > 12 hdd 0.54619 1.00000 559G 43832M 516G 7.65 0.90 93 osd.12 > TOTAL 6708G 570G 6138G 8.50 > MIN/MAX VAR: 0.63/1.47 STDDEV: 2.06 > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com[mailto:ceph-users@lists.ceph.com] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list > ceph-users@lists.ceph.com http://lists.ceph.com/ > listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/ > listinfo.cgi/ceph-users-ceph.com] > > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
