Re: [ceph-users] [EXTERNAL] Ceph performance is too good (impossible..)...
Hi, you need to flush all caches before starting read tests. With fio you can probably do this if you keep the files that it creates. as root on all clients and all osd nodes run: echo 3 > /proc/sys/vm/drop_caches But fio is a little problematic for ceph because of the caches in the clients and the osd nodes. If you really need to know the read rates, go for large files, write them with dd, flush all caches, and the read the files with dd. Single threaded read dd shows less throughput compared to multiple threads dd read. readahead size does also matter. Good luck testing. - mike On 12/13/16 2:37 AM, V Plus wrote: The same.. see: A: (g=0): rw=read, bs=5M-5M/5M-5M/5M-5M, ioengine=*libaio*, iodepth=1 ... fio-2.2.10 Starting 16 processes A: (groupid=0, jobs=16): err= 0: pid=27579: Mon Dec 12 20:36:10 2016 mixed: io=122515MB, bw=6120.3MB/s, iops=1224, runt= 20018msec I think at the end, the only one way to solve this issue is to write the image before read testas suggested I have no clue why rbd engine does not work... On Mon, Dec 12, 2016 at 4:23 PM, Will.Boege <will.bo...@target.com <mailto:will.bo...@target.com>> wrote: Try adding --ioengine=libaio __ __ *From: *V Plus <v.plussh...@gmail.com <mailto:v.plussh...@gmail.com>> *Date: *Monday, December 12, 2016 at 2:40 PM *To: *"Will.Boege" <will.bo...@target.com <mailto:will.bo...@target.com>> *Subject: *Re: [EXTERNAL] [ceph-users] Ceph performance is too good (impossible..)... __ __ Hi Will, thanks very much.. However, I tried with your suggestions. Both are *not *working... 1. with FIO rbd engine: *[RBD_TEST] ioengine=rbd clientname=admin pool=rbd rbdname=fio_test invalidate=1 direct=1 group_reporting=1 unified_rw_reporting=1 time_based=1 rw=read bs=4MB numjobs=16 ramp_time=10 runtime=20* then I run "sudo fio rbd.job" and got: RBD_TEST: (g=0): rw=read, bs=4M-4M/4M-4M/4M-4M, ioengine=rbd, iodepth=1 ... fio-2.2.10 Starting 16 processes rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 rbd engine: RBD version: 0.1.10 Jobs: 12 (f=7): [R(5),_(1),R(4),_(1),R(2),_(2),R(1)] [100.0% done] [11253MB/0KB/0KB /s] [2813/0/0 iops] [eta 00m:00s] RBD_TEST: (groupid=0, jobs=16): err= 0: pid=17504: Mon Dec 12 15:32:52 2016 mixed: io=212312MB, *bw=10613MB/s*, iops=2653, runt= 20005msec 2. with blockalign [A] direct=1 group_reporting=1 unified_rw_reporting=1 size=100% time_based=1 filename=/dev/rbd0 rw=read bs=5MB numjobs=16 ramp_time=5 runtime=20 blockalign=512b [B] direct=1 group_reporting=1 unified_rw_reporting=1 size=100% time_based=1 filename=/dev/rbd1 rw=read bs=5MB numjobs=16 ramp_time=5 runtime=20 blockalign=512b sudo fio fioA.job -output a.txt & sudo fio fioB.job -output b.txt & wait Then I got: A: (groupid=0, jobs=16): err= 0: pid=19320: Mon Dec 12 15:35:32 2016 mixed: io=88590MB, bw=4424.7MB/s, iops=884, runt= 20022msec B: (groupid=0, jobs=16): err= 0: pid=19324: Mon Dec 12 15:35:32 2016 mixed: io=88020MB, *bw=4395.6MB/s*, iops=879, runt= 20025msec .. __ __ On Mon, Dec 12, 2016 at 10:45 AM, Will.Boege <will.bo...@target.com <mailto:will.bo...@target.com>> wrote: My understanding is that when using direct=1 on a raw block device FIO (aka-you) will have to handle all the sector alignment or the request will get buffered to perform the alignment. Try adding the –blockalign=512b option to your jobs, or better yet just use the native FIO RBD engine. Something like this (untested) - [A] ioengine=rbd clientname=admin pool=rbd rbdname=fio_test direct=1 group_reporting=1 unified_rw_reporting=1 time_based=1 rw=read bs=4MB numjobs=16 ramp_time=10 runtime=20 *From: *ceph-users <ceph-users-boun...@lists.ceph.com <mailto:ceph-users-boun...@lists.ceph.c
Re: [ceph-users] [EXTERNAL] Ceph performance is too good (impossible..)...
The same.. see: A: (g=0): rw=read, bs=5M-5M/5M-5M/5M-5M, ioengine=*libaio*, iodepth=1 ... fio-2.2.10 Starting 16 processes A: (groupid=0, jobs=16): err= 0: pid=27579: Mon Dec 12 20:36:10 2016 mixed: io=122515MB, bw=6120.3MB/s, iops=1224, runt= 20018msec I think at the end, the only one way to solve this issue is to write the image before read testas suggested I have no clue why rbd engine does not work... On Mon, Dec 12, 2016 at 4:23 PM, Will.Boege <will.bo...@target.com> wrote: > Try adding --ioengine=libaio > > > > *From: *V Plus <v.plussh...@gmail.com> > *Date: *Monday, December 12, 2016 at 2:40 PM > *To: *"Will.Boege" <will.bo...@target.com> > *Subject: *Re: [EXTERNAL] [ceph-users] Ceph performance is too good > (impossible..)... > > > > Hi Will, > > thanks very much.. > > However, I tried with your suggestions. > > Both are *not *working... > > 1. with FIO rbd engine: > > > > > > > > > > > > > > > *[RBD_TEST] ioengine=rbd clientname=admin pool=rbd rbdname=fio_test > invalidate=1direct=1 group_reporting=1 unified_rw_reporting=1 > time_based=1 rw=read bs=4MB numjobs=16 ramp_time=10 runtime=20* > > then I run "sudo fio rbd.job" and got: > > > RBD_TEST: (g=0): rw=read, bs=4M-4M/4M-4M/4M-4M, ioengine=rbd, iodepth=1 > ... > fio-2.2.10 > Starting 16 processes > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > rbd engine: RBD version: 0.1.10 > Jobs: 12 (f=7): [R(5),_(1),R(4),_(1),R(2),_(2),R(1)] [100.0% done] > [11253MB/0KB/0KB /s] [2813/0/0 iops] [eta 00m:00s] > RBD_TEST: (groupid=0, jobs=16): err= 0: pid=17504: Mon Dec 12 15:32:52 2016 > mixed: io=212312MB, *bw=10613MB/s*, iops=2653, runt= 20005msec > > 2. with blockalign > [A] > direct=1 > group_reporting=1 > unified_rw_reporting=1 > size=100% > time_based=1 > filename=/dev/rbd0 > rw=read > bs=5MB > numjobs=16 > ramp_time=5 > runtime=20 > blockalign=512b > > [B] > direct=1 > group_reporting=1 > unified_rw_reporting=1 > size=100% > time_based=1 > filename=/dev/rbd1 > rw=read > bs=5MB > numjobs=16 > ramp_time=5 > runtime=20 > blockalign=512b > > sudo fio fioA.job -output a.txt & sudo fio fioB.job -output b.txt & wait > > Then I got: > A: (groupid=0, jobs=16): err= 0: pid=19320: Mon Dec 12 15:35:32 2016 > mixed: io=88590MB, bw=4424.7MB/s, iops=884, runt= 20022msec > > B: (groupid=0, jobs=16): err= 0: pid=19324: Mon Dec 12 15:35:32 2016 > mixed: io=88020MB, *bw=4395.6MB/s*, iops=879, runt= 20025msec > > .. > > > > On Mon, Dec 12, 2016 at 10:45 AM, Will.Boege <will.bo...@target.com> > wrote: > > My understanding is that when using direct=1 on a raw block device FIO > (aka-you) will have to handle all the sector alignment or the request will > get buffered to perform the alignment. > > > > Try adding the –blockalign=512b option to your jobs, or better yet just > use the native FIO RBD engine. > > > > Something like this (untested) - > > > > [A] > > ioengine=rbd > > clientname=admin > > pool=rbd > > rbdname=fio_test > > direct=1 > > group_reporting=1 > > unified_rw_reporting=1 > > time_based=1 > > rw=read > > bs=4MB > > numjobs=16 > > ramp_time=10 > > runtime=20 > > > > *From: *ceph-users <ceph-users-boun...@lists.ceph.com> on behalf of V > Plus <v.plussh...@gmail.com> > *Date: *Sunday, December 11, 2016 at 7:44 PM > *To: *"ceph-users@lists.ceph.com" <ceph-users@lists.ceph.com> > *Subject: *[EXTERNAL] [ceph-users] Ceph performance is too good > (impossible..)... > > > > Hi Guys, > > we have a ceph cluster with 6 machines (6 OSD per host). > > 1. I created 2 images in Ceph, and map them to another host A (*outside *the > Ceph cluster). On host A, I got */dev/rbd0* and* /dev/rbd1*. > > 2. I start two fio job to perform READ test on rbd0 and rbd1. (fio job > descriptions can be found below) > > *"sudo fio fioA.job -output a.txt & sudo fio fioB.job -output b.txt & > wait"* > > 3. After the test, in a.txt, we got *bw=1162.7MB/s*, in b.txt,
Re: [ceph-users] [EXTERNAL] Ceph performance is too good (impossible..)...
My understanding is that when using direct=1 on a raw block device FIO (aka-you) will have to handle all the sector alignment or the request will get buffered to perform the alignment. Try adding the –blockalign=512b option to your jobs, or better yet just use the native FIO RBD engine. Something like this (untested) - [A] ioengine=rbd clientname=admin pool=rbd rbdname=fio_test direct=1 group_reporting=1 unified_rw_reporting=1 time_based=1 rw=read bs=4MB numjobs=16 ramp_time=10 runtime=20 From: ceph-users <ceph-users-boun...@lists.ceph.com> on behalf of V Plus <v.plussh...@gmail.com> Date: Sunday, December 11, 2016 at 7:44 PM To: "ceph-users@lists.ceph.com" <ceph-users@lists.ceph.com> Subject: [EXTERNAL] [ceph-users] Ceph performance is too good (impossible..)... Hi Guys, we have a ceph cluster with 6 machines (6 OSD per host). 1. I created 2 images in Ceph, and map them to another host A (outside the Ceph cluster). On host A, I got /dev/rbd0 and /dev/rbd1. 2. I start two fio job to perform READ test on rbd0 and rbd1. (fio job descriptions can be found below) "sudo fio fioA.job -output a.txt & sudo fio fioB.job -output b.txt & wait" 3. After the test, in a.txt, we got bw=1162.7MB/s, in b.txt, we get bw=3579.6MB/s. The results do NOT make sense because there is only one NIC on host A, and its limit is 10 Gbps (1.25GB/s). I suspect it is because of the cache setting. But I am sure that in file /etc/ceph/ceph.conf on host A,I already added: [client] rbd cache = false Could anyone give me a hint what is missing? why Thank you very much. fioA.job: [A] direct=1 group_reporting=1 unified_rw_reporting=1 size=100% time_based=1 filename=/dev/rbd0 rw=read bs=4MB numjobs=16 ramp_time=10 runtime=20 fioB.job: [B] direct=1 group_reporting=1 unified_rw_reporting=1 size=100% time_based=1 filename=/dev/rbd1 rw=read bs=4MB numjobs=16 ramp_time=10 runtime=20 Thanks... ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
