Re: [ceph-users] [EXTERNAL] Ceph performance is too good (impossible..)...

2016-12-16 Thread Mike Miller
Hi, you need to flush all caches before starting read tests. With fio you can probably do this if you keep the files that it creates. as root on all clients and all osd nodes run: echo 3 > /proc/sys/vm/drop_caches But fio is a little problematic for ceph because of the caches in the clients

Re: [ceph-users] [EXTERNAL] Ceph performance is too good (impossible..)...

2016-12-12 Thread V Plus
The same.. see: A: (g=0): rw=read, bs=5M-5M/5M-5M/5M-5M, ioengine=*libaio*, iodepth=1 ... fio-2.2.10 Starting 16 processes A: (groupid=0, jobs=16): err= 0: pid=27579: Mon Dec 12 20:36:10 2016 mixed: io=122515MB, bw=6120.3MB/s, iops=1224, runt= 20018msec I think at the end, the only one way to s

Re: [ceph-users] [EXTERNAL] Ceph performance is too good (impossible..)...

2016-12-12 Thread Will . Boege
My understanding is that when using direct=1 on a raw block device FIO (aka-you) will have to handle all the sector alignment or the request will get buffered to perform the alignment. Try adding the –blockalign=512b option to your jobs, or better yet just use the native FIO RBD engine. Someth