I/O Pattern
XFS FileStore
LevelDB
IOPs
Avg. Latency
IOPs
Avg. Latency
4K randwrite
1415
22.55 msec
853
37.48 msec
64K randwrite
311
214.86 msec
328
97.42 msec
4K randread
9477
3.346 msec
3000
11 msec
64K randread
3961
8.072 msec
4000
8 msec
Based on the above, it appears that LevelDB performs better than
FileStore only for 64K random writes - the latency is particularly low
compared to FileStore.
For the rest of the workloads, XFS FileStore seems to perform better.
Can you please let me know any config values that can be tuned for
better performance? Currently I'm using the same ceph.conf as you posted
as part of this thread.
Appreciate all help in this regard.
Thanks,
Sushma
On Tue, Jun 3, 2014 at 12:06 AM, Haomai Wang <[email protected]
<mailto:[email protected]>> wrote:
I don't know the actual size of "small io". And what's ceph version
you used.
But I think it's possible if KeyValueStore only has half performance
compared to FileStore in small io size. A new config value let user
can tunes it will be introduced and maybe help.
All in all, maybe you could tell more about "ceph_smalliobench"
On Tue, Jun 3, 2014 at 1:36 PM, Sushma R <[email protected]
<mailto:[email protected]>> wrote:
> Hi Haomai,
>
> I tried to compare the READ performance of FileStore and
KeyValueStore using
> the internal tool "ceph_smalliobench" and I see KeyValueStore's
performance
> is approx half that of FileStore. I'm using similar conf file as
yours. Is
> this the expected behavior or am I missing something?
>
> Thanks,
> Sushma
>
>
> On Fri, Feb 28, 2014 at 11:00 PM, Haomai Wang
<[email protected] <mailto:[email protected]>> wrote:
>>
>> On Sat, Mar 1, 2014 at 8:04 AM, Danny Al-Gaaf
<[email protected] <mailto:[email protected]>>
>> wrote:
>> > Hi,
>> >
>> > Am 28.02.2014 03:45, schrieb Haomai Wang:
>> > [...]
>> >> I use fio which rbd supported from
>> >>
TelekomCloud(https://github.com/TelekomCloud/fio/commits/rbd-engine)
>> >> to test rbd.
>> >
>> > I would recommend to no longer use this branch, it's outdated.
The rbd
>> > engine got contributed back to upstream fio and is now merged
[1]. For
>> > more information read [2].
>> >
>> > [1] https://github.com/axboe/fio/commits/master
>> > [2]
>> >
>> >
http://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-analysis_fio_rbd.html
>> >
>> >>
>> >> The fio command: fio -direct=1 -iodepth=64 -thread -rw=randwrite
>> >> -ioengine=rbd -bs=4k -size=19G -numjobs=1 -runtime=100
>> >> -group_reporting -name=ebs_test -pool=openstack -rbdname=image
>> >> -clientname=fio -invalidate=0
>> >
>> > Don't use runtime and size at the same time, since runtime
limits the
>> > size. What we normally do we let the fio job fill up the whole
rbd or we
>> > limit it only via runtime.
>> >
>> >> ============================================
>> >>
>> >> FileStore result:
>> >> ebs_test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K,
ioengine=rbd,
>> >> iodepth=64
>> >> fio-2.1.4
>> >> Starting 1 thread
>> >> rbd engine: RBD version: 0.1.8
>> >>
>> >> ebs_test: (groupid=0, jobs=1): err= 0: pid=30886: Thu Feb 27
08:09:18
>> >> 2014
>> >> write: io=283040KB, bw=6403.4KB/s, iops=1600, runt= 44202msec
>> >> slat (usec): min=116, max=2817, avg=195.78, stdev=56.45
>> >> clat (msec): min=8, max=661, avg=39.57, stdev=29.26
>> >> lat (msec): min=9, max=661, avg=39.77, stdev=29.25
>> >> clat percentiles (msec):
>> >> | 1.00th=[ 15], 5.00th=[ 20], 10.00th=[ 23],
20.00th=[
>> >> 28],
>> >> | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 37],
60.00th=[
>> >> 40],
>> >> | 70.00th=[ 43], 80.00th=[ 46], 90.00th=[ 51],
95.00th=[
>> >> 58],
>> >> | 99.00th=[ 128], 99.50th=[ 210], 99.90th=[ 457],
99.95th=[
>> >> 494],
>> >> | 99.99th=[ 545]
>> >> bw (KB /s): min= 2120, max=12656, per=100.00%, avg=6464.27,
>> >> stdev=1726.55
>> >> lat (msec) : 10=0.01%, 20=5.91%, 50=83.35%, 100=8.88%,
250=1.47%
>> >> lat (msec) : 500=0.34%, 750=0.05%
>> >> cpu : usr=29.83%, sys=1.36%, ctx=84002, majf=0,
minf=216
>> >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
32=17.4%,
>> >> >=64=82.6%
>> >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
64=0.0%,
>> >> >=64=0.0%
>> >> complete : 0=0.0%, 4=99.1%, 8=0.5%, 16=0.3%, 32=0.1%,
64=0.1%,
>> >> >=64=0.0%
>> >> issued : total=r=0/w=70760/d=0, short=r=0/w=0/d=0
>> >> latency : target=0, window=0, percentile=100.00%, depth=64
>> >>
>> >> Run status group 0 (all jobs):
>> >> WRITE: io=283040KB, aggrb=6403KB/s, minb=6403KB/s,
maxb=6403KB/s,
>> >> mint=44202msec, maxt=44202msec
>> >>
>> >> Disk stats (read/write):
>> >> sdb: ios=5/9512, merge=0/69, ticks=4/10649, in_queue=10645,
>> >> util=0.92%
>> >>
>> >> ===============================================
>> >>
>> >> KeyValueStore:
>> >> ebs_test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K,
ioengine=rbd,
>> >> iodepth=64
>> >> fio-2.1.4
>> >> Starting 1 thread
>> >> rbd engine: RBD version: 0.1.8
>> >>
>> >> ebs_test: (groupid=0, jobs=1): err= 0: pid=29137: Thu Feb 27
08:06:30
>> >> 2014
>> >> write: io=444376KB, bw=6280.2KB/s, iops=1570, runt= 70759msec
>> >> slat (usec): min=122, max=3237, avg=184.51, stdev=37.76
>> >> clat (msec): min=10, max=168, avg=40.57, stdev= 5.70
>> >> lat (msec): min=11, max=168, avg=40.75, stdev= 5.71
>> >> clat percentiles (msec):
>> >> | 1.00th=[ 34], 5.00th=[ 37], 10.00th=[ 39],
20.00th=[
>> >> 39],
>> >> | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 41],
60.00th=[
>> >> 41],
>> >> | 70.00th=[ 42], 80.00th=[ 42], 90.00th=[ 44],
95.00th=[
>> >> 45],
>> >> | 99.00th=[ 48], 99.50th=[ 50], 99.90th=[ 163],
99.95th=[
>> >> 167],
>> >> | 99.99th=[ 167]
>> >> bw (KB /s): min= 4590, max= 7480, per=100.00%, avg=6285.69,
>> >> stdev=374.22
>> >> lat (msec) : 20=0.02%, 50=99.58%, 100=0.23%, 250=0.17%
>> >> cpu : usr=29.11%, sys=1.10%, ctx=118564, majf=0,
minf=194
>> >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
32=0.7%,
>> >> >=64=99.3%
>> >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
64=0.0%,
>> >> >=64=0.0%
>> >> complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.0%,
64=0.1%,
>> >> >=64=0.0%
>> >> issued : total=r=0/w=111094/d=0, short=r=0/w=0/d=0
>> >> latency : target=0, window=0, percentile=100.00%, depth=64
>> >>
>> >> Run status group 0 (all jobs):
>> >> WRITE: io=444376KB, aggrb=6280KB/s, minb=6280KB/s,
maxb=6280KB/s,
>> >> mint=70759msec, maxt=70759msec
>> >>
>> >> Disk stats (read/write):
>> >> sdb: ios=0/15936, merge=0/272, ticks=0/17157, in_queue=17146,
>> >> util=0.94%
>> >>
>> >>
>> >> It's just a simple test, maybe exist some misleadings on the
config or
>> >> results. But
>> >> we can obviously see the conspicuous improvement for
KeyValueStore.
>> >
>> > The numbers are hard to compare. Since the tests wrote a different
>> > amount of data. This could influence the numbers.
>> >
>> > Do you mean improvements compared to former implementation or to
>> > FileStore?
>> >
>> > Without a retest with the latest fio rbd engine: there is not
so much
>> > difference between KVS and FS atm.
>> >
>> > Btw. Nice to see the rbd engine is useful to others ;-)
>>
>> Thanks for your advise and jobs on fio-rbd. :)
>>
>> The test isn't preciseness and just a simple test to show the
progress
>> of kvstore.
>>
>> >
>> > Regards
>> >
>> > Danny
>>
>>
>>
>> --
>> Best Regards,
>>
>> Wheat
>> _______________________________________________
>> ceph-users mailing list
>> [email protected] <mailto:[email protected]>
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
--
Best Regards,
Wheat
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
