Hi Alexandre, below is random io test results, almost the same iops.
Rocksdb results
ebs_test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=64
fio-2.1.4
Starting 1 thread
rbd engine: RBD version: 0.1.8
Jobs: 1 (f=1): [w] [100.0% done] [0KB/23094KB/0KB /s] [0/5773/0 iops] [eta
00m:00s]
ebs_test: (groupid=0, jobs=1): err= 0: pid=47154: Tue Mar 4 13:48:22 2014
write: io=3356.2MB, bw=17183KB/s, iops=4295, runt=200004msec
slat (usec): min=19, max=8855, avg=134.33, stdev=259.00
clat (usec): min=73, max=4397.6K, avg=12756.12, stdev=79341.35
lat (msec): min=1, max=4397, avg=12.89, stdev=79.34
clat percentiles (usec):
| 1.00th=[ 1432], 5.00th=[ 1752], 10.00th=[ 2128], 20.00th=[ 3408],
| 30.00th=[ 4768], 40.00th=[ 5856], 50.00th=[ 6880], 60.00th=[ 7904],
| 70.00th=[ 8896], 80.00th=[10048], 90.00th=[11968], 95.00th=[14016],
| 99.00th=[27520], 99.50th=[505856], 99.90th=[1204224], 99.95th=[1433600],
| 99.99th=[2834432]
bw (KB /s): min= 403, max=24392, per=100.00%, avg=17358.47, stdev=7446.69
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=8.36%, 4=15.77%, 10=55.27%, 20=19.17%, 50=0.51%
lat (msec) : 100=0.09%, 250=0.16%, 500=0.14%, 750=0.19%, 1000=0.15%
lat (msec) : 2000=0.16%, >=2000=0.01%
cpu : usr=18.04%, sys=4.15%, ctx=1875119, majf=0, minf=838
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=1.1%, 16=10.9%, 32=65.9%, >=64=22.1%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=97.6%, 8=0.4%, 16=0.4%, 32=0.6%, 64=0.9%, >=64=0.0%
issued : total=r=0/w=859165/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=3356.2MB, aggrb=17182KB/s, minb=17182KB/s, maxb=17182KB/s,
mint=200004msec, maxt=200004msec
Disk stats (read/write):
sda: ios=0/2191, merge=0/2904, ticks=0/936, in_queue=936, util=0.29%
leveldb results:
ebs_test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=64
fio-2.1.4
Starting 1 thread
rbd engine: RBD version: 0.1.8
Jobs: 1 (f=1): [w] [100.0% done] [0KB/9428KB/0KB /s] [0/2357/0 iops] [eta
00m:00s]
ebs_test: (groupid=0, jobs=1): err= 0: pid=112425: Tue Mar 4 14:54:00 2014
write: io=3404.9MB, bw=17431KB/s, iops=4357, runt=200016msec
slat (usec): min=20, max=7698, avg=114.01, stdev=201.06
clat (usec): min=220, max=3278.3K, avg=13340.59, stdev=76874.35
lat (msec): min=1, max=3278, avg=13.45, stdev=76.87
clat percentiles (usec):
| 1.00th=[ 1400], 5.00th=[ 1608], 10.00th=[ 1784], 20.00th=[ 2192],
| 30.00th=[ 2832], 40.00th=[ 3824], 50.00th=[ 5024], 60.00th=[ 6240],
| 70.00th=[ 7456], 80.00th=[ 8768], 90.00th=[10816], 95.00th=[13120],
| 99.00th=[284672], 99.50th=[610304], 99.90th=[1089536], 99.95th=[1286144],
| 99.99th=[1630208]
bw (KB /s): min= 24, max=25548, per=100.00%, avg=17606.69, stdev=6779.23
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=15.63%, 4=25.94%, 10=45.35%, 20=10.98%, 50=0.44%
lat (msec) : 100=0.17%, 250=0.40%, 500=0.42%, 750=0.34%, 1000=0.19%
lat (msec) : 2000=0.12%, >=2000=0.01%
cpu : usr=18.25%, sys=4.14%, ctx=1887389, majf=0, minf=742
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.5%, 16=6.0%, 32=55.9%, >=64=37.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=97.8%, 8=0.7%, 16=0.5%, 32=0.5%, 64=0.5%, >=64=0.0%
issued : total=r=0/w=871635/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=3404.9MB, aggrb=17431KB/s, minb=17431KB/s, maxb=17431KB/s,
mint=200016msec, maxt=200016msec
Disk stats (read/write):
sda: ios=0/2125, merge=0/2796, ticks=0/708, in_queue=708, util=0.23%
-----Original Message-----
From: Alexandre DERUMIER [mailto:[email protected]]
Sent: Tuesday, March 04, 2014 12:49 PM
To: Shu, Xinxin
Cc: [email protected]
Subject: Re: [RFC] add rocksdb support
>>Performance Test
>>Attached file is the performance comparison of rocksdb and leveldb on four
>>nodes with 40 osds, using 'rados bench' as the test tool. The performance
>>results is quite promising.
Thanks for your work, indeed performance seem to be promising !
>>Any comments or suggestions are greatly appreciated.
Could you do test with random io write with last fio (with rbd support) ?
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-February/008182.html
> The fio command: fio -direct=1 -iodepth=64 -thread -rw=randwrite
>> -ioengine=rbd -bs=4k -size=19G -numjobs=1 -runtime=100
>> -group_reporting -name=ebs_test -pool=openstack -rbdname=image
>> -clientname=fio -invalidate=0
----- Mail original -----
De: "Xinxin Shu" <[email protected]>
À: [email protected]
Envoyé: Lundi 3 Mars 2014 03:07:18
Objet: [RFC] add rocksdb support
Hi all,
This patch added rocksdb support for ceph, enabled rocksdb for omap directory.
Rocksdb source code can be get from link. To use use rocksdb, C++11 standard
should be enabled, gcc version >= 4.7 is required to get C++11 support. Rocksdb
can be installed with instructions described in the INSTALL.md file, and
rocksdb header files (include/rocksdb/*) and library (librocksdb.so*) need to
be copied to corresponding directories.
To enable rocksdb, add "--with-librocksdb" option to configure. The rocksdb
branch is here(https://github.com/xinxinsh/ceph/tree/rocksdb).
Performance Test
Attached file is the performance comparison of rocksdb and leveldb on four
nodes with 40 osds, using 'rados bench' as the test tool. The performance
results is quite promising.
Any comments or suggestions are greatly appreciated.
Rados bench BandWidth(MB/s) Average latency Leveldb rocksdb Leveldb rocksdb
write 4 threads 263.762 272.549 0.061 0.059 write 8 threads 449.834 457.811
0.071 0.070 write 16 threads 642.100 638.972 0.100 0.100 write 32 threads
705.897 717.598 0.181 0.178 write 64 threads 705.011 717.204 0.370 0.362 read 4
threads 873.588 841.704 0.073 0.076 read 8 threads 816.699 818.451 0.078 0.078
read 16 threads 808.810 798.053 0.079 0.080 read 32 threads 798.394 802.796
0.080 0.080 read 64 threads 792.848 790.593 0.081 0.081
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
