On 06/28/2012 06:10 AM, Stefan Priebe - Profihost AG wrote:
Hello list,
my cluster is now pretty stable i'm just wondering about the sequential
write values.
With rados bench command and 16 threads i get totally different values
than with KVM and rbd block device.
rados -p kvmpool bench 60 write -t 16:
pool size 2: Bandwidth (MB/sec): 1137.294
pool size 3: Bandwidth (MB/sec): 846.983
Inside KVM with fio:
fio --filename=$DISK --direct=1 --rw=write --bs=4M --size=200G
--numjobs=16 --runtime=60 --group_reporting --name=file1:
There are a number of differences between running that in a vm on rbd
and rados bench.
Keep in mind it's running on a filesystem, so requests go through the
guest fs and block layer before getting into librbd. These two layers
can break up those 4M writes, so you end up doing a bunch more small
I/Os which degrades performance a bunch. Running those 16 processes in
does not directly translate to 16 I/Os in flight from the guest kernel,
like rados bench is doing. If you use blktrace on the guest, or just
add --debug-ms 1, you can track the requests the guest is sending by
looking at the lines with 'osd_op\(.*'.
If you don't use direct I/O, and you enable rbd writeback caching,
librbd will be able to merge many of the smaller requests and
you should see much better throughput.
Josh
pool size 2:
write: io=32984MB, bw=562046KB/s, iops=137 , runt= 60094msec
pool size 3:
write: io=29124MB, bw=496024KB/s, iops=121 , runt= 60124msec
Even when i change the pool size to 3 i get with fio 520MB/s.
Any ideas? Is this expected?
Greets
Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
