Hi again,
when setup file-fsync-freq=1 (fsync for each time writing) and
file-fsync-freq=0 (never fsync by sysbench), the result gets huge
difference.
(one is 382.94Kb/sec, another is 25.921Mb/sec).
How do you think of it? thanks.
file-fsync-freq=1,
# sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw
--init-rng=on --max-time=300 --max-requests=0 --file-fsync-freq=1 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.
Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 1 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.
Operations performed: 4309 Read, 2873 Write, 367707 Other = 374889 Total
Read 67.328Mb Written 44.891Mb Total transferred 112.22Mb (382.94Kb/sec)
23.93 Requests/sec executed
Test execution summary:
total time: 300.0782s
total number of events: 7182
total time taken by event execution: 2.3207
per-request statistics:
min: 0.01ms
avg: 0.32ms
max: 80.17ms
approx. 95 percentile: 1.48ms
Threads fairness:
events (avg/stddev): 7182.0000/0.00
execution time (avg/stddev): 2.3207/0.00
file-fsync-freq=0,
# sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw
--init-rng=on --max-time=300 --max-requests=0 --file-fsync-freq=0 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.
Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.
Operations performed: 298613 Read, 199075 Write, 0 Other = 497688 Total
Read 4.5565Gb Written 3.0376Gb Total transferred 7.5941Gb (25.921Mb/sec)
1658.93 Requests/sec executed
Test execution summary:
total time: 300.0049s
total number of events: 497688
total time taken by event execution: 299.7026
per-request statistics:
min: 0.00ms
avg: 0.60ms
max: 2211.13ms
approx. 95 percentile: 1.21ms
Threads fairness:
events (avg/stddev): 497688.0000/0.00
execution time (avg/stddev): 299.7026/0.00
2016-05-25 15:01 GMT+08:00 Ken Peng <[email protected]>:
> Hello,
>
> We have a cluster with 20+ hosts and 200+ OSDs, each 4T SATA disk for an
> OSD, no SSD cache.
> OS is Ubuntu 16.04 LTS, ceph version 10.2.0
> Both data network and cluster network are 10Gbps.
> We run ceph as block storage service only (rbd client within VM).
>
> For testing within a VM with sysbench tool, we see that the seqwrite has a
> relatively good performance, it can reach 170.37Mb/sec, but random
> read/write always gets bad result, it can be only 474.63Kb/sec (shown as
> below).
>
> Can you help give the idea why the random IO is so worse? Thanks.
>
> This is what sysbench outputs,
>
> # sysbench --test=fileio --file-total-size=5G prepare
> sysbench 0.4.12: multi-threaded system evaluation benchmark
>
> 128 files, 40960Kb each, 5120Mb total
> Creating files for the test...
>
>
> # sysbench --test=fileio --file-total-size=5G --file-test-mode=seqwr
> --init-rng=on --max-time=300 --max-requests=0 run
> sysbench 0.4.12: multi-threaded system evaluation benchmark
>
> Running the test with following options:
> Number of threads: 1
> Initializing random number generator from timer.
>
>
> Extra file open flags: 0
> 128 files, 40Mb each
> 5Gb total file size
> Block size 16Kb
> Periodic FSYNC enabled, calling fsync() each 100 requests.
> Calling fsync() at the end of test, Enabled.
> Using synchronous I/O mode
> Doing sequential write (creation) test
> Threads started!
> Done.
>
> Operations performed: 0 Read, 327680 Write, 128 Other = 327808 Total
> Read 0b Written 5Gb Total transferred 5Gb (170.37Mb/sec)
> 10903.42 Requests/sec executed
>
> Test execution summary:
> total time: 30.0530s
> total number of events: 327680
> total time taken by event execution: 28.5936
> per-request statistics:
> min: 0.01ms
> avg: 0.09ms
> max: 192.84ms
> approx. 95 percentile: 0.03ms
>
> Threads fairness:
> events (avg/stddev): 327680.0000/0.00
> execution time (avg/stddev): 28.5936/0.00
>
>
>
> # sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw
> --init-rng=on --max-time=300 --max-requests=0 run
> sysbench 0.4.12: multi-threaded system evaluation benchmark
>
> Running the test with following options:
> Number of threads: 1
> Initializing random number generator from timer.
>
>
> Extra file open flags: 0
> 128 files, 40Mb each
> 5Gb total file size
> Block size 16Kb
> Number of random requests for random IO: 0
> Read/Write ratio for combined random IO test: 1.50
> Periodic FSYNC enabled, calling fsync() each 100 requests.
> Calling fsync() at the end of test, Enabled.
> Using synchronous I/O mode
> Doing random r/w test
> Threads started!
>
> Time limit exceeded, exiting...
> Done.
>
> Operations performed: 5340 Read, 3560 Write, 11269 Other = 20169 Total
> Read 83.438Mb Written 55.625Mb Total transferred 139.06Mb (474.63Kb/sec)
> 29.66 Requests/sec executed
>
> Test execution summary:
> total time: 300.0216s
> total number of events: 8900
> total time taken by event execution: 6.4774
> per-request statistics:
> min: 0.01ms
> avg: 0.73ms
> max: 90.18ms
> approx. 95 percentile: 1.60ms
>
> Threads fairness:
> events (avg/stddev): 8900.0000/0.00
> execution time (avg/stddev): 6.4774/0.00
>
>
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
