I am loading a Ramdisk driver in various block driver modes (RQ, BIO,
and Multi-Queue[MQ]) to compare IOPS.
When I load the driver in BIO mode and run fio, the Disk Stats shows a
utilization of 0.0%, which makes me wonder if the disk is being
accessed. On the other hand, when I load the driver in RQ or MQ mode,
the Disk Stats show a utilization of 100% (or near 100%). The IOPS
for BIO mode are greater than the IOPS for MQ mode, which is another
red flag that the BIO mode IOPS may not be accurate.
I am able to successfully perform read/write/verification tests
outside of fio in all block driver modes.
My question is how can I get an accurate IOPS performance measurement
for the driver in BIO mode? Is there a bug in fio, or am I missing
some parameter?
Also, it seems that IOPS should be higher in general for MQ mode. Are
there any performance tuning suggestions for MQ?
=======================================================
Driver loaded w/ blk_mode=BIO, ioremap=nocache ... => 0% Disk Utilization
fio --ioengine=libaio --group_reporting --bs=512 --numjobs=24
--cpus_allowed=1-24 --iodepth=64 --rw=read --time_based=1 --runtime=30
--name=test1 --thread --filename=/dev/mydsk0 --direct=1 --buffered=0
Output:
test1: (g=0): rw=read, bs=512-512/512-512/512-512, ioengine=libaio, iodepth=64
...
fio-2.1.10
Starting 24 threads
test1: (groupid=0, jobs=24): err= 0: pid=1987: Wed Aug 6 17:12:33 2014
read : io=21390MB, bw=730082KB/s, iops=1460.2K, runt= 30001msec
slat (usec): min=6, max=24026, avg=15.14, stdev=92.35
clat (usec): min=1, max=39542, avg=1035.74, stdev=804.55
lat (usec): min=8, max=39689, avg=1051.05, stdev=811.05
clat percentiles (usec):
| 1.00th=[ 652], 5.00th=[ 716], 10.00th=[ 764], 20.00th=[ 844],
| 30.00th=[ 884], 40.00th=[ 924], 50.00th=[ 956], 60.00th=[ 1004],
| 70.00th=[ 1064], 80.00th=[ 1128], 90.00th=[ 1256], 95.00th=[ 1368],
| 99.00th=[ 1656], 99.50th=[ 1880], 99.90th=[13888], 99.95th=[14016],
| 99.99th=[20608]
bw (KB /s): min=13309, max=37536, per=4.16%, avg=30407.77, stdev=4089.20
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=8.61%, 1000=50.70%
lat (msec) : 2=40.26%, 4=0.07%, 10=0.02%, 20=0.33%, 50=0.01%
cpu : usr=6.97%, sys=88.38%, ctx=10389, majf=0, minf=3944
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=43806400/w=0/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: io=21390MB, aggrb=730082KB/s, minb=730082KB/s,
maxb=730082KB/s, mint=30001msec, maxt=30001msec
Disk stats (read/write):
mydsk0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
=================================================================
Driver loaded w/ blk_mode=MQ, ioremap=nocache ... => 100% Disk Utilization
fio --ioengine=libaio --group_reporting --bs=512 --numjobs=24
--cpus_allowed=1-24 --iodepth=64 --rw=read --time_based=1 --runtime=30
--name=test1 --thread --filename=/dev/mydsk0 --direct=1 --buffered=0
Output:
test1: (g=0): rw=read, bs=512-512/512-512/512-512, ioengine=libaio, iodepth=64
...
fio-2.1.10
Starting 24 threads
test1: (groupid=0, jobs=24): err= 0: pid=2081: Wed Aug 6 17:13:39 2014
read : io=18317MB, bw=625210KB/s, iops=1250.5K, runt= 30001msec
slat (usec): min=8, max=41044, avg=17.75, stdev=104.21
clat (usec): min=1, max=59034, avg=1209.56, stdev=874.32
lat (usec): min=9, max=59053, avg=1227.45, stdev=881.06
clat percentiles (usec):
| 1.00th=[ 836], 5.00th=[ 924], 10.00th=[ 972], 20.00th=[ 1032],
| 30.00th=[ 1080], 40.00th=[ 1112], 50.00th=[ 1144], 60.00th=[ 1176],
| 70.00th=[ 1208], 80.00th=[ 1256], 90.00th=[ 1336], 95.00th=[ 1416],
| 99.00th=[ 1672], 99.50th=[ 1944], 99.90th=[14016], 99.95th=[14144],
| 99.99th=[19072]
bw (KB /s): min=11778, max=33155, per=4.17%, avg=26050.75, stdev=3243.73
lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=0.01%, 750=0.07%, 1000=14.23%
lat (msec) : 2=85.22%, 4=0.02%, 10=0.03%, 20=0.42%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=6.68%, sys=88.39%, ctx=7847, majf=0, minf=3896
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=37513871/w=0/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: io=18317MB, aggrb=625210KB/s, minb=625210KB/s,
maxb=625210KB/s, mint=30001msec, maxt=30001msec
Disk stats (read/write):
mydsk0: ios=37387530/0, merge=0/0, ticks=477219/0, in_queue=516648,
util=100.00%
CONFIDENTIALITY
This e-mail message and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged and/or
confidential information. If you are not the intended recipient of this e-mail
message, you are hereby notified that any dissemination, distribution or
copying of this e-mail message, and any attachments thereto, is strictly
prohibited. If you have received this e-mail message in error, please
immediately notify the sender and permanently delete the original and any
copies of this email and any prints thereof.
ABSENT AN EXPRESS STATEMENT TO THE CONTRARY HEREINABOVE, THIS E-MAIL IS NOT
INTENDED AS A SUBSTITUTE FOR A WRITING. Notwithstanding the Uniform Electronic
Transactions Act or the applicability of any other law of similar substance and
effect, absent an express statement to the contrary hereinabove, this e-mail
message its contents, and any attachments hereto are not intended to represent
an offer or acceptance to enter into a contract and are not otherwise intended
to bind the sender, Sanmina Corporation (or any of its subsidiaries), or any
other person or entity.
DISCLAIMER: The individual sending this e-mail is not an employee of Sanmina
Corporation or its subsidiaries, but serves as an independent contractor.
Accordingly, the individual sending this e-mail has no authority to legally
bind Sanmina or its subsidiaries. Only employees possessing the requisite
level of authority under Sanmina policies have the ability to bind Sanmina. If
you have a question regarding an individual's ability to bind Sanmina, please
contact Sanmina's Legal Department at (408) 964-3156.
--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
