It’s working as expected. I see improvement with passthrough enabled
(see below). Nate also confirmed the SMMU setup via JTAG.
Passthrough=0 -
root@null-8cfdf006971f:/home/ubuntu# fio --name=global --readonly
--group_reporting --direct=1 --ioengine=libaio --rw=read --eta-newline=1s
--size=1T --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s
--name=nvme_0 --filename=/dev/nvme0n1
nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio,
iodepth=32
fio-2.2.10
Starting 1 process
Jobs: 1 (f=1): [R(1)] [36.4% done] [2540MB/0KB/0KB /s] [5080/0/0 iops] [eta
00m:07s]
Jobs: 1 (f=1): [R(1)] [45.5% done] [2539MB/0KB/0KB /s] [5077/0/0 iops] [eta
00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [2540MB/0KB/0KB /s] [5079/0/0 iops] [eta
00m:05s]
Jobs: 1 (f=1): [R(1)] [63.6% done] [2540MB/0KB/0KB /s] [5079/0/0 iops] [eta
00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [2539MB/0KB/0KB /s] [5078/0/0 iops] [eta
00m:03s]
Jobs: 1 (f=1): [R(1)] [81.8% done] [2539MB/0KB/0KB /s] [5078/0/0 iops] [eta
00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [2540MB/0KB/0KB /s] [5080/0/0 iops] [eta
00m:01s]
Jobs: 1 (f=1): [R(1)] [100.0% done] [2539MB/0KB/0KB /s] [5078/0/0 iops] [eta
00m:00s]
nvme_0: (groupid=0, jobs=1): err= 0: pid=36481: Mon May 15 17:10:26 2017
read : io=25377MB, bw=2537.5MB/s, iops=5074, runt= 10001msec
slat (usec): min=32, max=440, avg=191.74, stdev=157.14
clat (usec): min=153, max=7641, avg=6106.54, stdev=107.40
lat (usec): min=442, max=7904, avg=6298.45, stdev=174.49
clat percentiles (usec):
| 1.00th=[ 5920], 5.00th=[ 5984], 10.00th=[ 6048], 20.00th=[ 6048],
| 30.00th=[ 6112], 40.00th=[ 6112], 50.00th=[ 6112], 60.00th=[ 6112],
| 70.00th=[ 6112], 80.00th=[ 6112], 90.00th=[ 6176], 95.00th=[ 6176],
| 99.00th=[ 6304], 99.50th=[ 6368], 99.90th=[ 6432], 99.95th=[ 6432],
| 99.99th=[ 7008]
bw (MB /s): min= 2498, max= 2540, per=99.98%, avg=2536.84, stdev= 9.44
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=99.96%
cpu : usr=3.84%, sys=96.12%, ctx=67, majf=0, minf=528
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=50754/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: io=25377MB, aggrb=2537.5MB/s, minb=2537.5MB/s, maxb=2537.5MB/s,
mint=10001msec, maxt=10001msec
Disk stats (read/write):
nvme0n1: ios=200550/0, merge=0/0, ticks=23848/0, in_queue=23828, util=91.79%
passthrough=1 –
root@null-8cfdf006971f:/home/ubuntu#
ing --direct=1 --ioengine=libaio --rw=read --eta-newline=1s --size=1T
--blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s --name=nvme_0
--filename=/dev/nvme0n1
nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio,
iodepth=32
fio-2.2.10
Starting 1 process
Jobs: 1 (f=1): [R(1)] [36.4% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta
00m:07s]
Jobs: 1 (f=1): [R(1)] [45.5% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta
00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [3239MB/0KB/0KB /s] [6477/0/0 iops] [eta
00m:05s]
Jobs: 1 (f=1): [R(1)] [63.6% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta
00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta
00m:03s]
Jobs: 1 (f=1): [R(1)] [81.8% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta
00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [3239MB/0KB/0KB /s] [6477/0/0 iops] [eta
00m:01s]
Jobs: 1 (f=1): [R(1)] [100.0% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta
00m:00s]
nvme_0: (groupid=0, jobs=1): err= 0: pid=1600: Mon May 15 17:13:46 2017
read : io=32384MB, bw=3236.8MB/s, iops=6473, runt= 10005msec
slat (usec): min=13, max=210, avg=16.19, stdev= 6.40
clat (usec): min=1505, max=9532, avg=4925.01, stdev=87.39
lat (usec): min=1616, max=9566, avg=4941.38, stdev=86.47
clat percentiles (usec):
| 1.00th=[ 4896], 5.00th=[ 4896], 10.00th=[ 4896], 20.00th=[ 4896],
| 30.00th=[ 4896], 40.00th=[ 4960], 50.00th=[ 4960], 60.00th=[ 4960],
| 70.00th=[ 4960], 80.00th=[ 4960], 90.00th=[ 4960], 95.00th=[ 4960],
| 99.00th=[ 4960], 99.50th=[ 4960], 99.90th=[ 4960], 99.95th=[ 4960],
| 99.99th=[ 8640]
bw (MB /s): min= 3209, max= 3237, per=99.96%, avg=3235.57, stdev= 6.18
lat (msec) : 2=0.03%, 4=0.03%, 10=99.94%
cpu : usr=1.80%, sys=16.99%, ctx=64755, majf=0, minf=528
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=64767/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: io=32384MB, aggrb=3236.8MB/s, minb=3236.8MB/s, maxb=3236.8MB/s,
mint=10005msec, maxt=10005msec
Disk stats (read/write):
nvme0n1: ios=255788/0, merge=0/0, ticks=1246388/0, in_queue=1246788,
util=99.05%
Jeffrey Hugo
Senior Engineer
Qualcomm Datacenter Technologies, Inc.
1-303-247-5002
From: Manoj Iyer [mailto:[email protected]]
Sent: Thursday, May 11, 2017 9:23 PM
To: Jeff Hugo <[email protected]>; [email protected]
Cc: Nate Watterson <[email protected]>; Timur Tabi
<[email protected]>
Subject: [Bug #1688158] Please help test kernel for iommu.passthrough=0/1
Jeff,
I have back-ported patches to address "Support SMMU passthrough using
the default domain". This adds a new kernel command line option
iommu.passthrough=0/1 to Configure DMA to bypass the IOMMU for ARM64.
http://bugs.launchpad.net/bugs/1688158
0 - Use IOMMU translation for DMA.
1 - Bypass the IOMMU for DMA.
The test case recommended by Nate using 'fio' is:
$ sudo apt update ; sudo apt install fio
$ sudo fio --name=global --readonly --group_reporting --direct=1
--ioengine=libaio --rw=read --eta-newline=1s --size=1T --blocksize=512k
--iodepth=32 --numjobs=1 --runtime=10s --name=nvme_0
--filename=/dev/nvme0n1
You can find the kernel and the instructions to install the kernel in the
Centriq test PPA:
https://launchpad.net/~centriq-team/+archive/ubuntu/test/
Nate and I have gone back and forth on testing these patches and finally
came down to having this test run on an NVME device. Unfortunately I
cant seem to find any nvme HW at this time, we would need to order
these drive and could take time.
Could you please test the kernel using the test case mentioned above and
reply back to this email with test case out put for both cases.
iommu.passthrough=0 and iommu.passthrough=1. We should see better
performance (iops may be) with IOMMU bypassed for DMA (ie
iommu.passthrough=1).
The bug is cced on this email, so if you replay with a summary output in
both cases it will be automatically recorded in the bug. Once I get your
test results I will be able to make a case for SRUing these patches.
Thanks a ton
Manoj Iyer
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1688158
Title:
[SRU][Zesty] Support SMMU passthrough using the default domain
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
