Thanks Jeff/Nate I will sru these patches shortly. On Mon, May 15, 2017 at 2:55 PM, Jeff Hugo <[email protected]> wrote: > It’s working as expected. I see improvement with passthrough > enabled (see below). Nate also confirmed the SMMU setup via JTAG. > > Passthrough=0 - > > root@null-8cfdf006971f:/home/ubuntu# fio --name=global --readonly > --group_reporting --direct=1 --ioengine=libaio --rw=read > --eta-newline=1s --size=1T --blocksize=512k --iodepth=32 --numjobs=1 > --runtime=10s --name=nvme_0 --filename=/dev/nvme0n1 > nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, > ioengine=libaio, iodepth=32 > fio-2.2.10 > Starting 1 process > Jobs: 1 (f=1): [R(1)] [36.4% done] [2540MB/0KB/0KB /s] [5080/0/0 > iops] [eta 00m:07s] > Jobs: 1 (f=1): [R(1)] [45.5% done] [2539MB/0KB/0KB /s] [5077/0/0 > iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [2540MB/0KB/0KB /s] > [5079/0/0 iops] [eta 00m:05s] > Jobs: 1 (f=1): [R(1)] [63.6% done] [2540MB/0KB/0KB /s] [5079/0/0 > iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [2539MB/0KB/0KB /s] > [5078/0/0 iops] [eta 00m:03s] > Jobs: 1 (f=1): [R(1)] [81.8% done] [2539MB/0KB/0KB /s] [5078/0/0 > iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [2540MB/0KB/0KB /s] > [5080/0/0 iops] [eta 00m:01s] > Jobs: 1 (f=1): [R(1)] [100.0% done] [2539MB/0KB/0KB /s] [5078/0/0 > iops] [eta 00m:00s] > nvme_0: (groupid=0, jobs=1): err= 0: pid=36481: Mon May 15 17:10:26 > 2017 > read : io=25377MB, bw=2537.5MB/s, iops=5074, runt= 10001msec > slat (usec): min=32, max=440, avg=191.74, stdev=157.14 > clat (usec): min=153, max=7641, avg=6106.54, stdev=107.40 > lat (usec): min=442, max=7904, avg=6298.45, stdev=174.49 > clat percentiles (usec): > | 1.00th=[ 5920], 5.00th=[ 5984], 10.00th=[ 6048], 20.00th=[ > 6048], > | 30.00th=[ 6112], 40.00th=[ 6112], 50.00th=[ 6112], 60.00th=[ > 6112], > | 70.00th=[ 6112], 80.00th=[ 6112], 90.00th=[ 6176], 95.00th=[ > 6176], > | 99.00th=[ 6304], 99.50th=[ 6368], 99.90th=[ 6432], 99.95th=[ > 6432], > | 99.99th=[ 7008] > bw (MB /s): min= 2498, max= 2540, per=99.98%, avg=2536.84, > stdev= 9.44 > lat (usec) : 250=0.01%, 500=0.01%, 750=0.01% > lat (msec) : 2=0.01%, 4=0.02%, 10=99.96% > cpu : usr=3.84%, sys=96.12%, ctx=67, majf=0, minf=528 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, > >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, > >=64=0.0% > issued : total=r=50754/w=0/d=0, short=r=0/w=0/d=0, > drop=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=32 > > Run status group 0 (all jobs): > READ: io=25377MB, aggrb=2537.5MB/s, minb=2537.5MB/s, > maxb=2537.5MB/s, mint=10001msec, maxt=10001msec > > Disk stats (read/write): > nvme0n1: ios=200550/0, merge=0/0, ticks=23848/0, in_queue=23828, > util=91.79% > > > > passthrough=1 – > > root@null-8cfdf006971f:/home/ubuntu# > ing --direct=1 --ioengine=libaio --rw=read --eta-newline=1s > --size=1T --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s > --name=nvme_0 --filename=/dev/nvme0n1 > nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, > ioengine=libaio, iodepth=32 > fio-2.2.10 > Starting 1 process > Jobs: 1 (f=1): [R(1)] [36.4% done] [3238MB/0KB/0KB /s] [6476/0/0 > iops] [eta 00m:07s] > Jobs: 1 (f=1): [R(1)] [45.5% done] [3238MB/0KB/0KB /s] [6476/0/0 > iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [3239MB/0KB/0KB /s] > [6477/0/0 iops] [eta 00m:05s] > Jobs: 1 (f=1): [R(1)] [63.6% done] [3238MB/0KB/0KB /s] [6476/0/0 > iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [3238MB/0KB/0KB /s] > [6476/0/0 iops] [eta 00m:03s] > Jobs: 1 (f=1): [R(1)] [81.8% done] [3238MB/0KB/0KB /s] [6476/0/0 > iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [3239MB/0KB/0KB /s] > [6477/0/0 iops] [eta 00m:01s] > Jobs: 1 (f=1): [R(1)] [100.0% done] [3238MB/0KB/0KB /s] [6476/0/0 > iops] [eta 00m:00s] > nvme_0: (groupid=0, jobs=1): err= 0: pid=1600: Mon May 15 17:13:46 > 2017 > read : io=32384MB, bw=3236.8MB/s, iops=6473, runt= 10005msec > slat (usec): min=13, max=210, avg=16.19, stdev= 6.40 > clat (usec): min=1505, max=9532, avg=4925.01, stdev=87.39 > lat (usec): min=1616, max=9566, avg=4941.38, stdev=86.47 > clat percentiles (usec): > | 1.00th=[ 4896], 5.00th=[ 4896], 10.00th=[ 4896], 20.00th=[ > 4896], > | 30.00th=[ 4896], 40.00th=[ 4960], 50.00th=[ 4960], 60.00th=[ > 4960], > | 70.00th=[ 4960], 80.00th=[ 4960], 90.00th=[ 4960], 95.00th=[ > 4960], > | 99.00th=[ 4960], 99.50th=[ 4960], 99.90th=[ 4960], 99.95th=[ > 4960], > | 99.99th=[ 8640] > bw (MB /s): min= 3209, max= 3237, per=99.96%, avg=3235.57, > stdev= 6.18 > lat (msec) : 2=0.03%, 4=0.03%, 10=99.94% > cpu : usr=1.80%, sys=16.99%, ctx=64755, majf=0, minf=528 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, > >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, > >=64=0.0% > issued : total=r=64767/w=0/d=0, short=r=0/w=0/d=0, > drop=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=32 > > Run status group 0 (all jobs): > READ: io=32384MB, aggrb=3236.8MB/s, minb=3236.8MB/s, > maxb=3236.8MB/s, mint=10005msec, maxt=10005msec > > Disk stats (read/write): > nvme0n1: ios=255788/0, merge=0/0, ticks=1246388/0, > in_queue=1246788, util=99.05% > > Jeffrey Hugo > Senior Engineer > Qualcomm Datacenter Technologies, Inc. > 1-303-247-5002 > > From: Manoj Iyer [mailto:[email protected]] > Sent: Thursday, May 11, 2017 9:23 PM > To: Jeff Hugo <[email protected]>; [email protected] > Cc: Nate Watterson <[email protected]>; Timur Tabi > <[email protected]> > Subject: [Bug #1688158] Please help test kernel for > iommu.passthrough=0/1 > > Jeff, > > I have back-ported patches to address "Support SMMU passthrough using > the default domain". This adds a new kernel command line option > iommu.passthrough=0/1 to Configure DMA to bypass the IOMMU for ARM64. > http://bugs.launchpad.net/bugs/1688158 > > 0 - Use IOMMU translation for DMA. > 1 - Bypass the IOMMU for DMA. > > The test case recommended by Nate using 'fio' is: > > $ sudo apt update ; sudo apt install fio > > $ sudo fio --name=global --readonly --group_reporting --direct=1 > --ioengine=libaio --rw=read --eta-newline=1s --size=1T > --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s > --name=nvme_0 --filename=/dev/nvme0n1 > > You can find the kernel and the instructions to install the kernel in > the Centriq test PPA: > https://launchpad.net/~centriq-team/+archive/ubuntu/test/ > > Nate and I have gone back and forth on testing these patches and > finally came down to having this test run on an NVME device. > Unfortunately I cant seem to find any nvme HW at this time, we would > need to order these drive and could take time. > > Could you please test the kernel using the test case mentioned above > and reply back to this email with test case out put for both cases. > iommu.passthrough=0 and iommu.passthrough=1. We should see better > performance (iops may be) with IOMMU bypassed for DMA (ie > iommu.passthrough=1). > > The bug is cced on this email, so if you replay with a summary output > in both cases it will be automatically recorded in the bug. Once I > get your test results I will be able to make a case for SRUing these > patches. > > Thanks a ton > Manoj Iyer
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1688158 Title: [SRU][Zesty] Support SMMU passthrough using the default domain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
