[Kernel-packages] [Bug 1688158] RE: [Bug #1688158] Please help test kernel for iommu.passthrough=0/1

2017-05-15 Thread Manoj Iyer 
Thanks Jeff/Nate I will sru these patches shortly.

On Mon, May 15, 2017 at 2:55 PM, Jeff Hugo  
wrote:
> It’s working as expected.  I see improvement with passthrough 
> enabled (see below).  Nate also confirmed the SMMU setup via JTAG.
> 
> Passthrough=0 -
> 
> root@null-8cfdf006971f:/home/ubuntu# fio --name=global --readonly 
> --group_reporting --direct=1 --ioengine=libaio --rw=read 
> --eta-newline=1s  --size=1T --blocksize=512k --iodepth=32 --numjobs=1 
>  --runtime=10s --name=nvme_0 --filename=/dev/nvme0n1
> nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, 
> ioengine=libaio, iodepth=32
> fio-2.2.10
> Starting 1 process
> Jobs: 1 (f=1): [R(1)] [36.4% done] [2540MB/0KB/0KB /s] [5080/0/0 
> iops] [eta 00m:07s]
> Jobs: 1 (f=1): [R(1)] [45.5% done] [2539MB/0KB/0KB /s] [5077/0/0 
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [2540MB/0KB/0KB /s] 
> [5079/0/0 iops] [eta 00m:05s]
> Jobs: 1 (f=1): [R(1)] [63.6% done] [2540MB/0KB/0KB /s] [5079/0/0 
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [2539MB/0KB/0KB /s] 
> [5078/0/0 iops] [eta 00m:03s]
> Jobs: 1 (f=1): [R(1)] [81.8% done] [2539MB/0KB/0KB /s] [5078/0/0 
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [2540MB/0KB/0KB /s] 
> [5080/0/0 iops] [eta 00m:01s]
> Jobs: 1 (f=1): [R(1)] [100.0% done] [2539MB/0KB/0KB /s] [5078/0/0 
> iops] [eta 00m:00s]
> nvme_0: (groupid=0, jobs=1): err= 0: pid=36481: Mon May 15 17:10:26 
> 2017
>   read : io=25377MB, bw=2537.5MB/s, iops=5074, runt= 10001msec
> slat (usec): min=32, max=440, avg=191.74, stdev=157.14
> clat (usec): min=153, max=7641, avg=6106.54, stdev=107.40
>  lat (usec): min=442, max=7904, avg=6298.45, stdev=174.49
> clat percentiles (usec):
>  |  1.00th=[ 5920],  5.00th=[ 5984], 10.00th=[ 6048], 20.00th=[ 
> 6048],
>  | 30.00th=[ 6112], 40.00th=[ 6112], 50.00th=[ 6112], 60.00th=[ 
> 6112],
>  | 70.00th=[ 6112], 80.00th=[ 6112], 90.00th=[ 6176], 95.00th=[ 
> 6176],
>  | 99.00th=[ 6304], 99.50th=[ 6368], 99.90th=[ 6432], 99.95th=[ 
> 6432],
>  | 99.99th=[ 7008]
> bw (MB  /s): min= 2498, max= 2540, per=99.98%, avg=2536.84, 
> stdev= 9.44
> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%
> lat (msec) : 2=0.01%, 4=0.02%, 10=99.96%
>   cpu  : usr=3.84%, sys=96.12%, ctx=67, majf=0, minf=528
>   IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, 
> >=64=0.0%
>  submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
> >=64=0.0%
>  complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, 
> >=64=0.0%
>  issued: total=r=50754/w=0/d=0, short=r=0/w=0/d=0, 
> drop=r=0/w=0/d=0
>  latency   : target=0, window=0, percentile=100.00%, depth=32
> 
> Run status group 0 (all jobs):
>READ: io=25377MB, aggrb=2537.5MB/s, minb=2537.5MB/s, 
> maxb=2537.5MB/s, mint=10001msec, maxt=10001msec
> 
> Disk stats (read/write):
>   nvme0n1: ios=200550/0, merge=0/0, ticks=23848/0, in_queue=23828, 
> util=91.79%
> 
> 
> 
> passthrough=1 –
> 
> root@null-8cfdf006971f:/home/ubuntu#
> ing --direct=1 --ioengine=libaio --rw=read --eta-newline=1s  
> --size=1T --blocksize=512k --iodepth=32 --numjobs=1  --runtime=10s 
> --name=nvme_0 --filename=/dev/nvme0n1
> nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, 
> ioengine=libaio, iodepth=32
> fio-2.2.10
> Starting 1 process
> Jobs: 1 (f=1): [R(1)] [36.4% done] [3238MB/0KB/0KB /s] [6476/0/0 
> iops] [eta 00m:07s]
> Jobs: 1 (f=1): [R(1)] [45.5% done] [3238MB/0KB/0KB /s] [6476/0/0 
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [3239MB/0KB/0KB /s] 
> [6477/0/0 iops] [eta 00m:05s]
> Jobs: 1 (f=1): [R(1)] [63.6% done] [3238MB/0KB/0KB /s] [6476/0/0 
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [3238MB/0KB/0KB /s] 
> [6476/0/0 iops] [eta 00m:03s]
> Jobs: 1 (f=1): [R(1)] [81.8% done] [3238MB/0KB/0KB /s] [6476/0/0 
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [3239MB/0KB/0KB /s] 
> [6477/0/0 iops] [eta 00m:01s]
> Jobs: 1 (f=1): [R(1)] [100.0% done] [3238MB/0KB/0KB /s] [6476/0/0 
> iops] [eta 00m:00s]
> nvme_0: (groupid=0, jobs=1): err= 0: pid=1600: Mon May 15 17:13:46 
> 2017
>   read : io=32384MB, bw=3236.8MB/s, iops=6473, runt= 10005msec
> slat (usec): min=13, max=210, avg=16.19, stdev= 6.40
> clat (usec): min=1505, max=9532, avg=4925.01, stdev=87.39
>  lat (usec): min=1616, max=9566, avg=4941.38, stdev=86.47
> clat percentiles (usec):
>  |  1.00th=[ 4896],  5.00th=[ 4896], 10.00th=[ 4896], 20.00th=[ 
> 4896],
>  | 30.00th=[ 4896], 40.00th=[ 4960], 50.00th=[ 4960], 60.00th=[ 
> 4960],
>  | 70.00th=[ 4960], 80.00th=[ 4960], 90.00th=[ 4960], 95.00th=[ 
> 4960],
>  | 99.00th=[ 4960], 99.50th=[ 4960], 99.90th=[ 4960], 99.95th=[ 
> 4960],
>  | 99.99th=[ 8640]
> bw (MB  /s): min= 3209, max= 3237, per=99.96%, avg=3235.57, 
> stdev= 6.18
> lat (msec) : 2=0.03%, 4=0.03%, 10=99.94%
>   cpu  : usr=1.80%, sys=16.99%, ctx=64755, majf=0, minf=528
>   IO depths: 1=0.1%, 2=0.1%, 4=0.1%,

[Kernel-packages] [Bug 1688158] RE: [Bug #1688158] Please help test kernel for iommu.passthrough=0/1

2017-05-15 Thread Jeffrey Hugo 
It’s working as expected.  I see improvement with passthrough enabled
(see below).  Nate also confirmed the SMMU setup via JTAG.

Passthrough=0 -

root@null-8cfdf006971f:/home/ubuntu# fio --name=global --readonly 
--group_reporting --direct=1 --ioengine=libaio --rw=read --eta-newline=1s  
--size=1T --blocksize=512k --iodepth=32 --numjobs=1  --runtime=10s 
--name=nvme_0 --filename=/dev/nvme0n1
nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio, 
iodepth=32
fio-2.2.10
Starting 1 process
Jobs: 1 (f=1): [R(1)] [36.4% done] [2540MB/0KB/0KB /s] [5080/0/0 iops] [eta 
00m:07s]
Jobs: 1 (f=1): [R(1)] [45.5% done] [2539MB/0KB/0KB /s] [5077/0/0 iops] [eta 
00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [2540MB/0KB/0KB /s] [5079/0/0 iops] [eta 
00m:05s]
Jobs: 1 (f=1): [R(1)] [63.6% done] [2540MB/0KB/0KB /s] [5079/0/0 iops] [eta 
00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [2539MB/0KB/0KB /s] [5078/0/0 iops] [eta 
00m:03s]
Jobs: 1 (f=1): [R(1)] [81.8% done] [2539MB/0KB/0KB /s] [5078/0/0 iops] [eta 
00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [2540MB/0KB/0KB /s] [5080/0/0 iops] [eta 
00m:01s]
Jobs: 1 (f=1): [R(1)] [100.0% done] [2539MB/0KB/0KB /s] [5078/0/0 iops] [eta 
00m:00s]
nvme_0: (groupid=0, jobs=1): err= 0: pid=36481: Mon May 15 17:10:26 2017
  read : io=25377MB, bw=2537.5MB/s, iops=5074, runt= 10001msec
slat (usec): min=32, max=440, avg=191.74, stdev=157.14
clat (usec): min=153, max=7641, avg=6106.54, stdev=107.40
 lat (usec): min=442, max=7904, avg=6298.45, stdev=174.49
clat percentiles (usec):
 |  1.00th=[ 5920],  5.00th=[ 5984], 10.00th=[ 6048], 20.00th=[ 6048],
 | 30.00th=[ 6112], 40.00th=[ 6112], 50.00th=[ 6112], 60.00th=[ 6112],
 | 70.00th=[ 6112], 80.00th=[ 6112], 90.00th=[ 6176], 95.00th=[ 6176],
 | 99.00th=[ 6304], 99.50th=[ 6368], 99.90th=[ 6432], 99.95th=[ 6432],
 | 99.99th=[ 7008]
bw (MB  /s): min= 2498, max= 2540, per=99.98%, avg=2536.84, stdev= 9.44
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=99.96%
  cpu  : usr=3.84%, sys=96.12%, ctx=67, majf=0, minf=528
  IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
 submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
 issued: total=r=50754/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
 latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=25377MB, aggrb=2537.5MB/s, minb=2537.5MB/s, maxb=2537.5MB/s, 
mint=10001msec, maxt=10001msec

Disk stats (read/write):
  nvme0n1: ios=200550/0, merge=0/0, ticks=23848/0, in_queue=23828, util=91.79%


passthrough=1 –

root@null-8cfdf006971f:/home/ubuntu#
ing --direct=1 --ioengine=libaio --rw=read --eta-newline=1s  --size=1T 
--blocksize=512k --iodepth=32 --numjobs=1  --runtime=10s --name=nvme_0 
--filename=/dev/nvme0n1
nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio, 
iodepth=32
fio-2.2.10
Starting 1 process
Jobs: 1 (f=1): [R(1)] [36.4% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 
00m:07s]
Jobs: 1 (f=1): [R(1)] [45.5% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 
00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [3239MB/0KB/0KB /s] [6477/0/0 iops] [eta 
00m:05s]
Jobs: 1 (f=1): [R(1)] [63.6% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 
00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 
00m:03s]
Jobs: 1 (f=1): [R(1)] [81.8% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 
00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [3239MB/0KB/0KB /s] [6477/0/0 iops] [eta 
00m:01s]
Jobs: 1 (f=1): [R(1)] [100.0% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 
00m:00s]
nvme_0: (groupid=0, jobs=1): err= 0: pid=1600: Mon May 15 17:13:46 2017
  read : io=32384MB, bw=3236.8MB/s, iops=6473, runt= 10005msec
slat (usec): min=13, max=210, avg=16.19, stdev= 6.40
clat (usec): min=1505, max=9532, avg=4925.01, stdev=87.39
 lat (usec): min=1616, max=9566, avg=4941.38, stdev=86.47
clat percentiles (usec):
 |  1.00th=[ 4896],  5.00th=[ 4896], 10.00th=[ 4896], 20.00th=[ 4896],
 | 30.00th=[ 4896], 40.00th=[ 4960], 50.00th=[ 4960], 60.00th=[ 4960],
 | 70.00th=[ 4960], 80.00th=[ 4960], 90.00th=[ 4960], 95.00th=[ 4960],
 | 99.00th=[ 4960], 99.50th=[ 4960], 99.90th=[ 4960], 99.95th=[ 4960],
 | 99.99th=[ 8640]
bw (MB  /s): min= 3209, max= 3237, per=99.96%, avg=3235.57, stdev= 6.18
lat (msec) : 2=0.03%, 4=0.03%, 10=99.94%
  cpu  : usr=1.80%, sys=16.99%, ctx=64755, majf=0, minf=528
  IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
 submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
 issued: total=r=64767/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
 latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all

[Kernel-packages] [Bug 1688158] RE: [Bug #1688158] Please help test kernel for iommu.passthrough=0/1

2017-05-12 Thread Manoj Iyer 
On Fri, May 12, 2017 at 9:06 AM, Jeff Hugo  
wrote:
> Just FYI, it will likely take me a few days to get to this.

Thanks Jeff, no problem.

> 
> Jeffrey Hugo
> Senior Engineer
> Qualcomm Datacenter Technologies, Inc.
> 1-303-247-5002
> 
> From: Manoj Iyer [mailto:manoj.i...@canonical.com]
> Sent: Thursday, May 11, 2017 9:23 PM
> To: Jeff Hugo ; 1688...@bugs.launchpad.net
> Cc: Nate Watterson ; Timur Tabi 
> 
> Subject: [Bug #1688158] Please help test kernel for 
> iommu.passthrough=0/1
> 
> Jeff,
> 
> I have back-ported patches to address "Support SMMU passthrough using 
> the default domain". This adds a new kernel command line option 
> iommu.passthrough=0/1 to Configure DMA to bypass the IOMMU for ARM64. 
> http://bugs.launchpad.net/bugs/1688158
> 
> 0 - Use IOMMU translation for DMA.
> 1 - Bypass the IOMMU for DMA.
> 
> The test case recommended by Nate using 'fio' is:
> 
> $ sudo apt update ; sudo apt install fio
> 
> $ sudo fio --name=global --readonly --group_reporting --direct=1 
> --ioengine=libaio --rw=read --eta-newline=1s  --size=1T 
> --blocksize=512k --iodepth=32 --numjobs=1  --runtime=10s 
> --name=nvme_0 --filename=/dev/nvme0n1
> 
> You can find the kernel and the instructions to install the kernel in 
> the Centriq test PPA:
> https://launchpad.net/~centriq-team/+archive/ubuntu/test/
> 
> Nate and I have gone back and forth on testing these patches and 
> finally came down to having this test run on an NVME device. 
> Unfortunately I cant seem to find any nvme  HW at this time, we would 
> need to order these drive and could take time.
> 
> Could you please test the kernel using the test case mentioned above 
> and reply back to this email with test case out put for both cases. 
> iommu.passthrough=0 and iommu.passthrough=1. We should see better 
> performance (iops may be) with IOMMU bypassed for DMA (ie 
> iommu.passthrough=1).
> 
> The bug is cced on this email, so if you replay with a summary output 
> in both cases it will be automatically recorded in the bug. Once I 
> get your test results I will be able to make a case for SRUing these 
> patches.
> 
> Thanks a ton
> Manoj Iyer

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1688158

Title:
  [SRU][Zesty] Support SMMU passthrough using the default domain

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  [Impact]
  Have the SMMU come up in a passthrough configuration, and then allow 
subsequent translation for things such as VFIO. Rather than do this in each 
SMMU driver, it's much cleaner to allow the default domain to be configured to 
be something other than DMA.

  This patch series implements a command-line option to configure the
  default domain type. Currently, it supports "dma" and "identity" which
  is sufficient for the passthrough use-case.

  [Fix]
  The following patch series in linux-next adds this support to the kernel.
  4a8d8a14c0d0 arm64: dma-mapping: Only swizzle DMA ops for IOMMU_DOMAIN_DMA
  fccb4e3b8ab0 iommu: Allow default domain type to be set on the kernel command 
line
  beb3c6a066bf iommu/arm-smmu-v3: Install bypass STEs for IOMMU_DOMAIN_IDENTITY 
domains
  67560edcd8e5 iommu/arm-smmu-v3: Make arm_smmu_install_ste_for_dev return void
  61bc671179f1 iommu/arm-smmu: Install bypass S2CRs for IOMMU_DOMAIN_IDENTITY 
domains
  0834cc28fa56 iommu/arm-smmu: Restrict domain attributes to UNMANAGED domains

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1688158] RE: [Bug #1688158] Please help test kernel for iommu.passthrough=0/1

2017-05-12 Thread Jeffrey Hugo 
Just FYI, it will likely take me a few days to get to this.

Jeffrey Hugo
Senior Engineer
Qualcomm Datacenter Technologies, Inc.
1-303-247-5002

From: Manoj Iyer [mailto:manoj.i...@canonical.com]
Sent: Thursday, May 11, 2017 9:23 PM
To: Jeff Hugo ; 1688...@bugs.launchpad.net
Cc: Nate Watterson ; Timur Tabi 

Subject: [Bug #1688158] Please help test kernel for iommu.passthrough=0/1

Jeff,

I have back-ported patches to address "Support SMMU passthrough using
the default domain". This adds a new kernel command line option
iommu.passthrough=0/1 to Configure DMA to bypass the IOMMU for ARM64.
http://bugs.launchpad.net/bugs/1688158

0 - Use IOMMU translation for DMA.
1 - Bypass the IOMMU for DMA.

The test case recommended by Nate using 'fio' is:

$ sudo apt update ; sudo apt install fio

$ sudo fio --name=global --readonly --group_reporting --direct=1
--ioengine=libaio --rw=read --eta-newline=1s  --size=1T --blocksize=512k
--iodepth=32 --numjobs=1  --runtime=10s --name=nvme_0
--filename=/dev/nvme0n1

You can find the kernel and the instructions to install the kernel in the 
Centriq test PPA:
https://launchpad.net/~centriq-team/+archive/ubuntu/test/

Nate and I have gone back and forth on testing these patches and finally
came down to having this test run on an NVME device. Unfortunately I
cant seem to find any nvme  HW at this time, we would need to order
these drive and could take time.

Could you please test the kernel using the test case mentioned above and
reply back to this email with test case out put for both cases.
iommu.passthrough=0 and iommu.passthrough=1. We should see better
performance (iops may be) with IOMMU bypassed for DMA (ie
iommu.passthrough=1).

The bug is cced on this email, so if you replay with a summary output in
both cases it will be automatically recorded in the bug. Once I get your
test results I will be able to make a case for SRUing these patches.

Thanks a ton
Manoj Iyer

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1688158

Title:
  [SRU][Zesty] Support SMMU passthrough using the default domain

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  [Impact]
  Have the SMMU come up in a passthrough configuration, and then allow 
subsequent translation for things such as VFIO. Rather than do this in each 
SMMU driver, it's much cleaner to allow the default domain to be configured to 
be something other than DMA.

  This patch series implements a command-line option to configure the
  default domain type. Currently, it supports "dma" and "identity" which
  is sufficient for the passthrough use-case.

  [Fix]
  The following patch series in linux-next adds this support to the kernel.
  4a8d8a14c0d0 arm64: dma-mapping: Only swizzle DMA ops for IOMMU_DOMAIN_DMA
  fccb4e3b8ab0 iommu: Allow default domain type to be set on the kernel command 
line
  beb3c6a066bf iommu/arm-smmu-v3: Install bypass STEs for IOMMU_DOMAIN_IDENTITY 
domains
  67560edcd8e5 iommu/arm-smmu-v3: Make arm_smmu_install_ste_for_dev return void
  61bc671179f1 iommu/arm-smmu: Install bypass S2CRs for IOMMU_DOMAIN_IDENTITY 
domains
  0834cc28fa56 iommu/arm-smmu: Restrict domain attributes to UNMANAGED domains

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp