ccing bug for the record on testing. On Wed, May 10, 2017 at 10:07 AM, Nate Watterson <[email protected]> wrote: > Unless this is a PCIe attached SATA disk, you will not see any impact > from the iommu.passthrough setting since the SATA SMMUs are not even > enabled! Additionally, the FIO results you sent last week I think > showed rates beyond the physical capabilities of SATA so you should > probably make sure that file system caching is disabled or just > access the disk directly using a readonly profile. Also, the SMMU is > sufficiently fast that you will typically not even see the SMMU > impact on a relatively low BW peripheral like SATA. I would recommend > getting an NVME drive. > -Nate > > From: Manoj Iyer [mailto:[email protected]] > Sent: Wednesday, May 10, 2017 10:05 AM > To: Nate Watterson <[email protected]> > Cc: Timur Tabi <[email protected]> > Subject: RE: how to test iommu.passthough=0/1 ?? > > > > On Wed, May 10, 2017 at 8:33 AM, Nate Watterson > <[email protected]> wrote: > >> Ok, what is the physical device that ./fio.virtualfs is being >> created on? Is it and NVME or SATA (internal or PCI attached) disk? > > Ah! it was on the SATA drive. Sorry misunderstood your initial > question. > > >> -----Original Message----- From: Manoj Iyer >> [mailto:[email protected]] Sent: Wednesday, May 10, 2017 9:06 >> AM To: Nate Watterson <[email protected]> Cc: Manoj Iyer >> <[email protected]>; Timur Tabi <[email protected]> >> Subject: RE: how to test iommu.passthough=0/1 ?? On Wed, 10 May >> 2017, Nate Watterson wrote: >>> Manoj, What was the backing store for the file mounted as >>> /dev/loop0 in the results you sent? >> dd if=/dev/zero of=./fio.virtualfs bs=1M count=5120 mkfs.ext4 >> ./fio.virtualfs sudo losetup /dev/loop0 ./fio.virtualfs >>> For iperf, I typically only test the target device as the iperf >>> server and directly connect a host PC to run the client threads. I >>> test this way because it usually has the worst performance with >>> SMMU enabled. Target CMD: iperf -s Client CMD: iperf -c >>> 192.168.2.222 -fg -i10 -t60 -P8 -Nate From: Manoj Iyer >>> [mailto:[email protected]] Sent: Tuesday, May 9, 2017 10:29 >>> PM To: Nate Watterson <[email protected]> Cc: Timur Tabi >>> <[email protected]> Subject: RE: how to test >>> iommu.passthough=0/1 ?? Nate, Thanks for that info. I will give >>> that a try. Did you see my iperf results ? those were kind of odd >>> too.. I thought with the mlx card I might be testing with PCIe >>> device .. but may I was hitting a limitation with iperf.. Also, >>> I tested the same kernel on Cavium system, and with >>> iommu.passthough=1/0 was I seeing a significant difference in the >>> iops with a file mounted to /dev/loop0. Its attached here.. On >>> Tue, May 9, 2017 at 8:52 PM, Nate Watterson >>> <[email protected]> wrote: Hi Manoj, Sorry for the late >>> reply. I finally found some time to test passthrough mode and >>> verified using the SMMU performance counters that the >>> ‘iommu.passthrough’ param is working as I would expect it to (1 >>> == Passthrough / 0 == non-Passthrough). I am not sure why you’re >>> seeing this strange behavior but based on the logs you’ve sent, I >>> have a few comments: Your boot log is showing that only the PCIe >>> SMMUs are enabled (this is good, it means your FW is recent). >>> Because of this, you will need to do your testing with a PCIe >>> attached device instead of the SD card or SSD (assuming it was >>> connected to the onboard SATA). I would recommend using an NVME >>> drive as the impact of enabling passthrough mode will only really >>> be apparent with devices capable of sustaining high IO throughput. >>> I would recommend changing your FIO profile to access the device >>> directly instead of going through the filesystem. Additionally, I >>> would recommend using a readonly profile without randomization to >>> preserve the disk and get consistent results. The following is the >>> FIO command that I use: fio --name=global --readonly >>> --group_reporting \ --direct=1 --ioengine=libaio --rw=read >>> --eta-newline=1s \ --size=1T --blocksize=512k --iodepth=32 >>> --numjobs=1 --runtime=10s \ --name=nvme_0 --filename=/dev/nvme0n1 >>> *** iommu.passthrough=1 *** READ: io=25301MB, aggrb=2528.3MB/s, >>> minb=2528.3MB/s, maxb=2528.3MB/s, mint=10007msec, maxt=10007msec >>> *** iommu.passthrough=0 *** READ: io=22657MB, aggrb=2265.5MB/s, >>> minb=2265.5MB/s, maxb=2265.5MB/s, mint=10001msec, maxt=10001msec >>> Hopefully this helps. Let me know if you have any more questions. >>> -Nate From: Manoj Iyer [mailto:[email protected]] Sent: >>> Thursday, May 4, 2017 4:55 PM To: Nate Watterson >>> <[email protected]> Cc: Timur Tabi <[email protected]> >>> Subject: RE: how to test iommu.passthough=0/1 ?? Nate, I am >>> attaching fio output for the CAF kernel that is available in the >>> centriq PPA. The results are not that much different. This time I >>> ran fio on the loop back disk and also on the mmcblk0 SD card. >>> Please see the attached test report. Thanks Manoj Iyer On Thu, >>> May 4, 2017 at 2:46 PM, Manoj Iyer <[email protected]> >>> wrote: I have attached the boot log from the system with >>> iommu.passthrough=1. Also I ran iperf on the mlx card >>> interface.. and again the results don't make sense. I was expecting >>> see better numbers with iommu.passthrough=1. Please see the >>> attached test data for UDP and TCP. Figuring out what firmware I >>> have was always a challenge. I had brought this up with the fw >>> folks that there is no way to say I am on version X Y or Z.. >>> anyways.. FW on this SDP: XBL.DF.2.0.R1-00406 QDF2400_REL CRM >>> Type: FF6D3C53-2748-4017-838F-07BACCC91341 Version: 0000016C >>> Version Name: QDF2400.FW.2.0.R1-00364 bait-qsvbuild-24-bldr-lnx >>> Which I think translates to FW: commit >>> 891a97c41712112ce38ba70220c9ca85c9e52869 Author: QC Publisher >>> <[email protected]> Date: Mon Apr 17 13:02:52 2017 >>> -0700 Commit label r00364.1 - N/A 0.0.364.1 >>> TRACKING-ID:920c38a9-68f4-4218-870e-e9c593e321ad On Thu, May 4, >>> 2017 at 2:21 PM, Nate Watterson <[email protected]> wrote: >>> Ok that makes a bit more sense. I still need the boot log though. >>> Is the USB device connected to one of the onboard USB controllers >>> or a PCI based USB controller? Do you happen to know what FW >>> version you’re running? -Nate From: Manoj Iyer >>> [mailto:[email protected]] Sent: Thursday, May 4, 2017 2:56 >>> PM To: Nate Watterson <[email protected]> Cc: Timur Tabi >>> <[email protected]> Subject: RE: how to test >>> iommu.passthough=0/1 ?? Opps my bad.. I am terribly sorry.. >>> /dev/sdd is a USB device. *-usb:2 description: >>> Mass storage device product: ADATA USB Flash Drive >>> vendor: ADATA description: SCSI Disk >>> physical id: 0.0.0 bus info: scsi@9:0.0.0 >>> logical name: /dev/sdd On Thu, May 4, 2017 at >>> 1:30 PM, Nate Watterson <[email protected]> wrote: Please >>> send the boot log. How is the SD card attached? I would have >>> expected it to show up as an mmcblk device but it appears to be >>> showing up at /dev/sdd. -Nate From: Nate Watterson Sent: >>> Thursday, May 4, 2017 2:12 PM To: 'Manoj Iyer' >>> <[email protected]> Cc: Timur Tabi <[email protected]> >>> Subject: RE: how to test iommu.passthough=0/1 ?? Some of those >>> results are shocking. I’ll try your FIO profile here a bit later >>> and let you know what I get. -Nate From: Manoj Iyer >>> [mailto:[email protected]] Sent: Thursday, May 4, 2017 1:00 >>> PM To: Nate Watterson <[email protected]> Cc: Timur Tabi >>> <[email protected]> Subject: RE: how to test >>> iommu.passthough=0/1 ?? I tried fio on a file on SSD mounted to >>> loopback device, and on the SD card on the AW SDP. I dont have any >>> additional disk on it so I am limited to those options. But the >>> results are a bit puzzling.. I see lower iops for >>> iommu.passthough=1. I have attached the test and output, may be I >>> am looking at this wrong... Any thoughts? On Thu, May 4, 2017 >>> at 10:25 AM, Nate Watterson <[email protected]> wrote: >>> Manoj, I don’t think there will be anything obvious in dmesg you >>> can check. You’ll pretty much have to run an IO benchmark (eg: >>> iperf/fio) in the *host* kernel on a device with sufficient IO >>> throughput to show the effects of the change. You should see a >>> significant performance benefit to having iommu.passthrough==1. >>> Note that this will have absolutely no impact at all on IO >>> performance within a VM. Let me know if you have any other >>> questions. -Nate From: Manoj Iyer >>> [mailto:[email protected]] Sent: Thursday, May 4, 2017 1:26 >>> AM To: Timur Tabi <[email protected]>; Nate Watterson >>> <[email protected]> Subject: how to test >>> iommu.passthough=0/1 ?? Nate/Timur, I backported the SMMU pass >>> though using default domain patch series from linux-next. I have a >>> public bug filed >>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158 to SRU >>> this patch series. But I cant figure out a way of testing >>> iommu.passthough=0/1. Could you pls suggest a test and tell me what >>> to look for dmesg etc to compare both cases? I understand it is >>> supposed to improve performance is there a way of checking that ? >>> I compared dmesg on both cases, and also launched vms and looked >>> to see if I can spot anything obvious.. Really appreciate your >>> help with suggestions on testing this. I need to post some testing >>> data with my SRU request. Thanks Manoj Iyer. >> -- ============================ Manoj Iyer Ubuntu/Canonical ARM >> Servers - Cloud ============================
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1688158 Title: [SRU][Zesty] Support SMMU passthrough using the default domain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
