ccing bug for the record on testing On Wed, May 10, 2017 at 8:33 AM, Nate Watterson <[email protected]> wrote: > Ok, what is the physical device that ./fio.virtualfs is being created > on? Is it and NVME or SATA (internal or PCI attached) disk? > > -----Original Message----- > From: Manoj Iyer [mailto:[email protected]] > Sent: Wednesday, May 10, 2017 9:06 AM > To: Nate Watterson <[email protected]> > Cc: Manoj Iyer <[email protected]>; Timur Tabi > <[email protected]> > Subject: RE: how to test iommu.passthough=0/1 ?? > > On Wed, 10 May 2017, Nate Watterson wrote: > >> >> Manoj, >> >> What was the backing store for the file mounted as /dev/loop0 in >> the results >> you sent? > > dd if=/dev/zero of=./fio.virtualfs bs=1M count=5120 > mkfs.ext4 ./fio.virtualfs > sudo losetup /dev/loop0 ./fio.virtualfs > > >> >> >> >> For iperf, I typically only test the target device as the iperf >> server and >> directly connect a host PC to run the client threads. I test this >> way >> because it usually has the worst performance with SMMU enabled. >> >> >> >> Target CMD: iperf -s >> >> Client CMD: iperf -c 192.168.2.222 -fg -i10 -t60 -P8 >> >> >> >> -Nate >> >> >> >> From: Manoj Iyer [mailto:[email protected]] >> Sent: Tuesday, May 9, 2017 10:29 PM >> To: Nate Watterson <[email protected]> >> Cc: Timur Tabi <[email protected]> >> Subject: RE: how to test iommu.passthough=0/1 ?? >> >> >> >> Nate, >> >> >> >> Thanks for that info. I will give that a try. Did you see my iperf >> results ? >> those were kind of odd too.. I thought with the mlx card I might be >> testing >> with PCIe device .. but may I was hitting a limitation with iperf.. >> >> >> >> Also, I tested the same kernel on Cavium system, and with >> iommu.passthough=1/0 was I seeing a significant difference in the >> iops with >> a file mounted to /dev/loop0. Its attached here.. >> >> >> >> >> On Tue, May 9, 2017 at 8:52 PM, Nate Watterson >> <[email protected]> >> wrote: >> >> Hi Manoj, >> >> Sorry for the late reply. I finally found some time to test >> passthrough mode and verified using the SMMU performance >> counters that the ‘iommu.passthrough’ param is working as >> I >> would expect it to (1 == Passthrough / 0 == non-Passthrough). >> I >> am not sure why you’re seeing this strange behavior but >> based on >> the logs you’ve sent, I have a few comments: >> >> >> >> Your boot log is showing that only the PCIe SMMUs are enabled >> (this is good, it means your FW is recent). Because of this, >> you >> will need to do your testing with a PCIe attached device >> instead >> of the SD card or SSD (assuming it was connected to the >> onboard >> SATA). I would recommend using an NVME drive as the impact of >> enabling passthrough mode will only really be apparent with >> devices capable of sustaining high IO throughput. >> >> >> >> I would recommend changing your FIO profile to access the >> device >> directly instead of going through the filesystem. >> Additionally, >> I would recommend using a readonly profile without >> randomization >> to preserve the disk and get consistent results. The >> following >> is the FIO command that I use: >> >> >> >> fio --name=global --readonly --group_reporting \ >> >> --direct=1 --ioengine=libaio --rw=read --eta-newline=1s \ >> >> --size=1T --blocksize=512k --iodepth=32 --numjobs=1 >> --runtime=10s \ >> >> --name=nvme_0 --filename=/dev/nvme0n1 >> >> >> >> *** iommu.passthrough=1 *** >> >> READ: io=25301MB, aggrb=2528.3MB/s, minb=2528.3MB/s, >> maxb=2528.3MB/s, mint=10007msec, maxt=10007msec >> >> >> >> *** iommu.passthrough=0 *** >> >> READ: io=22657MB, aggrb=2265.5MB/s, minb=2265.5MB/s, >> maxb=2265.5MB/s, mint=10001msec, maxt=10001msec >> >> >> >> Hopefully this helps. Let me know if you have any more >> questions. >> >> -Nate >> >> >> >> From: Manoj Iyer [mailto:[email protected]] >> Sent: Thursday, May 4, 2017 4:55 PM >> To: Nate Watterson <[email protected]> >> Cc: Timur Tabi <[email protected]> >> Subject: RE: how to test iommu.passthough=0/1 ?? >> >> >> >> Nate, >> >> >> >> I am attaching fio output for the CAF kernel that is available in >> the >> centriq PPA. The results are not that much different. This time I >> ran >> fio on the loop back disk and also on the mmcblk0 SD card. >> >> >> >> Please see the attached test report. >> >> >> >> Thanks >> >> Manoj Iyer >> >> On Thu, May 4, 2017 at 2:46 PM, Manoj Iyer >> <[email protected]> >> wrote: >> >> I have attached the boot log from the system with >> iommu.passthrough=1. >> >> >> >> Also I ran iperf on the mlx card interface.. and again the >> results don't make sense. I was expecting see better numbers >> with iommu.passthrough=1. Please see the attached test data for >> UDP and TCP. >> >> >> >> Figuring out what firmware I have was always a challenge. I had >> brought this up with the fw folks that there is no way to say I >> am on version X Y or Z.. anyways.. >> >> >> >> FW on this SDP: >> >> XBL.DF.2.0.R1-00406 QDF2400_REL CRM >> >> >> >> Type: FF6D3C53-2748-4017-838F-07BACCC91341 >> >> Version: 0000016C >> >> Version Name: QDF2400.FW.2.0.R1-00364 bait-qsvbuild-24-bldr-lnx >> >> >> >> Which I think translates to FW: >> >> commit 891a97c41712112ce38ba70220c9ca85c9e52869 >> >> Author: QC Publisher <[email protected]> >> >> Date: Mon Apr 17 13:02:52 2017 -0700 >> >> >> >> Commit label r00364.1 - N/A 0.0.364.1 >> >> >> >> TRACKING-ID:920c38a9-68f4-4218-870e-e9c593e321ad >> >> >> >> >> >> >> On Thu, May 4, 2017 at 2:21 PM, Nate Watterson >> <[email protected]> wrote: >> >> Ok that makes a bit more sense. I still need the >> boot log though. Is the USB device connected to one >> of the onboard USB controllers or a PCI based USB >> controller? Do you happen to know what FW version >> you’re running? >> >> -Nate >> >> >> >> From: Manoj Iyer [mailto:[email protected]] >> Sent: Thursday, May 4, 2017 2:56 PM >> To: Nate Watterson <[email protected]> >> Cc: Timur Tabi <[email protected]> >> Subject: RE: how to test iommu.passthough=0/1 ?? >> >> >> >> Opps my bad.. I am terribly sorry.. /dev/sdd is a >> USB device. >> >> >> >> *-usb:2 >> >> description: Mass storage device >> >> product: ADATA USB Flash Drive >> >> vendor: ADATA >> >> description: SCSI Disk >> >> physical id: 0.0.0 >> >> bus info: scsi@9:0.0.0 >> >> logical name: /dev/sdd >> >> >> >> >> On Thu, May 4, 2017 at 1:30 PM, Nate Watterson >> <[email protected]> wrote: >> >> Please send the boot log. How is the SD card >> attached? I would have expected it to show up >> as an mmcblk device but it appears to be >> showing up at /dev/sdd. >> >> -Nate >> >> >> >> From: Nate Watterson >> Sent: Thursday, May 4, 2017 2:12 PM >> To: 'Manoj Iyer' <[email protected]> >> Cc: Timur Tabi <[email protected]> >> Subject: RE: how to test iommu.passthough=0/1 >> ?? >> >> >> >> Some of those results are shocking. I’ll try your >> FIO profile here a bit later and let you know what I >> get. >> >> -Nate >> >> >> >> From: Manoj Iyer [mailto:[email protected]] >> Sent: Thursday, May 4, 2017 1:00 PM >> To: Nate Watterson <[email protected]> >> Cc: Timur Tabi <[email protected]> >> Subject: RE: how to test iommu.passthough=0/1 ?? >> >> >> >> I tried fio on a file on SSD mounted to loopback >> device, and on the SD card on the AW SDP. I dont >> have any additional disk on it so I am limited to >> those options. But the results are a bit puzzling.. >> I see lower iops for iommu.passthough=1. I have >> attached the test and output, may be I am looking at >> this wrong... >> >> >> >> Any thoughts? >> >> On Thu, May 4, 2017 at 10:25 AM, Nate Watterson >> <[email protected]> wrote: >> >> Manoj, >> >> I don’t think there will be anything >> obvious in dmesg you can check. You’ll >> pretty much have to run an IO benchmark >> (eg: iperf/fio) in the *host* kernel on >> a device with sufficient IO throughput >> to show the effects of the change. You >> should see a significant performance >> benefit to having iommu.passthrough==1. >> Note that this will have absolutely no >> impact at all on IO performance within a >> VM. Let me know if you have any other >> questions. >> >> >> >> -Nate >> >> >> >> From: Manoj Iyer >> [mailto:[email protected]] >> Sent: Thursday, May 4, 2017 1:26 AM >> To: Timur Tabi <[email protected]>; >> Nate Watterson >> <[email protected]> >> Subject: how to test >> iommu.passthough=0/1 ?? >> >> >> >> Nate/Timur, >> >> >> >> I backported the SMMU pass though using >> default domain patch series from linux-next. I >> have a public bug >> filed https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158 >> to SRU this patch series. But I cant figure >> out a way of testing iommu.passthough=0/1. >> Could you pls suggest a test and tell me what >> to look for dmesg etc to compare both cases? I >> understand it is supposed to improve >> performance is there a way of checking that ? >> >> >> >> I compared dmesg on both cases, and also >> launched vms and looked to see if I can spot >> anything obvious.. Really appreciate your help >> with suggestions on testing this. I need to >> post some testing data with my SRU request. >> >> >> >> Thanks >> >> Manoj Iyer. >> >> >> > > -- > ============================ > Manoj Iyer > Ubuntu/Canonical > ARM Servers - Cloud > ============================
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1688158 Title: [SRU][Zesty] Support SMMU passthrough using the default domain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
