High lock contention on async_umap_flush_lock when intel_iommu=on

[Charles Suresh]

When bridging traffic across two Intel X540-AT2 NICs using openvswitch(OVS),
I can get ~10 Mpps of small packet (64 byte) traffic on a Dell R620 system
using 2x Intel E5-2650 v2 @ 2.6 GHz using a Spirent generator.

Using the identical system and after enabling intel_iommu=on during boot,
the throughput drops to ~500 Kpps.

Looking at the output from perf, I see that ~80% of the CPUs are consumed
spinning on locks. The highest contention appears to be on async_umap_flush_lock
acquired in add_unmap. This area of code appears unchanged since it was
introduced back in 2008. The code review back then called out the lack of
scalability of the code as seen in this snippet from 
http://lkml.iu.edu/hypermail/linux/kernel/0802.3/2944.html :

---begin quote---
...
>> +static void add_unmap(struct dmar_domain *dom, struct iova *iova)
>> +{
>> + unsigned long flags;
>> +
>> + spin_lock_irqsave(&async_umap_flush_lock, flags);
> 
> How scalable is this?

Its not very, but I'm doing an insert walk of a list and need the locks.
...
---end quote---

>From looking at bugzilla no one else appears to have reported this issue
so I'm wondering if there is an easy way to side step this - is there a
workaround so that I don't need to use "intel_iommu=on"?

I'm testing with the "intel_iommu=on" setting so that I can get SR-IOV NICs
passed through to VMs running over KVM. Is this possible any other way?

Let me know if you want me to open a bug on bugzilla and I can upload the
perf.data and perf report output there instead of attaching large files to this
email. For now, if it helps, here's the snippet from the top of perf report:

---begin extract from perf report---
...
5.43%ksoftirqd/0  [kernel.kallsyms]   [k] _raw_spin_lock_irqsave
  
 |
 --- _raw_spin_lock_irqsave
|  
|--62.66%-- add_unmap
|  |  
|  |--99.93%-- intel_unmap_page.part.40
|  |  intel_unmap_page
|  |  ixgbe_poll
|  |  net_rx_action
|  |  __do_softirq
...
---end extract from perf report---

I’m not on the iommu alias - so I’d appreciate being cc’ed on any replies.

Thanks,
Charles
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v13 00/18] VFIO support for platform devices

[Baptiste Reynal]

Thanks for your answers.

I am now the sub-maintainer of vfio-platform. I will handle the rebase of
these serie and fix "[PATCH v3 0/6] vfio: type1: support for ARM SMMUS with
VFIO_IOMMU_TYPE1". New versions are coming soon.

Regards,
Baptiste

On Thu, Feb 26, 2015 at 6:32 PM, Marc Zyngier  wrote:

> Hi Baptiste,
>
> On 26/02/15 17:02, Baptiste Reynal wrote:
> > Hi everyone,
> >
> > Are there any comments on this patch series? If not, Is there
> > anything keeping this series from getting merged upstream?
>
> For a start, it looks like the dependency mentioned below is still not
> in, which is a bit of a problem.
>
> Also, trying to merge your branch below (after fixing the URL), I get
> some conflicts with the current mainline, so that's a bit of a show
> stopper too.
>
> Can you start by rebasing on top of 4.0-rc1 with all the dependencies,
> reposting the series, and cc-ing the original author of the patches (as
> I cannot see Antonios on the CC list)?
>
> Thanks,
>
> M.
>
> > Thanks,
> > Baptiste
> >
> > On Fri, Jan 30, 2015 at 2:46 PM, Baptiste Reynal <
> b.rey...@virtualopensystems.com>
> wrote:
> > This patch series aims to implement VFIO support for platform devices
> that
> > reside behind an IOMMU. Examples of such devices are devices behind an
> ARM
> > SMMU, or behind a Samsung Exynos System MMU.
> >
> > The API used is based on the existing VFIO API that is also used with PCI
> > devices. Only devices that include a basic set of IRQs and memory
> regions are
> > targeted; devices with complex relationships with other devices on a
> device
> > tree are not taken into account at this stage.
> >
> > This patch series may be applied on the following series/patches:
> >  - [PATCH v3 0/6] vfio: type1: support for ARM SMMUS with
> VFIO_IOMMU_TYPE1
> >
> > A copy can be cloned from the branch vfio-platform-v13 at:
> > g...@github.com:virtualopensystems/linux-kvm-arm.git
> >
> > Changes since v12:
> >  - Reorder chunks to be bisect-able
> > Changes since v11:
> >  - Drop support for ARM AMBA devices
> >  - vfio_platform_private.h is now self-contained
> >  - Fix masked IRQ initialization
> > Changes since v10:
> >  - Check if interrupt is already masked when setting a new trigger
> >  - Fixed kasprintf with unchecked return value in VFIO AMBA driver
> > Changes since v9:
> >  - Reworked the splitting of the patches that decouple virqfd from PCI
> >  - Some styling issues and typos
> >  - Removed superfluous includes
> >  - AMBA devices are now named vfio-amba- suffixed by the AMBA device id
> >  - Several other cleanups and fixes
> > Changes since v8:
> >  - Separate irq handler for edge and level triggered interrupts
> >  - Mutex based lock for VFIO fd open/release
> >  - Fixed bug where the first region of a platform device wasn't exposed
> >  - Read only regions can be MMAPed only read only
> >  - Code cleanups
> > Changes since v7:
> >  - Some initial placeholder functionality for PIO resources
> >  - Cleaned up code for IRQ triggering, masking and unmasking
> >  - Some functionality has been removed from this series and posted
> separately:
> >- VFIO_IOMMU_TYPE1 support for ARM SMMUs
> >- IOMMU NOEXEC patches
> >- driver_override functionality for AMBA devices
> >  - Several fixes
> > Changes since v6:
> >  - Integrated support for AMBA devices
> >  - Numerous cleanups and fixes
> > Changes since v5:
> >  - Full eventfd support for IRQ masking and unmasking.
> >  - Changed IOMMU_EXEC to IOMMU_NOEXEC, along with related flags in VFIO.
> >  - Other fixes based on reviewer comments.
> > Changes since v4:
> >  - Use static offsets for each region in the VFIO device fd
> >  - Include patch in the series for the ARM SMMU to expose IOMMU_EXEC
> >availability via IOMMU_CAP_DMA_EXEC
> >  - Rebased on VFIO multi domain support:
> >- IOMMU_EXEC is now available if at least one IOMMU in the container
> >  supports it
> >- Expose IOMMU_EXEC if available via the capability
> VFIO_IOMMU_PROT_EXEC
> >  - Some bug fixes
> > Changes since v3:
> >  - Use Kim Phillips' driver_probe_device()
> > Changes since v2:
> >  - Fixed Read/Write and MMAP on device regions
> >  - Removed dependency on Device Tree
> >  - Interrupts support
> >  - Interrupt masking/unmasking
> >  - Automask level sensitive interrupts
> >  - Introduced VFIO_DMA_MAP_FLAG_EXEC
> >  - Code clean ups
> >
> > Antonios Motakis (18):
> >   vfio/platform: initial skeleton of VFIO support for platform devices
> >   vfio: platform: probe to devices on the platform bus
> >   vfio: platform: add the VFIO PLATFORM module to Kconfig
> >   vfio/platform: return info for bound device
> >   vfio/platform: return info for device memory mapped IO regions
> >   vfio/platform: read and write support for the device fd
> >   vfio/platform: support MMAP of MMIO regions
> >   vfio/platform: return IRQ info
> >   vfio/platform: initial interrupts support code
> >   vfio/platform: trigger an interru