Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Mon, Nov 16, 2015 at 12:42 AM, David Woodhouse wrote: > > On Sun, 2015-11-15 at 22:54 -0800, Benjamin Serebrin wrote: > > We looked into Intel IOMMU performance a while ago and learned a few > > useful things. We generally did a parallel 200 thread TCP_RR test, > > as

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Sun, 2015-11-15 at 22:54 -0800, Benjamin Serebrin wrote: > We looked into Intel IOMMU performance a while ago and learned a few > useful things.  We generally did a parallel 200 thread TCP_RR test, > as this churns through mappings quickly and uses all available cores. > > First, the main

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

We looked into Intel IOMMU performance a while ago and learned a few useful things. We generally did a parallel 200 thread TCP_RR test, as this churns through mappings quickly and uses all available cores. First, the main bottleneck was software performance[1]. This study preceded the recent

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Thu, Nov 05, 2015 at 04:11:21PM -0500, David Miller wrote: > > And for the record Sowmini fixed a lot of the lock contention: > > commit ff7d37a502022149655c18035b99a53391be0383 > Author: Sowmini Varadhan > Date: Thu Apr 9 15:33:30 2015 -0400 > > Break up

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

From: Joerg Roedel Date: Thu, 5 Nov 2015 14:42:06 +0100 > Contended IOMMU locks are not only a problem on SPARC, but on x86 and > various other IOMMU drivers too. But I have some ideas on how to improve > the situation there. And for the record Sowmini fixed a lot of the lock

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

From: David Woodhouse Date: Thu, 29 Oct 2015 22:35:25 + > For the receive side, it shouldn't be beyond the wit of man to > introduce an API which allocates *and* DMA-maps a skb. Pass it to > netif_rx() still mapped, with a destructor that just shoves it back in > a pool

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Mon, Nov 02, 2015 at 07:32:19PM +0200, Shamir Rabinovitch wrote: > Correct. This issue is one of the concerns here in the previous replies. > I will take different approach which will not require the IOMMU bypass > per mapping. Will try to shift to the x86 'iommu=pt' approach. Yeah, it doesn't

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Tue, Nov 03, 2015 at 10:08:13AM +1100, Benjamin Herrenschmidt wrote: > On Mon, 2015-11-02 at 22:45 +0100, Arnd Bergmann wrote: > > > Then I would argue for naming this differently. Make it an optional > > > hint "DMA_ATTR_HIGH_PERF" or something like that. Whether this is > > > achieved via

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Tue, 2015-11-03 at 14:11 +0100, Christoph Hellwig wrote: > > xHCI for example, vs. something like 10G ethernet... but yes I agree it > > sucks. I don't like that sort of policy anywhere in drivers. On the > > other hand the platform doesn't have much information to make that sort > > of

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Thu, Oct 29, 2015 at 09:32:32AM +0200, Shamir Rabinovitch wrote: > For the IB case, setting and tearing DMA mappings for the drivers data buffers > is expensive. But we could for example consider to map all the HW control > objects > that validate the HW access to the drivers data buffers as

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Mon, 2015-11-02 at 14:07 +0200, Shamir Rabinovitch wrote: > On Mon, Nov 02, 2015 at 09:00:34PM +1100, Benjamin Herrenschmidt > wrote: > > > > Chosing on a per-mapping basis *in the back end* might still make > > some > > In my case, choosing mapping based on the hardware that will use this >

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Mon, Nov 02, 2015 at 03:44:27PM +0100, Joerg Roedel wrote: > > How do you envision this per-mapping by-pass to work? For the DMA-API > mappings you have only one device address space. For by-pass to work, > you need to map all physical memory of the machine into this space, > which leaves the

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Tue, Nov 03, 2015 at 07:13:28AM +1100, Benjamin Herrenschmidt wrote: > > Then I would argue for naming this differently. Make it an optional > hint "DMA_ATTR_HIGH_PERF" or something like that. Whether this is > achieved via using a bypass or other means in the backend not the > business of the

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Tuesday 03 November 2015 07:13:28 Benjamin Herrenschmidt wrote: > On Mon, 2015-11-02 at 14:07 +0200, Shamir Rabinovitch wrote: > > On Mon, Nov 02, 2015 at 09:00:34PM +1100, Benjamin Herrenschmidt > > wrote: > > > > > > Chosing on a per-mapping basis *in the back end* might still make > > >

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Mon, 2015-11-02 at 08:10 +1100, Benjamin Herrenschmidt wrote: > On Sun, 2015-11-01 at 09:45 +0200, Shamir Rabinovitch wrote: > > Not sure this use case is possible for Infiniband where application hold > > the data buffers and there is no way to force application to re use the > > buffer as

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Mon, 2015-11-02 at 22:48 +, David Woodhouse wrote: > > In the Intel case, the mapping setup is entirely per-device (except for > crap devices and devices behind a PCIe-PCI bridge, etc.). > > So you can happily have a passthrough mapping for *one* device, without > making that same mapping

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Mon, 2015-11-02 at 09:23 +0200, Shamir Rabinovitch wrote: > To summary - > > 1. The whole point of the IOMMU pass through was to get bigger address space > and faster map/unmap operations for performance critical hardware > 2. SPARC IOMMU in particular has the ability to DVMA which

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Mon, 2015-11-02 at 22:45 +0100, Arnd Bergmann wrote: > > Then I would argue for naming this differently. Make it an optional > > hint "DMA_ATTR_HIGH_PERF" or something like that. Whether this is > > achieved via using a bypass or other means in the backend not the > > business of the driver. >

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Fri, Oct 30, 2015 at 11:32:06AM +0100, Arnd Bergmann wrote: > I wonder if the 'iommu=force' attribute is too coarse-grained though, > and if we should perhaps allow a per-device setting on architectures > that allow this. Yeah, definitly. Currently we only have iommu=pt to enable pass-through

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Mon, Nov 02, 2015 at 09:00:34PM +1100, Benjamin Herrenschmidt wrote: > > Chosing on a per-mapping basis *in the back end* might still make some In my case, choosing mapping based on the hardware that will use this mappings makes more sense. Most hardware are not that performance sensitive as

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Thu, Oct 29, 2015 at 10:35:25PM +, David Woodhouse wrote: > > > > Could this be mitigated using pools? I don't know if the net code > > would play along easily. > > For the receive side, it shouldn't be beyond the wit of man to > introduce an API which allocates *and* DMA-maps a skb.

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Sun, 2015-11-01 at 09:45 +0200, Shamir Rabinovitch wrote: > Not sure this use case is possible for Infiniband where application hold > the data buffers and there is no way to force application to re use the > buffer as suggested. > > This is why I think there will be no easy way to bypass the

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Mon, Nov 02, 2015 at 08:10:49AM +1100, Benjamin Herrenschmidt wrote: > But but but ... > > What I don't understand is how that brings you any safety. Limited safety maybe? If some device DMA mappings are via IOMMU and this fall to some address range that is far from the bypass / pass

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Saturday 31 October 2015 10:17:22 Benjamin Herrenschmidt wrote: > On Fri, 2015-10-30 at 11:32 +0100, Arnd Bergmann wrote: > > On Thursday 29 October 2015 10:10:46 Benjamin Herrenschmidt wrote: > > > > > > > Maybe we should at least coordinate IOMMU 'paranoid/fast' modes > > > > across > > > >

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Fri, 2015-10-30 at 11:32 +0100, Arnd Bergmann wrote: > On Thursday 29 October 2015 10:10:46 Benjamin Herrenschmidt wrote: > > > > > Maybe we should at least coordinate IOMMU 'paranoid/fast' modes > > > across > > > architectures, and then the DMA_ATTR_IOMMU_BYPASS flag would have > > > a > > >

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Thursday 29 October 2015 10:10:46 Benjamin Herrenschmidt wrote: > > > Maybe we should at least coordinate IOMMU 'paranoid/fast' modes across > > architectures, and then the DMA_ATTR_IOMMU_BYPASS flag would have a > > sane meaning in the paranoid mode (and perhaps we'd want an ultra > >

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Oct 28, 2015 6:11 PM, "Benjamin Herrenschmidt" wrote: > > On Thu, 2015-10-29 at 09:42 +0900, David Woodhouse wrote: > > On Thu, 2015-10-29 at 09:32 +0900, Benjamin Herrenschmidt wrote: > > > > > On Power, I generally have 2 IOMMU windows for a device, one at the > > >

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Thu, 2015-10-29 at 11:31 -0700, Andy Lutomirski wrote: > On Oct 28, 2015 6:11 PM, "Benjamin Herrenschmidt" > wrote: > > > > On Thu, 2015-10-29 at 09:42 +0900, David Woodhouse wrote: > > > On Thu, 2015-10-29 at 09:32 +0900, Benjamin Herrenschmidt wrote: > > > > > > >

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Thu, Oct 29, 2015 at 09:42:12AM +0900, David Woodhouse wrote: > Aside from the lack of security, the other disadvantage of that is that > you have to pin *all* pages of a guest in case DMA happens; you don't > get to pin *only* those pages which are referenced by that guest's > IOMMU page

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Thu, 2015-10-29 at 11:31 -0700, Andy Lutomirski wrote: > On Oct 28, 2015 6:11 PM, "Benjamin Herrenschmidt" > wrote: > > > > On Thu, 2015-10-29 at 09:42 +0900, David Woodhouse wrote: > > > On Thu, 2015-10-29 at 09:32 +0900, Benjamin Herrenschmidt wrote: > > > > > > >

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Wed, 2015-10-28 at 13:10 +0200, Shamir Rabinovitch wrote: > On Wed, Oct 28, 2015 at 03:30:01PM +0900, David Woodhouse wrote: > > > > +For systems with IOMMU it is assumed all DMA translations use the > > > > IOMMU. > > > > Not entirely true. We have per-device dma_ops on a most architectures

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

From: David Woodhouse Date: Wed, 28 Oct 2015 22:31:50 +0900 > On Wed, 2015-10-28 at 13:10 +0200, Shamir Rabinovitch wrote: >> On Wed, Oct 28, 2015 at 03:30:01PM +0900, David Woodhouse wrote: >> > > > +For systems with IOMMU it is assumed all DMA translations use the >> > >

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Wed, Oct 28, 2015 at 03:30:01PM +0900, David Woodhouse wrote: > > > +For systems with IOMMU it is assumed all DMA translations use the IOMMU. > > Not entirely true. We have per-device dma_ops on a most architectures > already, and we were just talking about the need to add them to >

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Wed, 2015-10-28 at 07:07 -0700, David Miller wrote: > In the sparc64 case, the 64-bit DMA address space is divided into > IOMMU translated and non-IOMMU translated. > > You just set the high bits differently depending upon what you want. Wait, does that mean a (rogue) device could *always*

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Sun, 2015-10-25 at 09:07 -0700, Shamir Rabinovitch wrote: > > + > +DMA_ATTR_IOMMU_BYPASS > +- > + > > +For systems with IOMMU it is assumed all DMA translations use the IOMMU. Not entirely true. We have per-device dma_ops on a most architectures already, and we were just

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Thu, 2015-10-29 at 09:42 +0900, David Woodhouse wrote: > On Thu, 2015-10-29 at 09:32 +0900, Benjamin Herrenschmidt wrote: > > > On Power, I generally have 2 IOMMU windows for a device, one at the > > bottom is remapped, and is generally used for 32-bit devices and the > > one at the top us

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

From: David Woodhouse Date: Wed, 28 Oct 2015 22:57:12 +0900 > On Wed, 2015-10-28 at 07:07 -0700, David Miller wrote: >> In the sparc64 case, the 64-bit DMA address space is divided into >> IOMMU translated and non-IOMMU translated. >> >> You just set the high bits

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Wed, 2015-10-28 at 22:31 +0900, David Woodhouse wrote: > We have an option in the Intel IOMMU for pass-through mode too, which > basically *is* a total bypass. In practice, what's the difference > between that and a "simple translation that does not require any > [translation]"? We set up a

Re: [PATCH v1 2/2] dma-mapping-common: add DMA attribute - DMA_ATTR_IOMMU_BYPASS

On Thu, 2015-10-29 at 09:32 +0900, Benjamin Herrenschmidt wrote: > On Power, I generally have 2 IOMMU windows for a device, one at the > bottom is remapped, and is generally used for 32-bit devices and the > one at the top us setup as a bypass So in the normal case of decent 64-bit devices (and