Re: [PATCH v3 0/3] create sysfs representation of ACPI HMAT

[Matthew Wilcox]

On Wed, Dec 27, 2017 at 10:10:34AM +0100, Brice Goglin wrote:
> > Perhaps we can enlist /proc/iomem or a similar enumeration interface
> > to tell userspace the NUMA node and whether the kernel thinks it has
> > better or worse performance characteristics relative to base
> > system-RAM, i.e. new IORES_DESC_* values. I'm worried that if we start
> > publishing absolute numbers in sysfs userspace will default to looking
> > for specific magic numbers in sysfs vs asking the kernel for memory
> > that has performance characteristics relative to base "System RAM". In
> > other words the absolute performance information that the HMAT
> > publishes is useful to the kernel, but it's not clear that userspace
> > needs that vs a relative indicator for making NUMA node preference
> > decisions.
> 
> Some HPC users will benchmark the machine to discovery actual
> performance numbers anyway.
> However, most users won't do this. They will want to know relative
> performance of different nodes. If you normalize HMAT values by dividing
> them with system-RAM values, that's likely OK. If you just say "that
> node is faster than system RAM", it's not precise enough.

So "this memory has 800% bandwidth of normal" and "this memory has 70%
bandwidth of normal"?

[PATCH] pci/hotplug: Delete error messages for a failed memory allocation in four functions

[SF Markus Elfring]

From: Markus Elfring 
Date: Fri, 29 Dec 2017 15:47:00 +0100

Omit extra messages for a memory allocation failure in these functions.

This issue was detected by using the Coccinelle software.

Signed-off-by: Markus Elfring 
---
 drivers/pci/hotplug/pciehp_hpc.c  | 5 ++---
 drivers/pci/hotplug/pnv_php.c | 8 +---
 drivers/pci/hotplug/shpchp_core.c | 5 ++---
 3 files changed, 5 insertions(+), 13 deletions(-)

diff --git a/drivers/pci/hotplug/pciehp_hpc.c b/drivers/pci/hotplug/pciehp_hpc.c
index 7bab0606f1a9..ed27635ec740 100644
--- a/drivers/pci/hotplug/pciehp_hpc.c
+++ b/drivers/pci/hotplug/pciehp_hpc.c
@@ -838,10 +838,9 @@ struct controller *pcie_init(struct pcie_device *dev)
struct pci_dev *pdev = dev->port;
 
ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
-   if (!ctrl) {
-   dev_err(>device, "%s: Out of memory\n", __func__);
+   if (!ctrl)
goto abort;
-   }
+
ctrl->pcie = dev;
pcie_capability_read_dword(pdev, PCI_EXP_SLTCAP, _cap);
 
diff --git a/drivers/pci/hotplug/pnv_php.c b/drivers/pci/hotplug/pnv_php.c
index 74f6a17e4614..4ab318a0800d 100644
--- a/drivers/pci/hotplug/pnv_php.c
+++ b/drivers/pci/hotplug/pnv_php.c
@@ -266,7 +266,6 @@ static int pnv_php_add_devtree(struct pnv_php_slot 
*php_slot)
fdt1 = kzalloc(0x1, GFP_KERNEL);
if (!fdt1) {
ret = -ENOMEM;
-   dev_warn(_slot->pdev->dev, "Cannot alloc FDT blob\n");
goto out;
}
 
@@ -280,8 +279,6 @@ static int pnv_php_add_devtree(struct pnv_php_slot 
*php_slot)
fdt = kzalloc(fdt_totalsize(fdt1), GFP_KERNEL);
if (!fdt) {
ret = -ENOMEM;
-   dev_warn(_slot->pdev->dev, "Cannot %d bytes memory\n",
-fdt_totalsize(fdt1));
goto free_fdt1;
}
 
@@ -756,11 +753,8 @@ static irqreturn_t pnv_php_interrupt(int irq, void *data)
 * fine as the PCI devices (PE) aren't functional any more.
 */
event = kzalloc(sizeof(*event), GFP_ATOMIC);
-   if (!event) {
-   dev_warn(>dev, "PCI slot [%s] missed hotplug event 
0x%04x\n",
-php_slot->name, sts);
+   if (!event)
return IRQ_HANDLED;
-   }
 
dev_info(>dev, "PCI slot [%s] %s (IRQ: %d)\n",
 php_slot->name, added ? "added" : "removed", irq);
diff --git a/drivers/pci/hotplug/shpchp_core.c 
b/drivers/pci/hotplug/shpchp_core.c
index 7bfb87bd2b7e..2f2b87acf179 100644
--- a/drivers/pci/hotplug/shpchp_core.c
+++ b/drivers/pci/hotplug/shpchp_core.c
@@ -305,10 +305,9 @@ static int shpc_probe(struct pci_dev *pdev, const struct 
pci_device_id *ent)
return -ENODEV;
 
ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
-   if (!ctrl) {
-   dev_err(>dev, "%s: Out of memory\n", __func__);
+   if (!ctrl)
goto err_out_none;
-   }
+
INIT_LIST_HEAD(>slot_list);
 
rc = shpc_init(ctrl, pdev);
-- 
2.15.1

Re: [PATCH 27/67] dma-direct: add dma address sanity checks

[Geert Uytterhoeven]

Hi Christoph,

On Fri, Dec 29, 2017 at 9:18 AM, Christoph Hellwig  wrote:
> Roughly based on the x86 pci-nommu implementation.
>
> Signed-off-by: Christoph Hellwig 

Thanks for your patch!

> --- a/lib/dma-direct.c
> +++ b/lib/dma-direct.c
> @@ -9,6 +9,24 @@
>  #include 
>  #include 
>
> +#define DIRECT_MAPPING_ERROR   0
> +
> +static bool
> +check_addr(struct device *dev, dma_addr_t dma_addr, size_t size,
> +   const char *caller)
> +{
> +   if (unlikely(dev && !dma_capable(dev, dma_addr, size))) {
> +   if (*dev->dma_mask >= DMA_BIT_MASK(32)) {
> +   dev_err(dev,
> +   "%s: overflow %llx+%zu of device mask %llx\n",

Please use "%pad" to format dma_addr_t ...

> +   caller, (long long)dma_addr, size,

... and use _addr.

> +   (long long)*dev->dma_mask);

This cast is not needed, as u64 is unsigned long long in kernelspace on
all architectures.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

Re: consolidate direct dma mapping and swiotlb support

[Vladimir Murzin]

On 29/12/17 08:18, Christoph Hellwig wrote:
> Almost every architecture supports a direct dma mapping implementation,
> where no iommu is used and the device dma address is a 1:1 mapping to
> the physical address or has a simple linear offset.  Currently the
> code for this implementation is most duplicated over the architectures,
> and the duplicated again in the swiotlb code, and then duplicated again
> for special cases like the x86 memory encryption DMA ops.
> 
> This series takes the existing very simple dma-noop dma mapping
> implementation, enhances it with all the x86 features and quirks, and
> creates a common set of architecture hooks for it and the swiotlb code.
> 
> It then switches a large number of architectures to this generic
> direct map implement and the new generic swiotlb dma_map ops.
> 
> Note that for now this only handles architectures that do cache coherent
> DMA, but a similar consolidation for non-coherent architectures is in the
> work for later merge windows.

Is it available in your dma-mapping.git or somewhere else?

Cheers
Vladimir

> 
> ___
> linux-arm-kernel mailing list
> linux-arm-ker...@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH 17/67] microblaze: rename dma_direct to dma_microblaze

[Julian Calaby]

Hi Christoph,

On Fri, Dec 29, 2017 at 7:18 PM, Christoph Hellwig  wrote:
> This frees the dma_direct_* namespace for a generic implementation.

Don't you mean "dma_nommu" not "dma_microblaze" in the subject line?

Thanks,

-- 
Julian Calaby

Email: julian.cal...@gmail.com
Profile: http://www.google.com/profiles/julian.calaby/

[PATCH 67/67] swiotlb: remove various exports

[Christoph Hellwig]

All these symbols are only used by arch dma_ops implementations or
xen-swiotlb.  None of which can be modular.

Signed-off-by: Christoph Hellwig 
---
 lib/swiotlb.c | 13 -
 1 file changed, 13 deletions(-)

diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 77a40b508db8..823e1055a394 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -591,7 +591,6 @@ phys_addr_t swiotlb_tbl_map_single(struct device *hwdev,
 
return tlb_addr;
 }
-EXPORT_SYMBOL_GPL(swiotlb_tbl_map_single);
 
 /*
  * Allocates bounce buffer and returns its kernel virtual address.
@@ -661,7 +660,6 @@ void swiotlb_tbl_unmap_single(struct device *hwdev, 
phys_addr_t tlb_addr,
}
spin_unlock_irqrestore(_tlb_lock, flags);
 }
-EXPORT_SYMBOL_GPL(swiotlb_tbl_unmap_single);
 
 void swiotlb_tbl_sync_single(struct device *hwdev, phys_addr_t tlb_addr,
 size_t size, enum dma_data_direction dir,
@@ -693,7 +691,6 @@ void swiotlb_tbl_sync_single(struct device *hwdev, 
phys_addr_t tlb_addr,
BUG();
}
 }
-EXPORT_SYMBOL_GPL(swiotlb_tbl_sync_single);
 
 static void *
 swiotlb_alloc_buffer(struct device *dev, size_t size, dma_addr_t *dma_handle)
@@ -827,7 +824,6 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page 
*page,
 
return __phys_to_dma(dev, io_tlb_overflow_buffer);
 }
-EXPORT_SYMBOL_GPL(swiotlb_map_page);
 
 /*
  * Unmap a single streaming mode DMA translation.  The dma_addr and size must
@@ -868,7 +864,6 @@ void swiotlb_unmap_page(struct device *hwdev, dma_addr_t 
dev_addr,
 {
unmap_single(hwdev, dev_addr, size, dir, attrs);
 }
-EXPORT_SYMBOL_GPL(swiotlb_unmap_page);
 
 /*
  * Make physical memory consistent for a single streaming mode DMA translation
@@ -906,7 +901,6 @@ swiotlb_sync_single_for_cpu(struct device *hwdev, 
dma_addr_t dev_addr,
 {
swiotlb_sync_single(hwdev, dev_addr, size, dir, SYNC_FOR_CPU);
 }
-EXPORT_SYMBOL(swiotlb_sync_single_for_cpu);
 
 void
 swiotlb_sync_single_for_device(struct device *hwdev, dma_addr_t dev_addr,
@@ -914,7 +908,6 @@ swiotlb_sync_single_for_device(struct device *hwdev, 
dma_addr_t dev_addr,
 {
swiotlb_sync_single(hwdev, dev_addr, size, dir, SYNC_FOR_DEVICE);
 }
-EXPORT_SYMBOL(swiotlb_sync_single_for_device);
 
 /*
  * Map a set of buffers described by scatterlist in streaming mode for DMA.
@@ -966,7 +959,6 @@ swiotlb_map_sg_attrs(struct device *hwdev, struct 
scatterlist *sgl, int nelems,
}
return nelems;
 }
-EXPORT_SYMBOL(swiotlb_map_sg_attrs);
 
 /*
  * Unmap a set of streaming mode DMA translations.  Again, cpu read rules
@@ -986,7 +978,6 @@ swiotlb_unmap_sg_attrs(struct device *hwdev, struct 
scatterlist *sgl,
unmap_single(hwdev, sg->dma_address, sg_dma_len(sg), dir,
 attrs);
 }
-EXPORT_SYMBOL(swiotlb_unmap_sg_attrs);
 
 /*
  * Make physical memory consistent for a set of streaming mode DMA translations
@@ -1014,7 +1005,6 @@ swiotlb_sync_sg_for_cpu(struct device *hwdev, struct 
scatterlist *sg,
 {
swiotlb_sync_sg(hwdev, sg, nelems, dir, SYNC_FOR_CPU);
 }
-EXPORT_SYMBOL(swiotlb_sync_sg_for_cpu);
 
 void
 swiotlb_sync_sg_for_device(struct device *hwdev, struct scatterlist *sg,
@@ -1022,14 +1012,12 @@ swiotlb_sync_sg_for_device(struct device *hwdev, struct 
scatterlist *sg,
 {
swiotlb_sync_sg(hwdev, sg, nelems, dir, SYNC_FOR_DEVICE);
 }
-EXPORT_SYMBOL(swiotlb_sync_sg_for_device);
 
 int
 swiotlb_dma_mapping_error(struct device *hwdev, dma_addr_t dma_addr)
 {
return (dma_addr == __phys_to_dma(hwdev, io_tlb_overflow_buffer));
 }
-EXPORT_SYMBOL(swiotlb_dma_mapping_error);
 
 /*
  * Return whether the given device DMA address mask can be supported
@@ -1042,7 +1030,6 @@ swiotlb_dma_supported(struct device *hwdev, u64 mask)
 {
return __phys_to_dma(hwdev, io_tlb_end - 1) <= mask;
 }
-EXPORT_SYMBOL(swiotlb_dma_supported);
 
 #ifdef CONFIG_DMA_DIRECT_OPS
 void *swiotlb_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle,
-- 
2.14.2

[PATCH 66/67] swiotlb: remove swiotlb_{alloc,free}_coherent

[Christoph Hellwig]

Unused now that everyone uses swiotlb_{alloc,free}.

Signed-off-by: Christoph Hellwig 
---
 include/linux/swiotlb.h |  8 
 lib/swiotlb.c   | 41 -
 2 files changed, 49 deletions(-)

diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index 5b1f2a00491c..965be92c33b5 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -72,14 +72,6 @@ void *swiotlb_alloc(struct device *hwdev, size_t size, 
dma_addr_t *dma_handle,
 void swiotlb_free(struct device *dev, size_t size, void *vaddr,
dma_addr_t dma_addr, unsigned long attrs);
 
-extern void
-*swiotlb_alloc_coherent(struct device *hwdev, size_t size,
-   dma_addr_t *dma_handle, gfp_t flags);
-
-extern void
-swiotlb_free_coherent(struct device *hwdev, size_t size,
- void *vaddr, dma_addr_t dma_handle);
-
 extern dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
   unsigned long offset, size_t size,
   enum dma_data_direction dir,
diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 4ea0b5710618..77a40b508db8 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -157,13 +157,6 @@ unsigned long swiotlb_size_or_default(void)
return size ? size : (IO_TLB_DEFAULT_SIZE);
 }
 
-/* Note that this doesn't work with highmem page */
-static dma_addr_t swiotlb_virt_to_bus(struct device *hwdev,
- volatile void *address)
-{
-   return phys_to_dma(hwdev, virt_to_phys(address));
-}
-
 static bool no_iotlb_memory;
 
 void swiotlb_print_info(void)
@@ -743,31 +736,6 @@ swiotlb_alloc_buffer(struct device *dev, size_t size, 
dma_addr_t *dma_handle)
return NULL;
 }
 
-void *
-swiotlb_alloc_coherent(struct device *hwdev, size_t size,
-  dma_addr_t *dma_handle, gfp_t flags)
-{
-   int order = get_order(size);
-   void *ret;
-
-   ret = (void *)__get_free_pages(flags, order);
-   if (ret) {
-   *dma_handle = swiotlb_virt_to_bus(hwdev, ret);
-   if (*dma_handle  + size - 1 <= hwdev->coherent_dma_mask) {
-   memset(ret, 0, size);
-   return ret;
-   }
-
-   /*
-* The allocated memory isn't reachable by the device.
-*/
-   free_pages((unsigned long) ret, order);
-   }
-
-   return swiotlb_alloc_buffer(hwdev, size, dma_handle);
-}
-EXPORT_SYMBOL(swiotlb_alloc_coherent);
-
 static bool swiotlb_free_buffer(struct device *dev, size_t size,
dma_addr_t dma_addr)
 {
@@ -787,15 +755,6 @@ static bool swiotlb_free_buffer(struct device *dev, size_t 
size,
return true;
 }
 
-void
-swiotlb_free_coherent(struct device *hwdev, size_t size, void *vaddr,
- dma_addr_t dev_addr)
-{
-   if (!swiotlb_free_buffer(hwdev, size, dev_addr))
-   free_pages((unsigned long)vaddr, get_order(size));
-}
-EXPORT_SYMBOL(swiotlb_free_coherent);
-
 static void
 swiotlb_full(struct device *dev, size_t size, enum dma_data_direction dir,
 int do_panic)
-- 
2.14.2

[PATCH 65/67] arm64: use swiotlb_alloc and swiotlb_free

[Christoph Hellwig]

The generic swiotlb_alloc and swiotlb_free routines already take care
of CMA allocations and adding GFP_DMA32 where needed, so use them
instead of the arm specific helpers.

Signed-off-by: Christoph Hellwig 
---
 arch/arm64/Kconfig  |  1 +
 arch/arm64/mm/dma-mapping.c | 46 +++--
 2 files changed, 4 insertions(+), 43 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 6b6985f15d02..53205c02b18a 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -59,6 +59,7 @@ config ARM64
select COMMON_CLK
select CPU_PM if (SUSPEND || CPU_IDLE)
select DCACHE_WORD_ACCESS
+   select DMA_DIRECT_OPS
select EDAC_SUPPORT
select FRAME_POINTER
select GENERIC_ALLOCATOR
diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 0d641875b20e..a96ec0181818 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -91,46 +91,6 @@ static int __free_from_pool(void *start, size_t size)
return 1;
 }
 
-static void *__dma_alloc_coherent(struct device *dev, size_t size,
- dma_addr_t *dma_handle, gfp_t flags,
- unsigned long attrs)
-{
-   if (IS_ENABLED(CONFIG_ZONE_DMA32) &&
-   dev->coherent_dma_mask <= DMA_BIT_MASK(32))
-   flags |= GFP_DMA32;
-   if (dev_get_cma_area(dev) && gfpflags_allow_blocking(flags)) {
-   struct page *page;
-   void *addr;
-
-   page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT,
-get_order(size), flags);
-   if (!page)
-   return NULL;
-
-   *dma_handle = phys_to_dma(dev, page_to_phys(page));
-   addr = page_address(page);
-   memset(addr, 0, size);
-   return addr;
-   } else {
-   return swiotlb_alloc_coherent(dev, size, dma_handle, flags);
-   }
-}
-
-static void __dma_free_coherent(struct device *dev, size_t size,
-   void *vaddr, dma_addr_t dma_handle,
-   unsigned long attrs)
-{
-   bool freed;
-   phys_addr_t paddr = dma_to_phys(dev, dma_handle);
-
-
-   freed = dma_release_from_contiguous(dev,
-   phys_to_page(paddr),
-   size >> PAGE_SHIFT);
-   if (!freed)
-   swiotlb_free_coherent(dev, size, vaddr, dma_handle);
-}
-
 static void *__dma_alloc(struct device *dev, size_t size,
 dma_addr_t *dma_handle, gfp_t flags,
 unsigned long attrs)
@@ -152,7 +112,7 @@ static void *__dma_alloc(struct device *dev, size_t size,
return addr;
}
 
-   ptr = __dma_alloc_coherent(dev, size, dma_handle, flags, attrs);
+   ptr = swiotlb_alloc(dev, size, dma_handle, flags, attrs);
if (!ptr)
goto no_mem;
 
@@ -173,7 +133,7 @@ static void *__dma_alloc(struct device *dev, size_t size,
return coherent_ptr;
 
 no_map:
-   __dma_free_coherent(dev, size, ptr, *dma_handle, attrs);
+   swiotlb_free(dev, size, ptr, *dma_handle, attrs);
 no_mem:
return NULL;
 }
@@ -191,7 +151,7 @@ static void __dma_free(struct device *dev, size_t size,
return;
vunmap(vaddr);
}
-   __dma_free_coherent(dev, size, swiotlb_addr, dma_handle, attrs);
+   swiotlb_free(dev, size, swiotlb_addr, dma_handle, attrs);
 }
 
 static dma_addr_t __swiotlb_map_page(struct device *dev, struct page *page,
-- 
2.14.2

[PATCH 64/67] arm64: replace ZONE_DMA with ZONE_DMA32

[Christoph Hellwig]

arm64 uses ZONE_DMA for allocations below 32-bits.  These days we
name the zone for that ZONE_DMA32, which will allow to use the
dma-direct and generic swiotlb code as-is, so rename it.

Signed-off-by: Christoph Hellwig 
---
 arch/arm64/Kconfig  |  2 +-
 arch/arm64/mm/dma-mapping.c |  6 +++---
 arch/arm64/mm/init.c| 16 
 3 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index c9a7e9e1414f..6b6985f15d02 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -227,7 +227,7 @@ config GENERIC_CSUM
 config GENERIC_CALIBRATE_DELAY
def_bool y
 
-config ZONE_DMA
+config ZONE_DMA32
def_bool y
 
 config HAVE_GENERIC_GUP
diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 6840426bbe77..0d641875b20e 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -95,9 +95,9 @@ static void *__dma_alloc_coherent(struct device *dev, size_t 
size,
  dma_addr_t *dma_handle, gfp_t flags,
  unsigned long attrs)
 {
-   if (IS_ENABLED(CONFIG_ZONE_DMA) &&
+   if (IS_ENABLED(CONFIG_ZONE_DMA32) &&
dev->coherent_dma_mask <= DMA_BIT_MASK(32))
-   flags |= GFP_DMA;
+   flags |= GFP_DMA32;
if (dev_get_cma_area(dev) && gfpflags_allow_blocking(flags)) {
struct page *page;
void *addr;
@@ -397,7 +397,7 @@ static int __init atomic_pool_init(void)
page = dma_alloc_from_contiguous(NULL, nr_pages,
 pool_size_order, GFP_KERNEL);
else
-   page = alloc_pages(GFP_DMA, pool_size_order);
+   page = alloc_pages(GFP_DMA32, pool_size_order);
 
if (page) {
int ret;
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 00e7b900ca41..8f03276443c9 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -217,7 +217,7 @@ static void __init reserve_elfcorehdr(void)
 }
 #endif /* CONFIG_CRASH_DUMP */
 /*
- * Return the maximum physical address for ZONE_DMA (DMA_BIT_MASK(32)). It
+ * Return the maximum physical address for ZONE_DMA32 (DMA_BIT_MASK(32)). It
  * currently assumes that for memory starting above 4G, 32-bit devices will
  * use a DMA offset.
  */
@@ -233,8 +233,8 @@ static void __init zone_sizes_init(unsigned long min, 
unsigned long max)
 {
unsigned long max_zone_pfns[MAX_NR_ZONES]  = {0};
 
-   if (IS_ENABLED(CONFIG_ZONE_DMA))
-   max_zone_pfns[ZONE_DMA] = PFN_DOWN(max_zone_dma_phys());
+   if (IS_ENABLED(CONFIG_ZONE_DMA32))
+   max_zone_pfns[ZONE_DMA32] = PFN_DOWN(max_zone_dma_phys());
max_zone_pfns[ZONE_NORMAL] = max;
 
free_area_init_nodes(max_zone_pfns);
@@ -251,9 +251,9 @@ static void __init zone_sizes_init(unsigned long min, 
unsigned long max)
memset(zone_size, 0, sizeof(zone_size));
 
/* 4GB maximum for 32-bit only capable devices */
-#ifdef CONFIG_ZONE_DMA
+#ifdef CONFIG_ZONE_DMA32
max_dma = PFN_DOWN(arm64_dma_phys_limit);
-   zone_size[ZONE_DMA] = max_dma - min;
+   zone_size[ZONE_DMA32] = max_dma - min;
 #endif
zone_size[ZONE_NORMAL] = max - max_dma;
 
@@ -266,10 +266,10 @@ static void __init zone_sizes_init(unsigned long min, 
unsigned long max)
if (start >= max)
continue;
 
-#ifdef CONFIG_ZONE_DMA
+#ifdef CONFIG_ZONE_DMA32
if (start < max_dma) {
unsigned long dma_end = min(end, max_dma);
-   zhole_size[ZONE_DMA] -= dma_end - start;
+   zhole_size[ZONE_DMA32] -= dma_end - start;
}
 #endif
if (end > max_dma) {
@@ -467,7 +467,7 @@ void __init arm64_memblock_init(void)
early_init_fdt_scan_reserved_mem();
 
/* 4GB maximum for 32-bit only capable devices */
-   if (IS_ENABLED(CONFIG_ZONE_DMA))
+   if (IS_ENABLED(CONFIG_ZONE_DMA32))
arm64_dma_phys_limit = max_zone_dma_phys();
else
arm64_dma_phys_limit = PHYS_MASK + 1;
-- 
2.14.2

[PATCH 63/67] mips: use swiotlb_{alloc,free}

[Christoph Hellwig]

These already include the GFP_DMA/GFP_DMA32 usage, and will use CMA
memory if enabled, thus avoiding the GFP_NORETRY hack.

Signed-off-by: Christoph Hellwig 
---
 arch/mips/cavium-octeon/Kconfig   |  1 +
 arch/mips/cavium-octeon/dma-octeon.c  | 26 +++---
 arch/mips/loongson64/Kconfig  |  1 +
 arch/mips/loongson64/common/dma-swiotlb.c | 21 ++---
 4 files changed, 7 insertions(+), 42 deletions(-)

diff --git a/arch/mips/cavium-octeon/Kconfig b/arch/mips/cavium-octeon/Kconfig
index 204a1670fd9b..b5eee1a57d6c 100644
--- a/arch/mips/cavium-octeon/Kconfig
+++ b/arch/mips/cavium-octeon/Kconfig
@@ -75,6 +75,7 @@ config NEED_SG_DMA_LENGTH
 
 config SWIOTLB
def_bool y
+   select DMA_DIRECT_OPS
select IOMMU_HELPER
select NEED_SG_DMA_LENGTH
 
diff --git a/arch/mips/cavium-octeon/dma-octeon.c 
b/arch/mips/cavium-octeon/dma-octeon.c
index 6440ad3f9e3b..7b335ab21697 100644
--- a/arch/mips/cavium-octeon/dma-octeon.c
+++ b/arch/mips/cavium-octeon/dma-octeon.c
@@ -159,33 +159,13 @@ static void octeon_dma_sync_sg_for_device(struct device 
*dev,
 static void *octeon_dma_alloc_coherent(struct device *dev, size_t size,
dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs)
 {
-   void *ret;
-
-   if (IS_ENABLED(CONFIG_ZONE_DMA) && dev == NULL)
-   gfp |= __GFP_DMA;
-   else if (IS_ENABLED(CONFIG_ZONE_DMA) &&
-dev->coherent_dma_mask <= DMA_BIT_MASK(24))
-   gfp |= __GFP_DMA;
-   else if (IS_ENABLED(CONFIG_ZONE_DMA32) &&
-dev->coherent_dma_mask <= DMA_BIT_MASK(32))
-   gfp |= __GFP_DMA32;
-
-   /* Don't invoke OOM killer */
-   gfp |= __GFP_NORETRY;
-
-   ret = swiotlb_alloc_coherent(dev, size, dma_handle, gfp);
+   void *ret = swiotlb_alloc(dev, size, dma_handle, gfp, attrs);
 
mb();
 
return ret;
 }
 
-static void octeon_dma_free_coherent(struct device *dev, size_t size,
-   void *vaddr, dma_addr_t dma_handle, unsigned long attrs)
-{
-   swiotlb_free_coherent(dev, size, vaddr, dma_handle);
-}
-
 static dma_addr_t octeon_unity_phys_to_dma(struct device *dev, phys_addr_t 
paddr)
 {
return paddr;
@@ -225,7 +205,7 @@ EXPORT_SYMBOL(__dma_to_phys);
 static struct octeon_dma_map_ops octeon_linear_dma_map_ops = {
.dma_map_ops = {
.alloc = octeon_dma_alloc_coherent,
-   .free = octeon_dma_free_coherent,
+   .free = swiotlb_free,
.map_page = octeon_dma_map_page,
.unmap_page = swiotlb_unmap_page,
.map_sg = octeon_dma_map_sg,
@@ -311,7 +291,7 @@ void __init plat_swiotlb_setup(void)
 static struct octeon_dma_map_ops _octeon_pci_dma_map_ops = {
.dma_map_ops = {
.alloc = octeon_dma_alloc_coherent,
-   .free = octeon_dma_free_coherent,
+   .free = swiotlb_free,
.map_page = octeon_dma_map_page,
.unmap_page = swiotlb_unmap_page,
.map_sg = octeon_dma_map_sg,
diff --git a/arch/mips/loongson64/Kconfig b/arch/mips/loongson64/Kconfig
index 0d249fc3cfe9..6f109bb54cdb 100644
--- a/arch/mips/loongson64/Kconfig
+++ b/arch/mips/loongson64/Kconfig
@@ -136,6 +136,7 @@ config SWIOTLB
bool "Soft IOMMU Support for All-Memory DMA"
default y
depends on CPU_LOONGSON3
+   select DMA_DIRECT_OPS
select IOMMU_HELPER
select NEED_SG_DMA_LENGTH
select NEED_DMA_MAP_STATE
diff --git a/arch/mips/loongson64/common/dma-swiotlb.c 
b/arch/mips/loongson64/common/dma-swiotlb.c
index 0a02ea70e39f..6a739f8ae110 100644
--- a/arch/mips/loongson64/common/dma-swiotlb.c
+++ b/arch/mips/loongson64/common/dma-swiotlb.c
@@ -13,29 +13,12 @@
 static void *loongson_dma_alloc_coherent(struct device *dev, size_t size,
dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs)
 {
-   void *ret;
+   void *ret = swiotlb_alloc(dev, size, dma_handle, gfp, attrs);
 
-   if ((IS_ENABLED(CONFIG_ISA) && dev == NULL) ||
-   (IS_ENABLED(CONFIG_ZONE_DMA) &&
-dev->coherent_dma_mask < DMA_BIT_MASK(32)))
-   gfp |= __GFP_DMA;
-   else if (IS_ENABLED(CONFIG_ZONE_DMA32) &&
-dev->coherent_dma_mask < DMA_BIT_MASK(40))
-   gfp |= __GFP_DMA32;
-
-   gfp |= __GFP_NORETRY;
-
-   ret = swiotlb_alloc_coherent(dev, size, dma_handle, gfp);
mb();
return ret;
 }
 
-static void loongson_dma_free_coherent(struct device *dev, size_t size,
-   void *vaddr, dma_addr_t dma_handle, unsigned long attrs)
-{
-   swiotlb_free_coherent(dev, size, vaddr, dma_handle);
-}
-
 static dma_addr_t loongson_dma_map_page(struct device *dev, struct page *page,
unsigned long offset, size_t size,
enum dma_data_direction dir,
@@ -106,7 +89,7 @@ phys_addr_t __dma_to_phys(struct 

[PATCH 62/67] mips/netlogic: remove swiotlb support

[Christoph Hellwig]

nlm_swiotlb_dma_ops is unused code, so the whole swiotlb support is dead.
If it gets resurrected at some point it should use the generic
swiotlb_dma_ops instead.

Signed-off-by: Christoph Hellwig 
---
 arch/mips/include/asm/netlogic/common.h |  3 --
 arch/mips/netlogic/Kconfig  |  5 --
 arch/mips/netlogic/common/Makefile  |  1 -
 arch/mips/netlogic/common/nlm-dma.c | 94 -
 4 files changed, 103 deletions(-)
 delete mode 100644 arch/mips/netlogic/common/nlm-dma.c

diff --git a/arch/mips/include/asm/netlogic/common.h 
b/arch/mips/include/asm/netlogic/common.h
index a6e6cbebe046..57616649b4f3 100644
--- a/arch/mips/include/asm/netlogic/common.h
+++ b/arch/mips/include/asm/netlogic/common.h
@@ -87,9 +87,6 @@ unsigned int nlm_get_cpu_frequency(void);
 extern const struct plat_smp_ops nlm_smp_ops;
 extern char nlm_reset_entry[], nlm_reset_entry_end[];
 
-/* SWIOTLB */
-extern const struct dma_map_ops nlm_swiotlb_dma_ops;
-
 extern unsigned int nlm_threads_per_core;
 extern cpumask_t nlm_cpumask;
 
diff --git a/arch/mips/netlogic/Kconfig b/arch/mips/netlogic/Kconfig
index 8296b13affd2..7fcfc7fe9f14 100644
--- a/arch/mips/netlogic/Kconfig
+++ b/arch/mips/netlogic/Kconfig
@@ -89,9 +89,4 @@ config IOMMU_HELPER
 config NEED_SG_DMA_LENGTH
bool
 
-config SWIOTLB
-   def_bool y
-   select NEED_SG_DMA_LENGTH
-   select IOMMU_HELPER
-
 endif
diff --git a/arch/mips/netlogic/common/Makefile 
b/arch/mips/netlogic/common/Makefile
index 60d00b5d748e..89f6e3f39fed 100644
--- a/arch/mips/netlogic/common/Makefile
+++ b/arch/mips/netlogic/common/Makefile
@@ -1,6 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-y  += irq.o time.o
-obj-y  += nlm-dma.o
 obj-y  += reset.o
 obj-$(CONFIG_SMP)  += smp.o smpboot.o
 obj-$(CONFIG_EARLY_PRINTK) += earlycons.o
diff --git a/arch/mips/netlogic/common/nlm-dma.c 
b/arch/mips/netlogic/common/nlm-dma.c
deleted file mode 100644
index 49c975b6aa28..
--- a/arch/mips/netlogic/common/nlm-dma.c
+++ /dev/null
@@ -1,94 +0,0 @@
-/*
-*  Copyright (C) 2003-2013 Broadcom Corporation
-*  All Rights Reserved
- *
- * This software is available to you under a choice of one of two
- * licenses.  You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the Broadcom
- * license below:
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- *
- * 1. Redistributions of source code must retain the above copyright
- *notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *notice, this list of conditions and the following disclaimer in
- *the documentation and/or other materials provided with the
- *distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY BROADCOM ``AS IS'' AND ANY EXPRESS OR
- * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
- * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL BROADCOM OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
- * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
- * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
- * IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-
-#include 
-
-static char *nlm_swiotlb;
-
-static void *nlm_dma_alloc_coherent(struct device *dev, size_t size,
-   dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs)
-{
-#ifdef CONFIG_ZONE_DMA32
-   if (dev->coherent_dma_mask <= DMA_BIT_MASK(32))
-   gfp |= __GFP_DMA32;
-#endif
-
-   /* Don't invoke OOM killer */
-   gfp |= __GFP_NORETRY;
-
-   return swiotlb_alloc_coherent(dev, size, dma_handle, gfp);
-}
-
-static void nlm_dma_free_coherent(struct device *dev, size_t size,
-   void *vaddr, dma_addr_t dma_handle, unsigned long attrs)
-{
-   swiotlb_free_coherent(dev, size, vaddr, dma_handle);
-}
-
-const struct dma_map_ops nlm_swiotlb_dma_ops = {
-   .alloc = nlm_dma_alloc_coherent,
-   .free = nlm_dma_free_coherent,
-   .map_page = swiotlb_map_page,
-   .unmap_page = swiotlb_unmap_page,
-   .map_sg = swiotlb_map_sg_attrs,
-   .unmap_sg = swiotlb_unmap_sg_attrs,
-   .sync_single_for_cpu = swiotlb_sync_single_for_cpu,
-   .sync_single_for_device = swiotlb_sync_single_for_device,
-   

[PATCH 61/67] tile: use generic swiotlb_ops

[Christoph Hellwig]

These are identical to the tile ops, and would also support CMA
if enabled on tile.

Signed-off-by: Christoph Hellwig 
---
 arch/tile/Kconfig  |  1 +
 arch/tile/kernel/pci-dma.c | 36 +++-
 2 files changed, 4 insertions(+), 33 deletions(-)

diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
index 30c586686f29..ef9d403cbbe4 100644
--- a/arch/tile/Kconfig
+++ b/arch/tile/Kconfig
@@ -261,6 +261,7 @@ config NEED_SG_DMA_LENGTH
 config SWIOTLB
bool
default TILEGX
+   select DMA_DIRECT_OPS
select IOMMU_HELPER
select NEED_SG_DMA_LENGTH
select ARCH_HAS_DMA_SET_COHERENT_MASK
diff --git a/arch/tile/kernel/pci-dma.c b/arch/tile/kernel/pci-dma.c
index a9b48520eeb9..6e9365234b6a 100644
--- a/arch/tile/kernel/pci-dma.c
+++ b/arch/tile/kernel/pci-dma.c
@@ -511,39 +511,9 @@ EXPORT_SYMBOL(gx_pci_dma_map_ops);
 /* PCI DMA mapping functions for legacy PCI devices */
 
 #ifdef CONFIG_SWIOTLB
-static void *tile_swiotlb_alloc_coherent(struct device *dev, size_t size,
-dma_addr_t *dma_handle, gfp_t gfp,
-unsigned long attrs)
-{
-   gfp |= GFP_DMA32;
-   return swiotlb_alloc_coherent(dev, size, dma_handle, gfp);
-}
-
-static void tile_swiotlb_free_coherent(struct device *dev, size_t size,
-  void *vaddr, dma_addr_t dma_addr,
-  unsigned long attrs)
-{
-   swiotlb_free_coherent(dev, size, vaddr, dma_addr);
-}
-
-static const struct dma_map_ops pci_swiotlb_dma_ops = {
-   .alloc = tile_swiotlb_alloc_coherent,
-   .free = tile_swiotlb_free_coherent,
-   .map_page = swiotlb_map_page,
-   .unmap_page = swiotlb_unmap_page,
-   .map_sg = swiotlb_map_sg_attrs,
-   .unmap_sg = swiotlb_unmap_sg_attrs,
-   .sync_single_for_cpu = swiotlb_sync_single_for_cpu,
-   .sync_single_for_device = swiotlb_sync_single_for_device,
-   .sync_sg_for_cpu = swiotlb_sync_sg_for_cpu,
-   .sync_sg_for_device = swiotlb_sync_sg_for_device,
-   .dma_supported = swiotlb_dma_supported,
-   .mapping_error = swiotlb_dma_mapping_error,
-};
-
 static const struct dma_map_ops pci_hybrid_dma_ops = {
-   .alloc = tile_swiotlb_alloc_coherent,
-   .free = tile_swiotlb_free_coherent,
+   .alloc = swiotlb_alloc,
+   .free = swiotlb_free,
.map_page = tile_pci_dma_map_page,
.unmap_page = tile_pci_dma_unmap_page,
.map_sg = tile_pci_dma_map_sg,
@@ -554,7 +524,7 @@ static const struct dma_map_ops pci_hybrid_dma_ops = {
.sync_sg_for_device = tile_pci_dma_sync_sg_for_device,
 };
 
-const struct dma_map_ops *gx_legacy_pci_dma_map_ops = _swiotlb_dma_ops;
+const struct dma_map_ops *gx_legacy_pci_dma_map_ops = _dma_ops;
 const struct dma_map_ops *gx_hybrid_pci_dma_map_ops = _hybrid_dma_ops;
 #else
 const struct dma_map_ops *gx_legacy_pci_dma_map_ops;
-- 
2.14.2

[PATCH 60/67] tile: replace ZONE_DMA with ZONE_DMA32

[Christoph Hellwig]

tile uses ZONE_DMA for allocations below 32-bits.  These days we
name the zone for that ZONE_DMA32, which will allow to use the
dma-direct and generic swiotlb code as-is, so rename it.

Signed-off-by: Christoph Hellwig 
---
 arch/tile/Kconfig  | 2 +-
 arch/tile/kernel/pci-dma.c | 4 ++--
 arch/tile/kernel/setup.c   | 8 
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
index 02f269cfa538..30c586686f29 100644
--- a/arch/tile/Kconfig
+++ b/arch/tile/Kconfig
@@ -249,7 +249,7 @@ config HIGHMEM
 
  If unsure, say "true".
 
-config ZONE_DMA
+config ZONE_DMA32
def_bool y
 
 config IOMMU_HELPER
diff --git a/arch/tile/kernel/pci-dma.c b/arch/tile/kernel/pci-dma.c
index 9072e2c25e59..a9b48520eeb9 100644
--- a/arch/tile/kernel/pci-dma.c
+++ b/arch/tile/kernel/pci-dma.c
@@ -54,7 +54,7 @@ static void *tile_dma_alloc_coherent(struct device *dev, 
size_t size,
 * which case we will return NULL.  But such devices are uncommon.
 */
if (dma_mask <= DMA_BIT_MASK(32)) {
-   gfp |= GFP_DMA;
+   gfp |= GFP_DMA32;
node = 0;
}
 
@@ -515,7 +515,7 @@ static void *tile_swiotlb_alloc_coherent(struct device 
*dev, size_t size,
 dma_addr_t *dma_handle, gfp_t gfp,
 unsigned long attrs)
 {
-   gfp |= GFP_DMA;
+   gfp |= GFP_DMA32;
return swiotlb_alloc_coherent(dev, size, dma_handle, gfp);
 }
 
diff --git a/arch/tile/kernel/setup.c b/arch/tile/kernel/setup.c
index ad83c1e66dbd..eb4e198f6f93 100644
--- a/arch/tile/kernel/setup.c
+++ b/arch/tile/kernel/setup.c
@@ -814,11 +814,11 @@ static void __init zone_sizes_init(void)
 #endif
 
if (start < dma_end) {
-   zones_size[ZONE_DMA] = min(zones_size[ZONE_NORMAL],
+   zones_size[ZONE_DMA32] = min(zones_size[ZONE_NORMAL],
   dma_end - start);
-   zones_size[ZONE_NORMAL] -= zones_size[ZONE_DMA];
+   zones_size[ZONE_NORMAL] -= zones_size[ZONE_DMA32];
} else {
-   zones_size[ZONE_DMA] = 0;
+   zones_size[ZONE_DMA32] = 0;
}
 
/* Take zone metadata from controller 0 if we're isolnode. */
@@ -830,7 +830,7 @@ static void __init zone_sizes_init(void)
   PFN_UP(node_percpu[i]));
 
/* Track the type of memory on each node */
-   if (zones_size[ZONE_NORMAL] || zones_size[ZONE_DMA])
+   if (zones_size[ZONE_NORMAL] || zones_size[ZONE_DMA32])
node_set_state(i, N_NORMAL_MEMORY);
 #ifdef CONFIG_HIGHMEM
if (end != start)
-- 
2.14.2

[PATCH 59/67] unicore32: use generic swiotlb_ops

[Christoph Hellwig]

These are identical to the unicore32 ops, and would also support CMA
if enabled on unicore32.

Signed-off-by: Christoph Hellwig 
---
 arch/unicore32/include/asm/dma-mapping.h |  9 +-
 arch/unicore32/mm/Kconfig|  1 +
 arch/unicore32/mm/Makefile   |  2 --
 arch/unicore32/mm/dma-swiotlb.c  | 48 
 4 files changed, 2 insertions(+), 58 deletions(-)
 delete mode 100644 arch/unicore32/mm/dma-swiotlb.c

diff --git a/arch/unicore32/include/asm/dma-mapping.h 
b/arch/unicore32/include/asm/dma-mapping.h
index f2bfec273aa7..790bc2ef4af2 100644
--- a/arch/unicore32/include/asm/dma-mapping.h
+++ b/arch/unicore32/include/asm/dma-mapping.h
@@ -12,18 +12,11 @@
 #ifndef __UNICORE_DMA_MAPPING_H__
 #define __UNICORE_DMA_MAPPING_H__
 
-#ifdef __KERNEL__
-
-#include 
-#include 
 #include 
 
-extern const struct dma_map_ops swiotlb_dma_map_ops;
-
 static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
 {
-   return _dma_map_ops;
+   return _dma_ops;
 }
 
-#endif /* __KERNEL__ */
 #endif
diff --git a/arch/unicore32/mm/Kconfig b/arch/unicore32/mm/Kconfig
index c256460cd363..e9154a59d561 100644
--- a/arch/unicore32/mm/Kconfig
+++ b/arch/unicore32/mm/Kconfig
@@ -42,6 +42,7 @@ config CPU_TLB_SINGLE_ENTRY_DISABLE
 
 config SWIOTLB
def_bool y
+   select DMA_DIRECT_OPS
 
 config IOMMU_HELPER
def_bool SWIOTLB
diff --git a/arch/unicore32/mm/Makefile b/arch/unicore32/mm/Makefile
index 681c0ef5ec9e..8106260583ab 100644
--- a/arch/unicore32/mm/Makefile
+++ b/arch/unicore32/mm/Makefile
@@ -6,8 +6,6 @@
 obj-y  := extable.o fault.o init.o pgd.o mmu.o
 obj-y  += flush.o ioremap.o
 
-obj-$(CONFIG_SWIOTLB)  += dma-swiotlb.o
-
 obj-$(CONFIG_MODULES)  += proc-syms.o
 
 obj-$(CONFIG_ALIGNMENT_TRAP)   += alignment.o
diff --git a/arch/unicore32/mm/dma-swiotlb.c b/arch/unicore32/mm/dma-swiotlb.c
deleted file mode 100644
index 525413d6690e..
--- a/arch/unicore32/mm/dma-swiotlb.c
+++ /dev/null
@@ -1,48 +0,0 @@
-/*
- * Contains routines needed to support swiotlb for UniCore32.
- *
- * Copyright (C) 2010 Guan Xuetao
- *
- * This program is free software; you can redistribute  it and/or modify it
- * under  the terms of  the GNU General  Public License as published by the
- * Free Software Foundation;  either version 2 of the  License, or (at your
- * option) any later version.
- */
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-
-#include 
-
-static void *unicore_swiotlb_alloc_coherent(struct device *dev, size_t size,
-   dma_addr_t *dma_handle, gfp_t flags,
-   unsigned long attrs)
-{
-   return swiotlb_alloc_coherent(dev, size, dma_handle, flags);
-}
-
-static void unicore_swiotlb_free_coherent(struct device *dev, size_t size,
- void *vaddr, dma_addr_t dma_addr,
- unsigned long attrs)
-{
-   swiotlb_free_coherent(dev, size, vaddr, dma_addr);
-}
-
-const struct dma_map_ops swiotlb_dma_map_ops = {
-   .alloc = unicore_swiotlb_alloc_coherent,
-   .free = unicore_swiotlb_free_coherent,
-   .map_sg = swiotlb_map_sg_attrs,
-   .unmap_sg = swiotlb_unmap_sg_attrs,
-   .dma_supported = swiotlb_dma_supported,
-   .map_page = swiotlb_map_page,
-   .unmap_page = swiotlb_unmap_page,
-   .sync_single_for_cpu = swiotlb_sync_single_for_cpu,
-   .sync_single_for_device = swiotlb_sync_single_for_device,
-   .sync_sg_for_cpu = swiotlb_sync_sg_for_cpu,
-   .sync_sg_for_device = swiotlb_sync_sg_for_device,
-   .mapping_error = swiotlb_dma_mapping_error,
-};
-EXPORT_SYMBOL(swiotlb_dma_map_ops);
-- 
2.14.2

[PATCH 58/67] ia64: remove an ifdef around the content of pci-dma.c

[Christoph Hellwig]

The file is only compiled if CONFIG_INTEL_IOMMU is set to start with.

Signed-off-by: Christoph Hellwig 
---
 arch/ia64/kernel/pci-dma.c | 7 ---
 1 file changed, 7 deletions(-)

diff --git a/arch/ia64/kernel/pci-dma.c b/arch/ia64/kernel/pci-dma.c
index 35e0cad33b7d..b5df084c0af4 100644
--- a/arch/ia64/kernel/pci-dma.c
+++ b/arch/ia64/kernel/pci-dma.c
@@ -12,12 +12,7 @@
 #include 
 #include 
 #include 
-
-
-#ifdef CONFIG_INTEL_IOMMU
-
 #include 
-
 #include 
 
 dma_addr_t bad_dma_address __read_mostly;
@@ -115,5 +110,3 @@ void __init pci_iommu_alloc(void)
}
 #endif /* CONFIG_SWIOTLB */
 }
-
-#endif
-- 
2.14.2

[PATCH 57/67] ia64: clean up swiotlb support

[Christoph Hellwig]

Move the few remaining bits of swiotlb glue towards their callers,
and remove the pointless on ia64 swiotlb variable.

Signed-off-by: Christoph Hellwig 
---
 arch/ia64/include/asm/dma-mapping.h |  1 -
 arch/ia64/include/asm/swiotlb.h | 18 --
 arch/ia64/kernel/dma-mapping.c  |  9 +
 arch/ia64/kernel/pci-dma.c  | 12 ++--
 arch/ia64/kernel/pci-swiotlb.c  | 36 
 drivers/iommu/intel-iommu.c |  2 +-
 6 files changed, 20 insertions(+), 58 deletions(-)
 delete mode 100644 arch/ia64/include/asm/swiotlb.h
 delete mode 100644 arch/ia64/kernel/pci-swiotlb.c

diff --git a/arch/ia64/include/asm/dma-mapping.h 
b/arch/ia64/include/asm/dma-mapping.h
index eabee56d995c..76e4d6632d68 100644
--- a/arch/ia64/include/asm/dma-mapping.h
+++ b/arch/ia64/include/asm/dma-mapping.h
@@ -8,7 +8,6 @@
  */
 #include 
 #include 
-#include 
 #include 
 
 #define ARCH_HAS_DMA_GET_REQUIRED_MASK
diff --git a/arch/ia64/include/asm/swiotlb.h b/arch/ia64/include/asm/swiotlb.h
deleted file mode 100644
index 841e2c7d0b21..
--- a/arch/ia64/include/asm/swiotlb.h
+++ /dev/null
@@ -1,18 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef ASM_IA64__SWIOTLB_H
-#define ASM_IA64__SWIOTLB_H
-
-#include 
-#include 
-
-#ifdef CONFIG_SWIOTLB
-extern int swiotlb;
-extern void pci_swiotlb_init(void);
-#else
-#define swiotlb 0
-static inline void pci_swiotlb_init(void)
-{
-}
-#endif
-
-#endif /* ASM_IA64__SWIOTLB_H */
diff --git a/arch/ia64/kernel/dma-mapping.c b/arch/ia64/kernel/dma-mapping.c
index 7a82c9259609..f2d57e66fd86 100644
--- a/arch/ia64/kernel/dma-mapping.c
+++ b/arch/ia64/kernel/dma-mapping.c
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 #include 
+#include 
 #include 
 
 /* Set this to 1 if there is a HW IOMMU in the system */
@@ -23,3 +24,11 @@ const struct dma_map_ops *dma_get_ops(struct device *dev)
return dma_ops;
 }
 EXPORT_SYMBOL(dma_get_ops);
+
+#ifdef CONFIG_SWIOTLB
+void __init swiotlb_dma_init(void)
+{
+   dma_ops = _dma_ops;
+   swiotlb_init(1);
+}
+#endif
diff --git a/arch/ia64/kernel/pci-dma.c b/arch/ia64/kernel/pci-dma.c
index 3ba87c22dfbc..35e0cad33b7d 100644
--- a/arch/ia64/kernel/pci-dma.c
+++ b/arch/ia64/kernel/pci-dma.c
@@ -104,8 +104,16 @@ void __init pci_iommu_alloc(void)
detect_intel_iommu();
 
 #ifdef CONFIG_SWIOTLB
-   pci_swiotlb_init();
-#endif
+   if (!iommu_detected) {
+#ifdef CONFIG_IA64_GENERIC
+   printk(KERN_INFO "PCI-DMA: Re-initialize machine vector.\n");
+   machvec_init("dig");
+   swiotlb_dma_init();
+#else
+   panic("Unable to find Intel IOMMU");
+#endif /* CONFIG_IA64_GENERIC */
+   }
+#endif /* CONFIG_SWIOTLB */
 }
 
 #endif
diff --git a/arch/ia64/kernel/pci-swiotlb.c b/arch/ia64/kernel/pci-swiotlb.c
deleted file mode 100644
index 0f8d5fbd86bd..
--- a/arch/ia64/kernel/pci-swiotlb.c
+++ /dev/null
@@ -1,36 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/* Glue code to lib/swiotlb.c */
-
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-
-int swiotlb __read_mostly;
-EXPORT_SYMBOL(swiotlb);
-
-void __init swiotlb_dma_init(void)
-{
-   dma_ops = _dma_ops;
-   swiotlb_init(1);
-}
-
-void __init pci_swiotlb_init(void)
-{
-   if (!iommu_detected) {
-#ifdef CONFIG_IA64_GENERIC
-   swiotlb = 1;
-   printk(KERN_INFO "PCI-DMA: Re-initialize machine vector.\n");
-   machvec_init("dig");
-   swiotlb_init(1);
-   dma_ops = _dma_ops;
-#else
-   panic("Unable to find Intel IOMMU");
-#endif
-   }
-}
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 6c9df0773b78..569a9328e53e 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -4807,7 +4807,7 @@ int __init intel_iommu_init(void)
up_write(_global_lock);
pr_info("Intel(R) Virtualization Technology for Directed I/O\n");
 
-#ifdef CONFIG_SWIOTLB
+#if defined(CONFIG_X86) && defined(CONFIG_SWIOTLB)
swiotlb = 0;
 #endif
dma_ops = _dma_ops;
-- 
2.14.2

[PATCH 56/67] ia64: use generic swiotlb_ops

[Christoph Hellwig]

These are identical to the ia64 ops, and would also support CMA
if enabled on ia64.

Signed-off-by: Christoph Hellwig 
---
 arch/ia64/Kconfig|  5 +
 arch/ia64/hp/common/hwsw_iommu.c |  4 ++--
 arch/ia64/hp/common/sba_iommu.c  |  6 +++---
 arch/ia64/kernel/pci-swiotlb.c   | 38 +++---
 4 files changed, 13 insertions(+), 40 deletions(-)

diff --git a/arch/ia64/Kconfig b/arch/ia64/Kconfig
index 888acdb163cb..29148fe4bf5a 100644
--- a/arch/ia64/Kconfig
+++ b/arch/ia64/Kconfig
@@ -146,6 +146,7 @@ config IA64_GENERIC
bool "generic"
select NUMA
select ACPI_NUMA
+   select DMA_DIRECT_OPS
select SWIOTLB
select PCI_MSI
help
@@ -166,6 +167,7 @@ config IA64_GENERIC
 
 config IA64_DIG
bool "DIG-compliant"
+   select DMA_DIRECT_OPS
select SWIOTLB
 
 config IA64_DIG_VTD
@@ -181,6 +183,7 @@ config IA64_HP_ZX1
 
 config IA64_HP_ZX1_SWIOTLB
bool "HP-zx1/sx1000 with software I/O TLB"
+   select DMA_DIRECT_OPS
select SWIOTLB
help
  Build a kernel that runs on HP zx1 and sx1000 systems even when they
@@ -204,6 +207,7 @@ config IA64_SGI_UV
bool "SGI-UV"
select NUMA
select ACPI_NUMA
+   select DMA_DIRECT_OPS
select SWIOTLB
help
  Selecting this option will optimize the kernel for use on UV based
@@ -214,6 +218,7 @@ config IA64_SGI_UV
 
 config IA64_HP_SIM
bool "Ski-simulator"
+   select DMA_DIRECT_OPS
select SWIOTLB
depends on !PM
 
diff --git a/arch/ia64/hp/common/hwsw_iommu.c b/arch/ia64/hp/common/hwsw_iommu.c
index 41279f0442bd..58969039bed2 100644
--- a/arch/ia64/hp/common/hwsw_iommu.c
+++ b/arch/ia64/hp/common/hwsw_iommu.c
@@ -19,7 +19,7 @@
 #include 
 #include 
 
-extern const struct dma_map_ops sba_dma_ops, ia64_swiotlb_dma_ops;
+extern const struct dma_map_ops sba_dma_ops;
 
 /* swiotlb declarations & definitions: */
 extern int swiotlb_late_init_with_default_size (size_t size);
@@ -38,7 +38,7 @@ static inline int use_swiotlb(struct device *dev)
 const struct dma_map_ops *hwsw_dma_get_ops(struct device *dev)
 {
if (use_swiotlb(dev))
-   return _swiotlb_dma_ops;
+   return _dma_ops;
return _dma_ops;
 }
 EXPORT_SYMBOL(hwsw_dma_get_ops);
diff --git a/arch/ia64/hp/common/sba_iommu.c b/arch/ia64/hp/common/sba_iommu.c
index d68849ad2ee1..6f05aba9012f 100644
--- a/arch/ia64/hp/common/sba_iommu.c
+++ b/arch/ia64/hp/common/sba_iommu.c
@@ -2093,7 +2093,7 @@ static int __init acpi_sba_ioc_init_acpi(void)
 /* This has to run before acpi_scan_init(). */
 arch_initcall(acpi_sba_ioc_init_acpi);
 
-extern const struct dma_map_ops ia64_swiotlb_dma_ops;
+extern const struct dma_map_ops swiotlb_dma_ops;
 
 static int __init
 sba_init(void)
@@ -2108,7 +2108,7 @@ sba_init(void)
 * a successful kdump kernel boot is to use the swiotlb.
 */
if (is_kdump_kernel()) {
-   dma_ops = _swiotlb_dma_ops;
+   dma_ops = _dma_ops;
if (swiotlb_late_init_with_default_size(64 * (1<<20)) != 0)
panic("Unable to initialize software I/O TLB:"
  " Try machvec=dig boot option");
@@ -2130,7 +2130,7 @@ sba_init(void)
 * If we didn't find something sba_iommu can claim, we
 * need to setup the swiotlb and switch to the dig machvec.
 */
-   dma_ops = _swiotlb_dma_ops;
+   dma_ops = _dma_ops;
if (swiotlb_late_init_with_default_size(64 * (1<<20)) != 0)
panic("Unable to find SBA IOMMU or initialize "
  "software I/O TLB: Try machvec=dig boot option");
diff --git a/arch/ia64/kernel/pci-swiotlb.c b/arch/ia64/kernel/pci-swiotlb.c
index 4a9a6e58ad6a..0f8d5fbd86bd 100644
--- a/arch/ia64/kernel/pci-swiotlb.c
+++ b/arch/ia64/kernel/pci-swiotlb.c
@@ -6,8 +6,7 @@
 #include 
 #include 
 #include 
-
-#include 
+#include 
 #include 
 #include 
 #include 
@@ -15,40 +14,9 @@
 int swiotlb __read_mostly;
 EXPORT_SYMBOL(swiotlb);
 
-static void *ia64_swiotlb_alloc_coherent(struct device *dev, size_t size,
-dma_addr_t *dma_handle, gfp_t gfp,
-unsigned long attrs)
-{
-   if (dev->coherent_dma_mask != DMA_BIT_MASK(64))
-   gfp |= GFP_DMA32;
-   return swiotlb_alloc_coherent(dev, size, dma_handle, gfp);
-}
-
-static void ia64_swiotlb_free_coherent(struct device *dev, size_t size,
-  void *vaddr, dma_addr_t dma_addr,
-  unsigned long attrs)
-{
-   swiotlb_free_coherent(dev, size, vaddr, dma_addr);
-}
-
-const struct dma_map_ops ia64_swiotlb_dma_ops = {
-   .alloc = ia64_swiotlb_alloc_coherent,
-   .free = ia64_swiotlb_free_coherent,
-   .map_page = swiotlb_map_page,

[PATCH 55/67] ia64: replace ZONE_DMA with ZONE_DMA32

[Christoph Hellwig]

ia64 uses ZONE_DMA for allocations below 32-bits.  These days we
name the zone for that ZONE_DMA32, which will allow to use the
dma-direct and generic swiotlb code as-is, so rename it.

Signed-off-by: Christoph Hellwig 
---
 arch/ia64/Kconfig  | 2 +-
 arch/ia64/kernel/pci-swiotlb.c | 2 +-
 arch/ia64/mm/contig.c  | 4 ++--
 arch/ia64/mm/discontig.c   | 8 
 4 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/arch/ia64/Kconfig b/arch/ia64/Kconfig
index 4d18fca885ee..888acdb163cb 100644
--- a/arch/ia64/Kconfig
+++ b/arch/ia64/Kconfig
@@ -66,7 +66,7 @@ config 64BIT
select ATA_NONSTANDARD if ATA
default y
 
-config ZONE_DMA
+config ZONE_DMA32
def_bool y
depends on !IA64_SGI_SN2
 
diff --git a/arch/ia64/kernel/pci-swiotlb.c b/arch/ia64/kernel/pci-swiotlb.c
index f1ae873a8c35..4a9a6e58ad6a 100644
--- a/arch/ia64/kernel/pci-swiotlb.c
+++ b/arch/ia64/kernel/pci-swiotlb.c
@@ -20,7 +20,7 @@ static void *ia64_swiotlb_alloc_coherent(struct device *dev, 
size_t size,
 unsigned long attrs)
 {
if (dev->coherent_dma_mask != DMA_BIT_MASK(64))
-   gfp |= GFP_DMA;
+   gfp |= GFP_DMA32;
return swiotlb_alloc_coherent(dev, size, dma_handle, gfp);
 }
 
diff --git a/arch/ia64/mm/contig.c b/arch/ia64/mm/contig.c
index 52715a71aede..7d64b30913d1 100644
--- a/arch/ia64/mm/contig.c
+++ b/arch/ia64/mm/contig.c
@@ -237,9 +237,9 @@ paging_init (void)
unsigned long max_zone_pfns[MAX_NR_ZONES];
 
memset(max_zone_pfns, 0, sizeof(max_zone_pfns));
-#ifdef CONFIG_ZONE_DMA
+#ifdef CONFIG_ZONE_DMA32
max_dma = virt_to_phys((void *) MAX_DMA_ADDRESS) >> PAGE_SHIFT;
-   max_zone_pfns[ZONE_DMA] = max_dma;
+   max_zone_pfns[ZONE_DMA32] = max_dma;
 #endif
max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
 
diff --git a/arch/ia64/mm/discontig.c b/arch/ia64/mm/discontig.c
index 9b2d994cddf6..ac46f0d60b66 100644
--- a/arch/ia64/mm/discontig.c
+++ b/arch/ia64/mm/discontig.c
@@ -38,7 +38,7 @@ struct early_node_data {
struct ia64_node_data *node_data;
unsigned long pernode_addr;
unsigned long pernode_size;
-#ifdef CONFIG_ZONE_DMA
+#ifdef CONFIG_ZONE_DMA32
unsigned long num_dma_physpages;
 #endif
unsigned long min_pfn;
@@ -669,7 +669,7 @@ static __init int count_node_pages(unsigned long start, 
unsigned long len, int n
 {
unsigned long end = start + len;
 
-#ifdef CONFIG_ZONE_DMA
+#ifdef CONFIG_ZONE_DMA32
if (start <= __pa(MAX_DMA_ADDRESS))
mem_data[node].num_dma_physpages +=
(min(end, __pa(MAX_DMA_ADDRESS)) - start) >>PAGE_SHIFT;
@@ -724,8 +724,8 @@ void __init paging_init(void)
}
 
memset(max_zone_pfns, 0, sizeof(max_zone_pfns));
-#ifdef CONFIG_ZONE_DMA
-   max_zone_pfns[ZONE_DMA] = max_dma;
+#ifdef CONFIG_ZONE_DMA32
+   max_zone_pfns[ZONE_DMA32] = max_dma;
 #endif
max_zone_pfns[ZONE_NORMAL] = max_pfn;
free_area_init_nodes(max_zone_pfns);
-- 
2.14.2

[PATCH 54/67] x86: remove sta2x11_dma_ops

[Christoph Hellwig]

Both the swiotlb and the dma-direct code already call into phys_to_dma
to translate the DMA address.  So the sta2x11 into phys_to_dma and
dma_to_phys are enough to handle this "special" device, and we can use
the plain old swiotlb ops.

Signed-off-by: Christoph Hellwig 
---
 arch/x86/include/asm/device.h |  3 +++
 arch/x86/pci/sta2x11-fixup.c  | 46 +--
 2 files changed, 8 insertions(+), 41 deletions(-)

diff --git a/arch/x86/include/asm/device.h b/arch/x86/include/asm/device.h
index 5e12c63b47aa..812bd6c5d602 100644
--- a/arch/x86/include/asm/device.h
+++ b/arch/x86/include/asm/device.h
@@ -6,6 +6,9 @@ struct dev_archdata {
 #if defined(CONFIG_INTEL_IOMMU) || defined(CONFIG_AMD_IOMMU)
void *iommu; /* hook for IOMMU specific extension */
 #endif
+#ifdef CONFIG_STA2X11
+   bool is_sta2x11 : 1;
+#endif
 };
 
 #if defined(CONFIG_X86_DEV_DMA_OPS) && defined(CONFIG_PCI_DOMAINS)
diff --git a/arch/x86/pci/sta2x11-fixup.c b/arch/x86/pci/sta2x11-fixup.c
index 15ad3025e439..7a5bafb76d77 100644
--- a/arch/x86/pci/sta2x11-fixup.c
+++ b/arch/x86/pci/sta2x11-fixup.c
@@ -159,43 +159,6 @@ static dma_addr_t a2p(dma_addr_t a, struct pci_dev *pdev)
return p;
 }
 
-/**
- * sta2x11_swiotlb_alloc_coherent - Allocate swiotlb bounce buffers
- * returns virtual address. This is the only "special" function here.
- * @dev: PCI device
- * @size: Size of the buffer
- * @dma_handle: DMA address
- * @flags: memory flags
- */
-static void *sta2x11_swiotlb_alloc_coherent(struct device *dev,
-   size_t size,
-   dma_addr_t *dma_handle,
-   gfp_t flags,
-   unsigned long attrs)
-{
-   void *vaddr;
-
-   vaddr = swiotlb_alloc(dev, size, dma_handle, flags, attrs);
-   *dma_handle = p2a(*dma_handle, to_pci_dev(dev));
-   return vaddr;
-}
-
-/* We have our own dma_ops: the same as swiotlb but from alloc (above) */
-static const struct dma_map_ops sta2x11_dma_ops = {
-   .alloc = sta2x11_swiotlb_alloc_coherent,
-   .free = swiotlb_free,
-   .map_page = swiotlb_map_page,
-   .unmap_page = swiotlb_unmap_page,
-   .map_sg = swiotlb_map_sg_attrs,
-   .unmap_sg = swiotlb_unmap_sg_attrs,
-   .sync_single_for_cpu = swiotlb_sync_single_for_cpu,
-   .sync_single_for_device = swiotlb_sync_single_for_device,
-   .sync_sg_for_cpu = swiotlb_sync_sg_for_cpu,
-   .sync_sg_for_device = swiotlb_sync_sg_for_device,
-   .mapping_error = swiotlb_dma_mapping_error,
-   .dma_supported = dma_direct_supported,
-};
-
 /* At setup time, we use our own ops if the device is a ConneXt one */
 static void sta2x11_setup_pdev(struct pci_dev *pdev)
 {
@@ -205,7 +168,8 @@ static void sta2x11_setup_pdev(struct pci_dev *pdev)
return;
pci_set_consistent_dma_mask(pdev, STA2X11_AMBA_SIZE - 1);
pci_set_dma_mask(pdev, STA2X11_AMBA_SIZE - 1);
-   pdev->dev.dma_ops = _dma_ops;
+   pdev->dev.dma_ops = _dma_ops;
+   pdev->dev.archdata.is_sta2x11 = true;
 
/* We must enable all devices as master, for audio DMA to work */
pci_set_master(pdev);
@@ -225,7 +189,7 @@ bool dma_capable(struct device *dev, dma_addr_t addr, 
size_t size)
 {
struct sta2x11_mapping *map;
 
-   if (dev->dma_ops != _dma_ops) {
+   if (!dev->archdata.is_sta2x11) {
if (!dev->dma_mask)
return false;
return addr + size - 1 <= *dev->dma_mask;
@@ -249,7 +213,7 @@ bool dma_capable(struct device *dev, dma_addr_t addr, 
size_t size)
  */
 dma_addr_t __phys_to_dma(struct device *dev, phys_addr_t paddr)
 {
-   if (dev->dma_ops != _dma_ops)
+   if (!dev->archdata.is_sta2x11)
return paddr;
return p2a(paddr, to_pci_dev(dev));
 }
@@ -261,7 +225,7 @@ dma_addr_t __phys_to_dma(struct device *dev, phys_addr_t 
paddr)
  */
 phys_addr_t __dma_to_phys(struct device *dev, dma_addr_t daddr)
 {
-   if (dev->dma_ops != _dma_ops)
+   if (!dev->archdata.is_sta2x11)
return daddr;
return a2p(daddr, to_pci_dev(dev));
 }
-- 
2.14.2

[PATCH 53/67] swiotlb: remove swiotlb_set_mem_attributes

[Christoph Hellwig]

Now that set_memory_decrypted is always available we can just call
it directly.

Signed-off-by: Christoph Hellwig 
---
 arch/x86/include/asm/mem_encrypt.h |  2 --
 arch/x86/mm/mem_encrypt.c  |  9 -
 lib/swiotlb.c  | 12 ++--
 3 files changed, 6 insertions(+), 17 deletions(-)

diff --git a/arch/x86/include/asm/mem_encrypt.h 
b/arch/x86/include/asm/mem_encrypt.h
index c9459a4c3c68..549894d496da 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -48,8 +48,6 @@ int __init early_set_memory_encrypted(unsigned long vaddr, 
unsigned long size);
 /* Architecture __weak replacement functions */
 void __init mem_encrypt_init(void);
 
-void swiotlb_set_mem_attributes(void *vaddr, unsigned long size);
-
 bool sme_active(void);
 bool sev_active(void);
 
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index 93de36cc3dd9..b279e90c85cd 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -379,15 +379,6 @@ void __init mem_encrypt_init(void)
 : "Secure Memory Encryption (SME)");
 }
 
-void swiotlb_set_mem_attributes(void *vaddr, unsigned long size)
-{
-   WARN(PAGE_ALIGN(size) != size,
-"size is not page-aligned (%#lx)\n", size);
-
-   /* Make the SWIOTLB buffer area decrypted */
-   set_memory_decrypted((unsigned long)vaddr, size >> PAGE_SHIFT);
-}
-
 static void __init sme_clear_pgd(pgd_t *pgd_base, unsigned long start,
 unsigned long end)
 {
diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 85b2ad9299e3..4ea0b5710618 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -31,6 +31,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 #include 
@@ -156,8 +157,6 @@ unsigned long swiotlb_size_or_default(void)
return size ? size : (IO_TLB_DEFAULT_SIZE);
 }
 
-void __weak swiotlb_set_mem_attributes(void *vaddr, unsigned long size) { }
-
 /* Note that this doesn't work with highmem page */
 static dma_addr_t swiotlb_virt_to_bus(struct device *hwdev,
  volatile void *address)
@@ -202,12 +201,12 @@ void __init swiotlb_update_mem_attributes(void)
 
vaddr = phys_to_virt(io_tlb_start);
bytes = PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT);
-   swiotlb_set_mem_attributes(vaddr, bytes);
+   set_memory_decrypted((unsigned long)vaddr, bytes >> PAGE_SHIFT);
memset(vaddr, 0, bytes);
 
vaddr = phys_to_virt(io_tlb_overflow_buffer);
bytes = PAGE_ALIGN(io_tlb_overflow);
-   swiotlb_set_mem_attributes(vaddr, bytes);
+   set_memory_decrypted((unsigned long)vaddr, bytes >> PAGE_SHIFT);
memset(vaddr, 0, bytes);
 }
 
@@ -348,7 +347,7 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs)
io_tlb_start = virt_to_phys(tlb);
io_tlb_end = io_tlb_start + bytes;
 
-   swiotlb_set_mem_attributes(tlb, bytes);
+   set_memory_decrypted((unsigned long)tlb, bytes >> PAGE_SHIFT);
memset(tlb, 0, bytes);
 
/*
@@ -359,7 +358,8 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs)
if (!v_overflow_buffer)
goto cleanup2;
 
-   swiotlb_set_mem_attributes(v_overflow_buffer, io_tlb_overflow);
+   set_memory_decrypted((unsigned long)v_overflow_buffer,
+   io_tlb_overflow >> PAGE_SHIFT);
memset(v_overflow_buffer, 0, io_tlb_overflow);
io_tlb_overflow_buffer = virt_to_phys(v_overflow_buffer);
 
-- 
2.14.2

[PATCH 52/67] dma-direct: handle the memory encryption bit in common code

[Christoph Hellwig]

Gіve the basic phys_to_dma and dma_to_phys helpers a __-prefix and add
the memory encryption mask to the non-prefixed versions.  Use the
__-prefixed versions directly instead of clearing the mask again in
various places.

With that in place the generic dma-direct routines can be used to
allocate non-encrypted bounce buffers, and the x86 SEV case can use
the generic swiotlb ops.

Signed-off-by: Christoph Hellwig 
---
 arch/arm/include/asm/dma-direct.h  |  4 +-
 arch/mips/cavium-octeon/dma-octeon.c   | 10 +--
 .../include/asm/mach-cavium-octeon/dma-coherence.h |  4 +-
 .../include/asm/mach-loongson64/dma-coherence.h| 10 +--
 arch/mips/loongson64/common/dma-swiotlb.c  |  4 +-
 arch/powerpc/include/asm/dma-direct.h  |  4 +-
 arch/x86/Kconfig   |  2 +-
 arch/x86/include/asm/dma-direct.h  | 25 +---
 arch/x86/mm/mem_encrypt.c  | 73 +-
 arch/x86/pci/sta2x11-fixup.c   |  6 +-
 include/linux/dma-direct.h | 23 ++-
 lib/dma-direct.c   | 24 +--
 lib/swiotlb.c  | 25 +++-
 13 files changed, 76 insertions(+), 138 deletions(-)

diff --git a/arch/arm/include/asm/dma-direct.h 
b/arch/arm/include/asm/dma-direct.h
index 5b0a8a421894..b67e5fc1fe43 100644
--- a/arch/arm/include/asm/dma-direct.h
+++ b/arch/arm/include/asm/dma-direct.h
@@ -2,13 +2,13 @@
 #ifndef ASM_ARM_DMA_DIRECT_H
 #define ASM_ARM_DMA_DIRECT_H 1
 
-static inline dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr)
+static inline dma_addr_t __phys_to_dma(struct device *dev, phys_addr_t paddr)
 {
unsigned int offset = paddr & ~PAGE_MASK;
return pfn_to_dma(dev, __phys_to_pfn(paddr)) + offset;
 }
 
-static inline phys_addr_t dma_to_phys(struct device *dev, dma_addr_t dev_addr)
+static inline phys_addr_t __dma_to_phys(struct device *dev, dma_addr_t 
dev_addr)
 {
unsigned int offset = dev_addr & ~PAGE_MASK;
return __pfn_to_phys(dma_to_pfn(dev, dev_addr)) + offset;
diff --git a/arch/mips/cavium-octeon/dma-octeon.c 
b/arch/mips/cavium-octeon/dma-octeon.c
index 5baf79fce643..6440ad3f9e3b 100644
--- a/arch/mips/cavium-octeon/dma-octeon.c
+++ b/arch/mips/cavium-octeon/dma-octeon.c
@@ -10,7 +10,7 @@
  * IP32 changes by Ilya.
  * Copyright (C) 2010 Cavium Networks, Inc.
  */
-#include 
+#include 
 #include 
 #include 
 #include 
@@ -202,7 +202,7 @@ struct octeon_dma_map_ops {
phys_addr_t (*dma_to_phys)(struct device *dev, dma_addr_t daddr);
 };
 
-dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr)
+dma_addr_t __phys_to_dma(struct device *dev, phys_addr_t paddr)
 {
struct octeon_dma_map_ops *ops = container_of(get_dma_ops(dev),
  struct octeon_dma_map_ops,
@@ -210,9 +210,9 @@ dma_addr_t phys_to_dma(struct device *dev, phys_addr_t 
paddr)
 
return ops->phys_to_dma(dev, paddr);
 }
-EXPORT_SYMBOL(phys_to_dma);
+EXPORT_SYMBOL(__phys_to_dma);
 
-phys_addr_t dma_to_phys(struct device *dev, dma_addr_t daddr)
+phys_addr_t __dma_to_phys(struct device *dev, dma_addr_t daddr)
 {
struct octeon_dma_map_ops *ops = container_of(get_dma_ops(dev),
  struct octeon_dma_map_ops,
@@ -220,7 +220,7 @@ phys_addr_t dma_to_phys(struct device *dev, dma_addr_t 
daddr)
 
return ops->dma_to_phys(dev, daddr);
 }
-EXPORT_SYMBOL(dma_to_phys);
+EXPORT_SYMBOL(__dma_to_phys);
 
 static struct octeon_dma_map_ops octeon_linear_dma_map_ops = {
.dma_map_ops = {
diff --git a/arch/mips/include/asm/mach-cavium-octeon/dma-coherence.h 
b/arch/mips/include/asm/mach-cavium-octeon/dma-coherence.h
index f00833acb626..6f8e024f4f97 100644
--- a/arch/mips/include/asm/mach-cavium-octeon/dma-coherence.h
+++ b/arch/mips/include/asm/mach-cavium-octeon/dma-coherence.h
@@ -69,8 +69,8 @@ static inline bool dma_capable(struct device *dev, dma_addr_t 
addr, size_t size)
return addr + size <= *dev->dma_mask;
 }
 
-dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr);
-phys_addr_t dma_to_phys(struct device *dev, dma_addr_t daddr);
+dma_addr_t __phys_to_dma(struct device *dev, phys_addr_t paddr);
+phys_addr_t __dma_to_phys(struct device *dev, dma_addr_t daddr);
 
 struct dma_map_ops;
 extern const struct dma_map_ops *octeon_pci_dma_map_ops;
diff --git a/arch/mips/include/asm/mach-loongson64/dma-coherence.h 
b/arch/mips/include/asm/mach-loongson64/dma-coherence.h
index 5cfda8f893e9..94fd224dddee 100644
--- a/arch/mips/include/asm/mach-loongson64/dma-coherence.h
+++ b/arch/mips/include/asm/mach-loongson64/dma-coherence.h
@@ -25,13 +25,13 @@ static inline bool dma_capable(struct device *dev, 
dma_addr_t addr, size_t size)
return addr + size <= *dev->dma_mask;
 }
 
-extern dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr);

[PATCH 51/67] set_memory.h: provide set_memory_{en,de}crypted stubs

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 include/linux/set_memory.h | 12 
 1 file changed, 12 insertions(+)

diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h
index e5140648f638..da5178216da5 100644
--- a/include/linux/set_memory.h
+++ b/include/linux/set_memory.h
@@ -17,4 +17,16 @@ static inline int set_memory_x(unsigned long addr,  int 
numpages) { return 0; }
 static inline int set_memory_nx(unsigned long addr, int numpages) { return 0; }
 #endif
 
+#ifndef CONFIG_ARCH_HAS_MEM_ENCRYPT
+static inline int set_memory_encrypted(unsigned long addr, int numpages)
+{
+   return 0;
+}
+
+static inline int set_memory_decrypted(unsigned long addr, int numpages)
+{
+   return 0;
+}
+#endif /* CONFIG_ARCH_HAS_MEM_ENCRYPT */
+
 #endif /* _LINUX_SET_MEMORY_H_ */
-- 
2.14.2

[PATCH 50/67] swiotlb: refactor coherent buffer allocation

[Christoph Hellwig]

Factor out a new swiotlb_alloc_buffer helper that allocates DMA coherent
memory from the swiotlb bounce buffer.

This allows to simplify the swiotlb_alloc implemenation that uses
dma_direct_alloc to try to allocate a reachable buffer first.

Signed-off-by: Christoph Hellwig 
---
 lib/swiotlb.c | 100 ++
 1 file changed, 51 insertions(+), 49 deletions(-)

diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index adb4dd0091fa..905eea6353a3 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -709,67 +709,69 @@ void swiotlb_tbl_sync_single(struct device *hwdev, 
phys_addr_t tlb_addr,
 }
 EXPORT_SYMBOL_GPL(swiotlb_tbl_sync_single);
 
+static void *
+swiotlb_alloc_buffer(struct device *dev, size_t size, dma_addr_t *dma_handle)
+{
+   phys_addr_t phys_addr;
+
+   if (swiotlb_force == SWIOTLB_NO_FORCE)
+   goto out_warn;
+
+   phys_addr = swiotlb_tbl_map_single(dev,
+   swiotlb_phys_to_dma(dev, io_tlb_start),
+   0, size, DMA_FROM_DEVICE, 0);
+   if (phys_addr == SWIOTLB_MAP_ERROR)
+   goto out_warn;
+
+   *dma_handle = swiotlb_phys_to_dma(dev, phys_addr);
+
+   /* Confirm address can be DMA'd by device */
+   if (*dma_handle + size - 1 > dev->coherent_dma_mask)
+   goto out_unmap;
+
+   memset(phys_to_virt(phys_addr), 0, size);
+   return phys_to_virt(phys_addr);
+
+out_unmap:
+   dev_warn(dev, "hwdev DMA mask = 0x%016Lx, dev_addr = 0x%016Lx\n",
+   (unsigned long long)dev->coherent_dma_mask,
+   (unsigned long long)*dma_handle);
+
+   /*
+* DMA_TO_DEVICE to avoid memcpy in unmap_single.
+* DMA_ATTR_SKIP_CPU_SYNC is optional.
+*/
+   swiotlb_tbl_unmap_single(dev, phys_addr, size, DMA_TO_DEVICE,
+   DMA_ATTR_SKIP_CPU_SYNC);
+out_warn:
+   dev_warn(dev,
+   "swiotlb: coherent allocation failed, size=%zu\n", size);
+   dump_stack();
+   return NULL;
+}
+
 void *
 swiotlb_alloc_coherent(struct device *hwdev, size_t size,
   dma_addr_t *dma_handle, gfp_t flags)
 {
-   dma_addr_t dev_addr;
-   void *ret;
int order = get_order(size);
+   void *ret;
 
ret = (void *)__get_free_pages(flags, order);
if (ret) {
-   dev_addr = swiotlb_virt_to_bus(hwdev, ret);
-   if (dev_addr + size - 1 > hwdev->coherent_dma_mask) {
-   /*
-* The allocated memory isn't reachable by the device.
-*/
-   free_pages((unsigned long) ret, order);
-   ret = NULL;
+   *dma_handle = swiotlb_virt_to_bus(hwdev, ret);
+   if (*dma_handle  + size - 1 <= hwdev->coherent_dma_mask) {
+   memset(ret, 0, size);
+   return ret;
}
-   }
-   if (!ret) {
+
/*
-* We are either out of memory or the device can't DMA to
-* GFP_DMA memory; fall back on map_single(), which
-* will grab memory from the lowest available address range.
+* The allocated memory isn't reachable by the device.
 */
-   phys_addr_t paddr = map_single(hwdev, 0, size,
-  DMA_FROM_DEVICE, 0);
-   if (paddr == SWIOTLB_MAP_ERROR)
-   goto err_warn;
-
-   ret = phys_to_virt(paddr);
-   dev_addr = swiotlb_phys_to_dma(hwdev, paddr);
-
-   /* Confirm address can be DMA'd by device */
-   if (dev_addr + size - 1 > hwdev->coherent_dma_mask) {
-   printk("hwdev DMA mask = 0x%016Lx, dev_addr = 
0x%016Lx\n",
-  (unsigned long long)hwdev->coherent_dma_mask,
-  (unsigned long long)dev_addr);
-
-   /*
-* DMA_TO_DEVICE to avoid memcpy in unmap_single.
-* The DMA_ATTR_SKIP_CPU_SYNC is optional.
-*/
-   swiotlb_tbl_unmap_single(hwdev, paddr,
-size, DMA_TO_DEVICE,
-DMA_ATTR_SKIP_CPU_SYNC);
-   goto err_warn;
-   }
+   free_pages((unsigned long) ret, order);
}
 
-   *dma_handle = dev_addr;
-   memset(ret, 0, size);
-
-   return ret;
-
-err_warn:
-   pr_warn("swiotlb: coherent allocation failed for device %s size=%zu\n",
-   dev_name(hwdev), size);
-   dump_stack();
-
-   return NULL;
+   return swiotlb_alloc_buffer(hwdev, size, dma_handle);
 }
 EXPORT_SYMBOL(swiotlb_alloc_coherent);
 
@@ -1105,7 +1107,7 @@ void *swiotlb_alloc(struct device *dev, size_t size, 
dma_addr_t 

[PATCH 49/67] swiotlb: refactor coherent buffer freeing

[Christoph Hellwig]

Factor out a new swiotlb_free_buffer helper that checks if an address
is allocated from the swiotlb bounce buffer, and if yes frees it.

This allows to simplify the swiotlb_free implemenation that uses
dma_direct_free to free the non-bounce buffer allocations.

Signed-off-by: Christoph Hellwig 
---
 lib/swiotlb.c | 35 +--
 1 file changed, 21 insertions(+), 14 deletions(-)

diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index a14fff30ee9d..adb4dd0091fa 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -773,22 +773,31 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,
 }
 EXPORT_SYMBOL(swiotlb_alloc_coherent);
 
+static bool swiotlb_free_buffer(struct device *dev, size_t size,
+   dma_addr_t dma_addr)
+{
+   phys_addr_t phys_addr = dma_to_phys(dev, dma_addr);
+
+   WARN_ON_ONCE(irqs_disabled());
+
+   if (!is_swiotlb_buffer(phys_addr))
+   return false;
+
+   /*
+* DMA_TO_DEVICE to avoid memcpy in swiotlb_tbl_unmap_single.
+* DMA_ATTR_SKIP_CPU_SYNC is optional.
+*/
+   swiotlb_tbl_unmap_single(dev, phys_addr, size, DMA_TO_DEVICE,
+DMA_ATTR_SKIP_CPU_SYNC);
+   return true;
+}
+
 void
 swiotlb_free_coherent(struct device *hwdev, size_t size, void *vaddr,
  dma_addr_t dev_addr)
 {
-   phys_addr_t paddr = dma_to_phys(hwdev, dev_addr);
-
-   WARN_ON(irqs_disabled());
-   if (!is_swiotlb_buffer(paddr))
+   if (!swiotlb_free_buffer(hwdev, size, dev_addr))
free_pages((unsigned long)vaddr, get_order(size));
-   else
-   /*
-* DMA_TO_DEVICE to avoid memcpy in swiotlb_tbl_unmap_single.
-* DMA_ATTR_SKIP_CPU_SYNC is optional.
-*/
-   swiotlb_tbl_unmap_single(hwdev, paddr, size, DMA_TO_DEVICE,
-DMA_ATTR_SKIP_CPU_SYNC);
 }
 EXPORT_SYMBOL(swiotlb_free_coherent);
 
@@ -1103,9 +1112,7 @@ void *swiotlb_alloc(struct device *dev, size_t size, 
dma_addr_t *dma_handle,
 void swiotlb_free(struct device *dev, size_t size, void *vaddr,
dma_addr_t dma_addr, unsigned long attrs)
 {
-   if (is_swiotlb_buffer(dma_to_phys(dev, dma_addr)))
-   swiotlb_free_coherent(dev, size, vaddr, dma_addr);
-   else
+   if (!swiotlb_free_buffer(dev, size, dma_addr))
dma_direct_free(dev, size, vaddr, dma_addr, attrs);
 }
 
-- 
2.14.2

[PATCH 48/67] swiotlb: rely on dev->coherent_dma_mask

[Christoph Hellwig]

These days the coherent DMA mask is always set, so don't work around the
lack of it.

Signed-off-by: Christoph Hellwig 
---
 lib/swiotlb.c | 10 +++---
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index e0b8980334c3..a14fff30ee9d 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -716,15 +716,11 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,
dma_addr_t dev_addr;
void *ret;
int order = get_order(size);
-   u64 dma_mask = DMA_BIT_MASK(32);
-
-   if (hwdev && hwdev->coherent_dma_mask)
-   dma_mask = hwdev->coherent_dma_mask;
 
ret = (void *)__get_free_pages(flags, order);
if (ret) {
dev_addr = swiotlb_virt_to_bus(hwdev, ret);
-   if (dev_addr + size - 1 > dma_mask) {
+   if (dev_addr + size - 1 > hwdev->coherent_dma_mask) {
/*
 * The allocated memory isn't reachable by the device.
 */
@@ -747,9 +743,9 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,
dev_addr = swiotlb_phys_to_dma(hwdev, paddr);
 
/* Confirm address can be DMA'd by device */
-   if (dev_addr + size - 1 > dma_mask) {
+   if (dev_addr + size - 1 > hwdev->coherent_dma_mask) {
printk("hwdev DMA mask = 0x%016Lx, dev_addr = 
0x%016Lx\n",
-  (unsigned long long)dma_mask,
+  (unsigned long long)hwdev->coherent_dma_mask,
   (unsigned long long)dev_addr);
 
/*
-- 
2.14.2

[PATCH 47/67] swiotlb: wire up ->dma_supported in swiotlb_dma_ops

[Christoph Hellwig]

To properly reject too small DMA masks based on the addressability of the
bounce buffer.

Signed-off-by: Christoph Hellwig 
---
 lib/swiotlb.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 9c100f0173bf..e0b8980334c3 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -1125,5 +1125,6 @@ const struct dma_map_ops swiotlb_dma_ops = {
.unmap_sg   = swiotlb_unmap_sg_attrs,
.map_page   = swiotlb_map_page,
.unmap_page = swiotlb_unmap_page,
+   .dma_supported  = swiotlb_dma_supported,
 };
 #endif /* CONFIG_DMA_DIRECT_OPS */
-- 
2.14.2

[PATCH 46/67] swiotlb: lift x86 swiotlb_dma_ops to common code

[Christoph Hellwig]

Including the useful helpers for coherent allocations that first try the
full blown direct mapping.

Signed-off-by: Christoph Hellwig 
---
 arch/x86/include/asm/swiotlb.h |  8 
 arch/x86/kernel/pci-swiotlb.c  | 45 --
 arch/x86/pci/sta2x11-fixup.c   |  4 ++--
 include/linux/swiotlb.h|  8 
 lib/swiotlb.c  | 43 
 5 files changed, 53 insertions(+), 55 deletions(-)

diff --git a/arch/x86/include/asm/swiotlb.h b/arch/x86/include/asm/swiotlb.h
index 1c6a6cb230ff..ff6c92eff035 100644
--- a/arch/x86/include/asm/swiotlb.h
+++ b/arch/x86/include/asm/swiotlb.h
@@ -27,12 +27,4 @@ static inline void pci_swiotlb_late_init(void)
 {
 }
 #endif
-
-extern void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size,
-   dma_addr_t *dma_handle, gfp_t flags,
-   unsigned long attrs);
-extern void x86_swiotlb_free_coherent(struct device *dev, size_t size,
-   void *vaddr, dma_addr_t dma_addr,
-   unsigned long attrs);
-
 #endif /* _ASM_X86_SWIOTLB_H */
diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c
index 57dea60c2473..661583662430 100644
--- a/arch/x86/kernel/pci-swiotlb.c
+++ b/arch/x86/kernel/pci-swiotlb.c
@@ -17,51 +17,6 @@
 
 int swiotlb __read_mostly;
 
-void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size,
-   dma_addr_t *dma_handle, gfp_t flags,
-   unsigned long attrs)
-{
-   void *vaddr;
-
-   /*
-* Don't print a warning when the first allocation attempt fails.
-* swiotlb_alloc_coherent() will print a warning when the DMA
-* memory allocation ultimately failed.
-*/
-   flags |= __GFP_NOWARN;
-
-   vaddr = dma_direct_alloc(hwdev, size, dma_handle, flags, attrs);
-   if (vaddr)
-   return vaddr;
-
-   return swiotlb_alloc_coherent(hwdev, size, dma_handle, flags);
-}
-
-void x86_swiotlb_free_coherent(struct device *dev, size_t size,
- void *vaddr, dma_addr_t dma_addr,
- unsigned long attrs)
-{
-   if (is_swiotlb_buffer(dma_to_phys(dev, dma_addr)))
-   swiotlb_free_coherent(dev, size, vaddr, dma_addr);
-   else
-   dma_direct_free(dev, size, vaddr, dma_addr, attrs);
-}
-
-static const struct dma_map_ops swiotlb_dma_ops = {
-   .mapping_error = swiotlb_dma_mapping_error,
-   .alloc = x86_swiotlb_alloc_coherent,
-   .free = x86_swiotlb_free_coherent,
-   .sync_single_for_cpu = swiotlb_sync_single_for_cpu,
-   .sync_single_for_device = swiotlb_sync_single_for_device,
-   .sync_sg_for_cpu = swiotlb_sync_sg_for_cpu,
-   .sync_sg_for_device = swiotlb_sync_sg_for_device,
-   .map_sg = swiotlb_map_sg_attrs,
-   .unmap_sg = swiotlb_unmap_sg_attrs,
-   .map_page = swiotlb_map_page,
-   .unmap_page = swiotlb_unmap_page,
-   .dma_supported = NULL,
-};
-
 /*
  * pci_swiotlb_detect_override - set swiotlb to 1 if necessary
  *
diff --git a/arch/x86/pci/sta2x11-fixup.c b/arch/x86/pci/sta2x11-fixup.c
index 6c712fe11bdc..4b69b008d5aa 100644
--- a/arch/x86/pci/sta2x11-fixup.c
+++ b/arch/x86/pci/sta2x11-fixup.c
@@ -175,7 +175,7 @@ static void *sta2x11_swiotlb_alloc_coherent(struct device 
*dev,
 {
void *vaddr;
 
-   vaddr = x86_swiotlb_alloc_coherent(dev, size, dma_handle, flags, attrs);
+   vaddr = swiotlb_alloc(dev, size, dma_handle, flags, attrs);
*dma_handle = p2a(*dma_handle, to_pci_dev(dev));
return vaddr;
 }
@@ -183,7 +183,7 @@ static void *sta2x11_swiotlb_alloc_coherent(struct device 
*dev,
 /* We have our own dma_ops: the same as swiotlb but from alloc (above) */
 static const struct dma_map_ops sta2x11_dma_ops = {
.alloc = sta2x11_swiotlb_alloc_coherent,
-   .free = x86_swiotlb_free_coherent,
+   .free = swiotlb_free,
.map_page = swiotlb_map_page,
.unmap_page = swiotlb_unmap_page,
.map_sg = swiotlb_map_sg_attrs,
diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index 606375e35d87..5b1f2a00491c 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -66,6 +66,12 @@ extern void swiotlb_tbl_sync_single(struct device *hwdev,
enum dma_sync_target target);
 
 /* Accessory functions. */
+
+void *swiotlb_alloc(struct device *hwdev, size_t size, dma_addr_t *dma_handle,
+   gfp_t flags, unsigned long attrs);
+void swiotlb_free(struct device *dev, size_t size, void *vaddr,
+   dma_addr_t dma_addr, unsigned long attrs);
+
 extern void
 *swiotlb_alloc_coherent(struct device *hwdev, size_t size,
dma_addr_t *dma_handle, gfp_t flags);
@@ -126,4 +132,6 @@ extern 

[PATCH 45/67] swiotlb: rename swiotlb_free to swiotlb_exit

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 arch/powerpc/kernel/dma-swiotlb.c | 2 +-
 arch/x86/kernel/pci-swiotlb.c | 2 +-
 include/linux/swiotlb.h   | 4 ++--
 lib/swiotlb.c | 2 +-
 4 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/arch/powerpc/kernel/dma-swiotlb.c 
b/arch/powerpc/kernel/dma-swiotlb.c
index 506ac4fafac5..88f3963ca30f 100644
--- a/arch/powerpc/kernel/dma-swiotlb.c
+++ b/arch/powerpc/kernel/dma-swiotlb.c
@@ -121,7 +121,7 @@ static int __init check_swiotlb_enabled(void)
if (ppc_swiotlb_enable)
swiotlb_print_info();
else
-   swiotlb_free();
+   swiotlb_exit();
 
return 0;
 }
diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c
index 7a11a3e4f697..57dea60c2473 100644
--- a/arch/x86/kernel/pci-swiotlb.c
+++ b/arch/x86/kernel/pci-swiotlb.c
@@ -119,7 +119,7 @@ void __init pci_swiotlb_late_init(void)
 {
/* An IOMMU turned us off. */
if (!swiotlb)
-   swiotlb_free();
+   swiotlb_exit();
else {
printk(KERN_INFO "PCI-DMA: "
   "Using software bounce buffering for IO (SWIOTLB)\n");
diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index 24ed817082ee..606375e35d87 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -115,10 +115,10 @@ extern int
 swiotlb_dma_supported(struct device *hwdev, u64 mask);
 
 #ifdef CONFIG_SWIOTLB
-extern void __init swiotlb_free(void);
+extern void __init swiotlb_exit(void);
 unsigned int swiotlb_max_segment(void);
 #else
-static inline void swiotlb_free(void) { }
+static inline void swiotlb_exit(void) { }
 static inline unsigned int swiotlb_max_segment(void) { return 0; }
 #endif
 
diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 6583f3512386..c1fcd3a32d07 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -417,7 +417,7 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs)
return -ENOMEM;
 }
 
-void __init swiotlb_free(void)
+void __init swiotlb_exit(void)
 {
if (!io_tlb_orig_addr)
return;
-- 
2.14.2

[PATCH 44/67] powerpc: rename swiotlb_dma_ops

[Christoph Hellwig]

We'll need that name for a generic implementation soon.

Signed-off-by: Christoph Hellwig 
---
 arch/powerpc/include/asm/swiotlb.h | 2 +-
 arch/powerpc/kernel/dma-swiotlb.c  | 4 ++--
 arch/powerpc/kernel/dma.c  | 2 +-
 arch/powerpc/sysdev/fsl_pci.c  | 2 +-
 4 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/arch/powerpc/include/asm/swiotlb.h 
b/arch/powerpc/include/asm/swiotlb.h
index 9341ee804d19..f65ecf57b66c 100644
--- a/arch/powerpc/include/asm/swiotlb.h
+++ b/arch/powerpc/include/asm/swiotlb.h
@@ -13,7 +13,7 @@
 
 #include 
 
-extern const struct dma_map_ops swiotlb_dma_ops;
+extern const struct dma_map_ops powerpc_swiotlb_dma_ops;
 
 extern unsigned int ppc_swiotlb_enable;
 int __init swiotlb_setup_bus_notifier(void);
diff --git a/arch/powerpc/kernel/dma-swiotlb.c 
b/arch/powerpc/kernel/dma-swiotlb.c
index f1e99b9cee97..506ac4fafac5 100644
--- a/arch/powerpc/kernel/dma-swiotlb.c
+++ b/arch/powerpc/kernel/dma-swiotlb.c
@@ -46,7 +46,7 @@ static u64 swiotlb_powerpc_get_required(struct device *dev)
  * map_page, and unmap_page on highmem, use normal dma_ops
  * for everything else.
  */
-const struct dma_map_ops swiotlb_dma_ops = {
+const struct dma_map_ops powerpc_swiotlb_dma_ops = {
.alloc = __dma_nommu_alloc_coherent,
.free = __dma_nommu_free_coherent,
.mmap = dma_nommu_mmap_coherent,
@@ -89,7 +89,7 @@ static int ppc_swiotlb_bus_notify(struct notifier_block *nb,
 
/* May need to bounce if the device can't address all of DRAM */
if ((dma_get_mask(dev) + 1) < memblock_end_of_DRAM())
-   set_dma_ops(dev, _dma_ops);
+   set_dma_ops(dev, _swiotlb_dma_ops);
 
return NOTIFY_DONE;
 }
diff --git a/arch/powerpc/kernel/dma.c b/arch/powerpc/kernel/dma.c
index 1723001d5de1..b787692b91ee 100644
--- a/arch/powerpc/kernel/dma.c
+++ b/arch/powerpc/kernel/dma.c
@@ -33,7 +33,7 @@ static u64 __maybe_unused get_pfn_limit(struct device *dev)
struct dev_archdata __maybe_unused *sd = >archdata;
 
 #ifdef CONFIG_SWIOTLB
-   if (sd->max_direct_dma_addr && dev->dma_ops == _dma_ops)
+   if (sd->max_direct_dma_addr && dev->dma_ops == _swiotlb_dma_ops)
pfn = min_t(u64, pfn, sd->max_direct_dma_addr >> PAGE_SHIFT);
 #endif
 
diff --git a/arch/powerpc/sysdev/fsl_pci.c b/arch/powerpc/sysdev/fsl_pci.c
index e4d0133bbeeb..61e07c78d64f 100644
--- a/arch/powerpc/sysdev/fsl_pci.c
+++ b/arch/powerpc/sysdev/fsl_pci.c
@@ -118,7 +118,7 @@ static void setup_swiotlb_ops(struct pci_controller *hose)
 {
if (ppc_swiotlb_enable) {
hose->controller_ops.dma_dev_setup = pci_dma_dev_setup_swiotlb;
-   set_pci_dma_ops(_dma_ops);
+   set_pci_dma_ops(_swiotlb_dma_ops);
}
 }
 #else
-- 
2.14.2

[PATCH 43/67] ia64: rename swiotlb_dma_ops

[Christoph Hellwig]

We'll need that name for a generic implementation soon.

Signed-off-by: Christoph Hellwig 
---
 arch/ia64/hp/common/hwsw_iommu.c | 4 ++--
 arch/ia64/hp/common/sba_iommu.c  | 6 +++---
 arch/ia64/kernel/pci-swiotlb.c   | 6 +++---
 3 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/arch/ia64/hp/common/hwsw_iommu.c b/arch/ia64/hp/common/hwsw_iommu.c
index 63d8e1d2477f..41279f0442bd 100644
--- a/arch/ia64/hp/common/hwsw_iommu.c
+++ b/arch/ia64/hp/common/hwsw_iommu.c
@@ -19,7 +19,7 @@
 #include 
 #include 
 
-extern const struct dma_map_ops sba_dma_ops, swiotlb_dma_ops;
+extern const struct dma_map_ops sba_dma_ops, ia64_swiotlb_dma_ops;
 
 /* swiotlb declarations & definitions: */
 extern int swiotlb_late_init_with_default_size (size_t size);
@@ -38,7 +38,7 @@ static inline int use_swiotlb(struct device *dev)
 const struct dma_map_ops *hwsw_dma_get_ops(struct device *dev)
 {
if (use_swiotlb(dev))
-   return _dma_ops;
+   return _swiotlb_dma_ops;
return _dma_ops;
 }
 EXPORT_SYMBOL(hwsw_dma_get_ops);
diff --git a/arch/ia64/hp/common/sba_iommu.c b/arch/ia64/hp/common/sba_iommu.c
index 6f05aba9012f..d68849ad2ee1 100644
--- a/arch/ia64/hp/common/sba_iommu.c
+++ b/arch/ia64/hp/common/sba_iommu.c
@@ -2093,7 +2093,7 @@ static int __init acpi_sba_ioc_init_acpi(void)
 /* This has to run before acpi_scan_init(). */
 arch_initcall(acpi_sba_ioc_init_acpi);
 
-extern const struct dma_map_ops swiotlb_dma_ops;
+extern const struct dma_map_ops ia64_swiotlb_dma_ops;
 
 static int __init
 sba_init(void)
@@ -2108,7 +2108,7 @@ sba_init(void)
 * a successful kdump kernel boot is to use the swiotlb.
 */
if (is_kdump_kernel()) {
-   dma_ops = _dma_ops;
+   dma_ops = _swiotlb_dma_ops;
if (swiotlb_late_init_with_default_size(64 * (1<<20)) != 0)
panic("Unable to initialize software I/O TLB:"
  " Try machvec=dig boot option");
@@ -2130,7 +2130,7 @@ sba_init(void)
 * If we didn't find something sba_iommu can claim, we
 * need to setup the swiotlb and switch to the dig machvec.
 */
-   dma_ops = _dma_ops;
+   dma_ops = _swiotlb_dma_ops;
if (swiotlb_late_init_with_default_size(64 * (1<<20)) != 0)
panic("Unable to find SBA IOMMU or initialize "
  "software I/O TLB: Try machvec=dig boot option");
diff --git a/arch/ia64/kernel/pci-swiotlb.c b/arch/ia64/kernel/pci-swiotlb.c
index 5e50939aa03e..f1ae873a8c35 100644
--- a/arch/ia64/kernel/pci-swiotlb.c
+++ b/arch/ia64/kernel/pci-swiotlb.c
@@ -31,7 +31,7 @@ static void ia64_swiotlb_free_coherent(struct device *dev, 
size_t size,
swiotlb_free_coherent(dev, size, vaddr, dma_addr);
 }
 
-const struct dma_map_ops swiotlb_dma_ops = {
+const struct dma_map_ops ia64_swiotlb_dma_ops = {
.alloc = ia64_swiotlb_alloc_coherent,
.free = ia64_swiotlb_free_coherent,
.map_page = swiotlb_map_page,
@@ -48,7 +48,7 @@ const struct dma_map_ops swiotlb_dma_ops = {
 
 void __init swiotlb_dma_init(void)
 {
-   dma_ops = _dma_ops;
+   dma_ops = _swiotlb_dma_ops;
swiotlb_init(1);
 }
 
@@ -60,7 +60,7 @@ void __init pci_swiotlb_init(void)
printk(KERN_INFO "PCI-DMA: Re-initialize machine vector.\n");
machvec_init("dig");
swiotlb_init(1);
-   dma_ops = _dma_ops;
+   dma_ops = _swiotlb_dma_ops;
 #else
panic("Unable to find Intel IOMMU");
 #endif
-- 
2.14.2

[PATCH 42/67] arm64: rename swiotlb_dma_ops

[Christoph Hellwig]

We'll need that name for a generic implementation soon.

Signed-off-by: Christoph Hellwig 
---
 arch/arm64/mm/dma-mapping.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index f3a637b98487..6840426bbe77 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -368,7 +368,7 @@ static int __swiotlb_dma_mapping_error(struct device 
*hwdev, dma_addr_t addr)
return 0;
 }
 
-static const struct dma_map_ops swiotlb_dma_ops = {
+static const struct dma_map_ops arm64_swiotlb_dma_ops = {
.alloc = __dma_alloc,
.free = __dma_free,
.mmap = __swiotlb_mmap,
@@ -923,7 +923,7 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, 
u64 size,
const struct iommu_ops *iommu, bool coherent)
 {
if (!dev->dma_ops)
-   dev->dma_ops = _dma_ops;
+   dev->dma_ops = _swiotlb_dma_ops;
 
dev->archdata.dma_coherent = coherent;
__iommu_setup_dma_ops(dev, dma_base, size, iommu);
-- 
2.14.2

[PATCH 41/67] x86: remove dma_alloc_coherent_gfp_flags

[Christoph Hellwig]

All dma_ops implementations used on x86 now take care of setting their own
required GFP_ masks for the allocation.  And given that the common code
now clears harmful flags itself that means we can stop the flags in all
the iommu implementations as well.

Signed-off-by: Christoph Hellwig 
---
 arch/x86/include/asm/dma-mapping.h | 11 ---
 arch/x86/kernel/amd_gart_64.c  |  1 -
 arch/x86/kernel/pci-calgary_64.c   |  2 --
 arch/x86/kernel/pci-dma.c  |  2 --
 arch/x86/mm/mem_encrypt.c  |  7 ---
 drivers/iommu/amd_iommu.c  |  1 -
 drivers/iommu/intel-iommu.c|  1 -
 7 files changed, 25 deletions(-)

diff --git a/arch/x86/include/asm/dma-mapping.h 
b/arch/x86/include/asm/dma-mapping.h
index df9816b385eb..89ce4bfd241f 100644
--- a/arch/x86/include/asm/dma-mapping.h
+++ b/arch/x86/include/asm/dma-mapping.h
@@ -36,15 +36,4 @@ int arch_dma_supported(struct device *dev, u64 mask);
 bool arch_dma_alloc_attrs(struct device **dev, gfp_t *gfp);
 #define arch_dma_alloc_attrs arch_dma_alloc_attrs
 
-static inline gfp_t dma_alloc_coherent_gfp_flags(struct device *dev, gfp_t gfp)
-{
-   if (dev->coherent_dma_mask <= DMA_BIT_MASK(24))
-   gfp |= GFP_DMA;
-#ifdef CONFIG_X86_64
-   if (dev->coherent_dma_mask <= DMA_BIT_MASK(32) && !(gfp & GFP_DMA))
-   gfp |= GFP_DMA32;
-#endif
-   return gfp;
-}
-
 #endif
diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c
index 92054815023e..7466dd458e0f 100644
--- a/arch/x86/kernel/amd_gart_64.c
+++ b/arch/x86/kernel/amd_gart_64.c
@@ -487,7 +487,6 @@ gart_alloc_coherent(struct device *dev, size_t size, 
dma_addr_t *dma_addr,
if (!force_iommu || dev->coherent_dma_mask <= DMA_BIT_MASK(24))
return dma_direct_alloc(dev, size, dma_addr, flag, attrs);
 
-   flag &= ~(__GFP_DMA | __GFP_HIGHMEM | __GFP_DMA32);
page = alloc_pages(flag | __GFP_ZERO, get_order(size));
if (!page)
return NULL;
diff --git a/arch/x86/kernel/pci-calgary_64.c b/arch/x86/kernel/pci-calgary_64.c
index 5647853053bd..bbfc8b1e9104 100644
--- a/arch/x86/kernel/pci-calgary_64.c
+++ b/arch/x86/kernel/pci-calgary_64.c
@@ -446,8 +446,6 @@ static void* calgary_alloc_coherent(struct device *dev, 
size_t size,
npages = size >> PAGE_SHIFT;
order = get_order(size);
 
-   flag &= ~(__GFP_DMA | __GFP_HIGHMEM | __GFP_DMA32);
-
/* alloc enough pages (and possibly more) */
ret = (void *)__get_free_pages(flag, order);
if (!ret)
diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
index db0b88ea8d1b..14437116ffea 100644
--- a/arch/x86/kernel/pci-dma.c
+++ b/arch/x86/kernel/pci-dma.c
@@ -82,8 +82,6 @@ bool arch_dma_alloc_attrs(struct device **dev, gfp_t *gfp)
if (!*dev)
*dev = _dma_fallback_dev;
 
-   *gfp = dma_alloc_coherent_gfp_flags(*dev, *gfp);
-
if (!is_device_dma_capable(*dev))
return false;
return true;
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index 479586b8ca9b..1c786e751b49 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -208,13 +208,6 @@ static void *sev_alloc(struct device *dev, size_t size, 
dma_addr_t *dma_handle,
void *vaddr = NULL;
 
order = get_order(size);
-
-   /*
-* Memory will be memset to zero after marking decrypted, so don't
-* bother clearing it before.
-*/
-   gfp &= ~__GFP_ZERO;
-
page = alloc_pages_node(dev_to_node(dev), gfp, order);
if (page) {
dma_addr_t addr;
diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
index a2ad149ab0bf..51ce6db86fdd 100644
--- a/drivers/iommu/amd_iommu.c
+++ b/drivers/iommu/amd_iommu.c
@@ -2600,7 +2600,6 @@ static void *alloc_coherent(struct device *dev, size_t 
size,
dma_dom   = to_dma_ops_domain(domain);
size  = PAGE_ALIGN(size);
dma_mask  = dev->coherent_dma_mask;
-   flag &= ~(__GFP_DMA | __GFP_HIGHMEM | __GFP_DMA32);
flag |= __GFP_ZERO;
 
page = alloc_pages(flag | __GFP_NOWARN,  get_order(size));
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 0de8bfe89061..6c9df0773b78 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -3718,7 +3718,6 @@ static void *intel_alloc_coherent(struct device *dev, 
size_t size,
 
size = PAGE_ALIGN(size);
order = get_order(size);
-   flags &= ~(GFP_DMA | GFP_DMA32);
 
if (gfpflags_allow_blocking(flags)) {
unsigned int count = size >> PAGE_SHIFT;
-- 
2.14.2

[PATCH 40/67] iommu/intel-iommu: use dma_direct_* helpers for the direct mapping case

[Christoph Hellwig]

This simplifies the code a bit, and prepares for future cleanups.

Signed-off-by: Christoph Hellwig 
---
 drivers/iommu/Kconfig   |  1 +
 drivers/iommu/intel-iommu.c | 17 -
 2 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index dc7c1914645d..df171cb85822 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -143,6 +143,7 @@ config DMAR_TABLE
 config INTEL_IOMMU
bool "Support for Intel IOMMU using DMA Remapping Devices"
depends on PCI_MSI && ACPI && (X86 || IA64_GENERIC)
+   select DMA_DIRECT_OPS
select IOMMU_API
select IOMMU_IOVA
select DMAR_TABLE
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 921caf4f0c3e..0de8bfe89061 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -31,6 +31,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -3712,17 +3713,12 @@ static void *intel_alloc_coherent(struct device *dev, 
size_t size,
struct page *page = NULL;
int order;
 
+   if (iommu_no_mapping(dev))
+   return dma_direct_alloc(dev, size, dma_handle, flags, attrs);
+
size = PAGE_ALIGN(size);
order = get_order(size);
-
-   if (!iommu_no_mapping(dev))
-   flags &= ~(GFP_DMA | GFP_DMA32);
-   else if (dev->coherent_dma_mask < dma_get_required_mask(dev)) {
-   if (dev->coherent_dma_mask < DMA_BIT_MASK(32))
-   flags |= GFP_DMA;
-   else
-   flags |= GFP_DMA32;
-   }
+   flags &= ~(GFP_DMA | GFP_DMA32);
 
if (gfpflags_allow_blocking(flags)) {
unsigned int count = size >> PAGE_SHIFT;
@@ -3758,6 +3754,9 @@ static void intel_free_coherent(struct device *dev, 
size_t size, void *vaddr,
int order;
struct page *page = virt_to_page(vaddr);
 
+   if (iommu_no_mapping(dev))
+   return dma_direct_free(dev, size, vaddr, dma_handle, attrs);
+
size = PAGE_ALIGN(size);
order = get_order(size);
 
-- 
2.14.2

[PATCH 38/67] x86/amd_gart: clean up gart_alloc_coherent

[Christoph Hellwig]

Don't rely on the gfp mask from dma_alloc_coherent_gfp_flags to make the
fallback decision, and streamline the code flow a bit.

Signed-off-by: Christoph Hellwig 
---
 arch/x86/kernel/amd_gart_64.c | 36 ++--
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c
index 52e3abcf3e70..92054815023e 100644
--- a/arch/x86/kernel/amd_gart_64.c
+++ b/arch/x86/kernel/amd_gart_64.c
@@ -484,26 +484,26 @@ gart_alloc_coherent(struct device *dev, size_t size, 
dma_addr_t *dma_addr,
unsigned long align_mask;
struct page *page;
 
-   if (force_iommu && !(flag & GFP_DMA)) {
-   flag &= ~(__GFP_DMA | __GFP_HIGHMEM | __GFP_DMA32);
-   page = alloc_pages(flag | __GFP_ZERO, get_order(size));
-   if (!page)
-   return NULL;
-
-   align_mask = (1UL << get_order(size)) - 1;
-   paddr = dma_map_area(dev, page_to_phys(page), size,
-DMA_BIDIRECTIONAL, align_mask);
-
-   flush_gart();
-   if (paddr != bad_dma_addr) {
-   *dma_addr = paddr;
-   return page_address(page);
-   }
-   __free_pages(page, get_order(size));
-   } else
+   if (!force_iommu || dev->coherent_dma_mask <= DMA_BIT_MASK(24))
return dma_direct_alloc(dev, size, dma_addr, flag, attrs);
 
-   return NULL;
+   flag &= ~(__GFP_DMA | __GFP_HIGHMEM | __GFP_DMA32);
+   page = alloc_pages(flag | __GFP_ZERO, get_order(size));
+   if (!page)
+   return NULL;
+
+   align_mask = (1UL << get_order(size)) - 1;
+   paddr = dma_map_area(dev, page_to_phys(page), size, DMA_BIDIRECTIONAL,
+   align_mask);
+
+   flush_gart();
+   if (unlikely(paddr == bad_dma_addr)) {
+   __free_pages(page, get_order(size));
+   return NULL;
+   }
+
+   *dma_addr = paddr;
+   return page_address(page);
 }
 
 /* free a coherent mapping */
-- 
2.14.2

[PATCH 39/67] iommu/amd_iommu: use dma_direct_* helpers for the direct mapping case

[Christoph Hellwig]

This adds support for CMA allocations, but is otherwise identical.

Signed-off-by: Christoph Hellwig 
---
 drivers/iommu/Kconfig |  1 +
 drivers/iommu/amd_iommu.c | 27 +--
 2 files changed, 10 insertions(+), 18 deletions(-)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index f3a21343e636..dc7c1914645d 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -107,6 +107,7 @@ config IOMMU_PGTABLES_L2
 # AMD IOMMU support
 config AMD_IOMMU
bool "AMD IOMMU support"
+   select DMA_DIRECT_OPS
select SWIOTLB
select PCI_MSI
select PCI_ATS
diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
index ea4734de5357..a2ad149ab0bf 100644
--- a/drivers/iommu/amd_iommu.c
+++ b/drivers/iommu/amd_iommu.c
@@ -2592,11 +2592,9 @@ static void *alloc_coherent(struct device *dev, size_t 
size,
struct page *page;
 
domain = get_domain(dev);
-   if (PTR_ERR(domain) == -EINVAL) {
-   page = alloc_pages(flag, get_order(size));
-   *dma_addr = page_to_phys(page);
-   return page_address(page);
-   } else if (IS_ERR(domain))
+   if (PTR_ERR(domain) == -EINVAL)
+   return dma_direct_alloc(dev, size, dma_addr, flag, attrs);
+   else if (IS_ERR(domain))
return NULL;
 
dma_dom   = to_dma_ops_domain(domain);
@@ -2642,24 +2640,17 @@ static void free_coherent(struct device *dev, size_t 
size,
  void *virt_addr, dma_addr_t dma_addr,
  unsigned long attrs)
 {
-   struct protection_domain *domain;
-   struct dma_ops_domain *dma_dom;
-   struct page *page;
+   struct protection_domain *domain = get_domain(dev);
 
-   page = virt_to_page(virt_addr);
size = PAGE_ALIGN(size);
 
-   domain = get_domain(dev);
-   if (IS_ERR(domain))
-   goto free_mem;
-
-   dma_dom = to_dma_ops_domain(domain);
+   if (!IS_ERR(domain)) {
+   struct dma_ops_domain *dma_dom = to_dma_ops_domain(domain);
 
-   __unmap_single(dma_dom, dma_addr, size, DMA_BIDIRECTIONAL);
+   __unmap_single(dma_dom, dma_addr, size, DMA_BIDIRECTIONAL);
+   }
 
-free_mem:
-   if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT))
-   __free_pages(page, get_order(size));
+   dma_direct_free(dev, size, virt_addr, dma_addr, attrs);
 }
 
 /*
-- 
2.14.2

[PATCH 37/67] x86: use dma-direct

[Christoph Hellwig]

The generic dma-direct implementation is now functionally equivalent to
the x86 nommu dma_map implementation, so switch over to using it.

Note that the various iommu drivers are switched from x86_dma_supported
to dma_direct_supported to provide identical functionality, although the
checks looks fairly questionable for at least some of them.

Signed-off-by: Christoph Hellwig 
---
 arch/x86/Kconfig   |  1 +
 arch/x86/include/asm/dma-mapping.h |  8 -
 arch/x86/include/asm/iommu.h   |  3 --
 arch/x86/kernel/Makefile   |  2 +-
 arch/x86/kernel/amd_gart_64.c  |  7 ++--
 arch/x86/kernel/pci-calgary_64.c   |  3 +-
 arch/x86/kernel/pci-dma.c  | 66 +-
 arch/x86/kernel/pci-swiotlb.c  |  5 ++-
 arch/x86/pci/sta2x11-fixup.c   |  2 +-
 drivers/iommu/amd_iommu.c  |  7 ++--
 drivers/iommu/intel-iommu.c|  3 +-
 11 files changed, 17 insertions(+), 90 deletions(-)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index f6f4328103c0..55ad01515075 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -83,6 +83,7 @@ config X86
select CLOCKSOURCE_VALIDATE_LAST_CYCLE
select CLOCKSOURCE_WATCHDOG
select DCACHE_WORD_ACCESS
+   select DMA_DIRECT_OPS
select EDAC_ATOMIC_SCRUB
select EDAC_SUPPORT
select GENERIC_CLOCKEVENTS
diff --git a/arch/x86/include/asm/dma-mapping.h 
b/arch/x86/include/asm/dma-mapping.h
index 545bf3721bc0..df9816b385eb 100644
--- a/arch/x86/include/asm/dma-mapping.h
+++ b/arch/x86/include/asm/dma-mapping.h
@@ -36,14 +36,6 @@ int arch_dma_supported(struct device *dev, u64 mask);
 bool arch_dma_alloc_attrs(struct device **dev, gfp_t *gfp);
 #define arch_dma_alloc_attrs arch_dma_alloc_attrs
 
-extern void *dma_generic_alloc_coherent(struct device *dev, size_t size,
-   dma_addr_t *dma_addr, gfp_t flag,
-   unsigned long attrs);
-
-extern void dma_generic_free_coherent(struct device *dev, size_t size,
- void *vaddr, dma_addr_t dma_addr,
- unsigned long attrs);
-
 static inline gfp_t dma_alloc_coherent_gfp_flags(struct device *dev, gfp_t gfp)
 {
if (dev->coherent_dma_mask <= DMA_BIT_MASK(24))
diff --git a/arch/x86/include/asm/iommu.h b/arch/x86/include/asm/iommu.h
index 1e5d5d92eb40..baedab8ac538 100644
--- a/arch/x86/include/asm/iommu.h
+++ b/arch/x86/include/asm/iommu.h
@@ -2,13 +2,10 @@
 #ifndef _ASM_X86_IOMMU_H
 #define _ASM_X86_IOMMU_H
 
-extern const struct dma_map_ops nommu_dma_ops;
 extern int force_iommu, no_iommu;
 extern int iommu_detected;
 extern int iommu_pass_through;
 
-int x86_dma_supported(struct device *dev, u64 mask);
-
 /* 10 seconds */
 #define DMAR_OPERATION_TIMEOUT ((cycles_t) tsc_khz*10*1000)
 
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 81bb565f4497..beee4332e69b 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -54,7 +54,7 @@ obj-$(CONFIG_X86_ESPFIX64)+= espfix_64.o
 obj-$(CONFIG_SYSFS)+= ksysfs.o
 obj-y  += bootflag.o e820.o
 obj-y  += pci-dma.o quirks.o topology.o kdebugfs.o
-obj-y  += alternative.o i8253.o pci-nommu.o hw_breakpoint.o
+obj-y  += alternative.o i8253.o hw_breakpoint.o
 obj-y  += tsc.o tsc_msr.o io_delay.o rtc.o
 obj-y  += pci-iommu_table.o
 obj-y  += resource.o
diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c
index ecd486cb06ab..52e3abcf3e70 100644
--- a/arch/x86/kernel/amd_gart_64.c
+++ b/arch/x86/kernel/amd_gart_64.c
@@ -501,8 +501,7 @@ gart_alloc_coherent(struct device *dev, size_t size, 
dma_addr_t *dma_addr,
}
__free_pages(page, get_order(size));
} else
-   return dma_generic_alloc_coherent(dev, size, dma_addr, flag,
- attrs);
+   return dma_direct_alloc(dev, size, dma_addr, flag, attrs);
 
return NULL;
 }
@@ -513,7 +512,7 @@ gart_free_coherent(struct device *dev, size_t size, void 
*vaddr,
   dma_addr_t dma_addr, unsigned long attrs)
 {
gart_unmap_page(dev, dma_addr, size, DMA_BIDIRECTIONAL, 0);
-   dma_generic_free_coherent(dev, size, vaddr, dma_addr, attrs);
+   dma_direct_free(dev, size, vaddr, dma_addr, attrs);
 }
 
 static int gart_mapping_error(struct device *dev, dma_addr_t dma_addr)
@@ -705,7 +704,7 @@ static const struct dma_map_ops gart_dma_ops = {
.alloc  = gart_alloc_coherent,
.free   = gart_free_coherent,
.mapping_error  = gart_mapping_error,
-   .dma_supported  = x86_dma_supported,
+   .dma_supported  = dma_direct_supported,
 };
 
 static void gart_iommu_shutdown(void)
diff 

[PATCH 36/67] x86: remove dma_alloc_coherent_mask

[Christoph Hellwig]

These days all devices (including the ISA fallback device) have a coherent
DMA mask set, so remove the workaround.

Signed-off-by: Christoph Hellwig 
---
 arch/x86/include/asm/dma-mapping.h | 18 ++
 arch/x86/kernel/pci-dma.c  | 10 --
 arch/x86/mm/mem_encrypt.c  |  4 +---
 drivers/xen/swiotlb-xen.c  | 16 +---
 4 files changed, 8 insertions(+), 40 deletions(-)

diff --git a/arch/x86/include/asm/dma-mapping.h 
b/arch/x86/include/asm/dma-mapping.h
index 6277c83c0eb1..545bf3721bc0 100644
--- a/arch/x86/include/asm/dma-mapping.h
+++ b/arch/x86/include/asm/dma-mapping.h
@@ -44,26 +44,12 @@ extern void dma_generic_free_coherent(struct device *dev, 
size_t size,
  void *vaddr, dma_addr_t dma_addr,
  unsigned long attrs);
 
-static inline unsigned long dma_alloc_coherent_mask(struct device *dev,
-   gfp_t gfp)
-{
-   unsigned long dma_mask = 0;
-
-   dma_mask = dev->coherent_dma_mask;
-   if (!dma_mask)
-   dma_mask = (gfp & GFP_DMA) ? DMA_BIT_MASK(24) : 
DMA_BIT_MASK(32);
-
-   return dma_mask;
-}
-
 static inline gfp_t dma_alloc_coherent_gfp_flags(struct device *dev, gfp_t gfp)
 {
-   unsigned long dma_mask = dma_alloc_coherent_mask(dev, gfp);
-
-   if (dma_mask <= DMA_BIT_MASK(24))
+   if (dev->coherent_dma_mask <= DMA_BIT_MASK(24))
gfp |= GFP_DMA;
 #ifdef CONFIG_X86_64
-   if (dma_mask <= DMA_BIT_MASK(32) && !(gfp & GFP_DMA))
+   if (dev->coherent_dma_mask <= DMA_BIT_MASK(32) && !(gfp & GFP_DMA))
gfp |= GFP_DMA32;
 #endif
return gfp;
diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
index df7ab02f959f..b59820872ec7 100644
--- a/arch/x86/kernel/pci-dma.c
+++ b/arch/x86/kernel/pci-dma.c
@@ -80,13 +80,10 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t 
size,
 dma_addr_t *dma_addr, gfp_t flag,
 unsigned long attrs)
 {
-   unsigned long dma_mask;
struct page *page;
unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
dma_addr_t addr;
 
-   dma_mask = dma_alloc_coherent_mask(dev, flag);
-
 again:
page = NULL;
/* CMA can be used only in the context which permits sleeping */
@@ -95,7 +92,7 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t 
size,
 flag);
if (page) {
addr = phys_to_dma(dev, page_to_phys(page));
-   if (addr + size > dma_mask) {
+   if (addr + size > dev->coherent_dma_mask) {
dma_release_from_contiguous(dev, page, count);
page = NULL;
}
@@ -108,10 +105,11 @@ void *dma_generic_alloc_coherent(struct device *dev, 
size_t size,
return NULL;
 
addr = phys_to_dma(dev, page_to_phys(page));
-   if (addr + size > dma_mask) {
+   if (addr + size > dev->coherent_dma_mask) {
__free_pages(page, get_order(size));
 
-   if (dma_mask < DMA_BIT_MASK(32) && !(flag & GFP_DMA)) {
+   if (dev->coherent_dma_mask < DMA_BIT_MASK(32) &&
+   !(flag & GFP_DMA)) {
flag = (flag & ~GFP_DMA32) | GFP_DMA;
goto again;
}
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index 764b916ef7da..479586b8ca9b 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -203,12 +203,10 @@ void __init sme_early_init(void)
 static void *sev_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle,
   gfp_t gfp, unsigned long attrs)
 {
-   unsigned long dma_mask;
unsigned int order;
struct page *page;
void *vaddr = NULL;
 
-   dma_mask = dma_alloc_coherent_mask(dev, gfp);
order = get_order(size);
 
/*
@@ -226,7 +224,7 @@ static void *sev_alloc(struct device *dev, size_t size, 
dma_addr_t *dma_handle,
 * mask with it already cleared.
 */
addr = __sme_clr(phys_to_dma(dev, page_to_phys(page)));
-   if ((addr + size) > dma_mask) {
+   if ((addr + size) > dev->coherent_dma_mask) {
__free_pages(page, get_order(size));
} else {
vaddr = page_address(page);
diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 5bb72d3f8337..e1c60899fdbc 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -53,20 +53,6 @@
  * API.
  */
 
-#ifndef CONFIG_X86
-static unsigned long dma_alloc_coherent_mask(struct device *dev,
-   gfp_t gfp)
-{
-   unsigned long dma_mask = 

[PATCH 35/67] h8300: use dma-direct

[Christoph Hellwig]

Replace the bare-bones h8300 direct dma mapping implementation with
the fully featured generic dma-direct one.

Signed-off-by: Christoph Hellwig 
---
 arch/h8300/Kconfig   |  1 +
 arch/h8300/include/asm/Kbuild|  1 +
 arch/h8300/include/asm/dma-mapping.h | 12 ---
 arch/h8300/kernel/Makefile   |  2 +-
 arch/h8300/kernel/dma.c  | 67 
 5 files changed, 3 insertions(+), 80 deletions(-)
 delete mode 100644 arch/h8300/include/asm/dma-mapping.h
 delete mode 100644 arch/h8300/kernel/dma.c

diff --git a/arch/h8300/Kconfig b/arch/h8300/Kconfig
index f8d3fde08190..091d6d04b5e5 100644
--- a/arch/h8300/Kconfig
+++ b/arch/h8300/Kconfig
@@ -23,6 +23,7 @@ config H8300
select HAVE_ARCH_KGDB
select HAVE_ARCH_HASH
select CPU_NO_EFFICIENT_FFS
+   select DMA_DIRECT_OPS
 
 config CPU_BIG_ENDIAN
def_bool y
diff --git a/arch/h8300/include/asm/Kbuild b/arch/h8300/include/asm/Kbuild
index bc077491d299..642752c94306 100644
--- a/arch/h8300/include/asm/Kbuild
+++ b/arch/h8300/include/asm/Kbuild
@@ -9,6 +9,7 @@ generic-y += delay.h
 generic-y += device.h
 generic-y += div64.h
 generic-y += dma.h
+generic-y += dma-mapping.h
 generic-y += emergency-restart.h
 generic-y += exec.h
 generic-y += extable.h
diff --git a/arch/h8300/include/asm/dma-mapping.h 
b/arch/h8300/include/asm/dma-mapping.h
deleted file mode 100644
index 21bb1fc3a6f1..
--- a/arch/h8300/include/asm/dma-mapping.h
+++ /dev/null
@@ -1,12 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _H8300_DMA_MAPPING_H
-#define _H8300_DMA_MAPPING_H
-
-extern const struct dma_map_ops h8300_dma_map_ops;
-
-static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
-{
-   return _dma_map_ops;
-}
-
-#endif
diff --git a/arch/h8300/kernel/Makefile b/arch/h8300/kernel/Makefile
index b62e830525c6..307aa51576dd 100644
--- a/arch/h8300/kernel/Makefile
+++ b/arch/h8300/kernel/Makefile
@@ -7,7 +7,7 @@ extra-y := vmlinux.lds
 
 obj-y := process.o traps.o ptrace.o \
 signal.o setup.o syscalls.o \
-irq.o entry.o dma.o
+irq.o entry.o
 
 obj-$(CONFIG_ROMKERNEL) += head_rom.o
 obj-$(CONFIG_RAMKERNEL) += head_ram.o
diff --git a/arch/h8300/kernel/dma.c b/arch/h8300/kernel/dma.c
deleted file mode 100644
index 4e27b74df973..
--- a/arch/h8300/kernel/dma.c
+++ /dev/null
@@ -1,67 +0,0 @@
-/*
- * This file is subject to the terms and conditions of the GNU General Public
- * License.  See the file COPYING in the main directory of this archive
- * for more details.
- */
-
-#include 
-#include 
-#include 
-#include 
-#include 
-
-static void *dma_alloc(struct device *dev, size_t size,
-  dma_addr_t *dma_handle, gfp_t gfp,
-  unsigned long attrs)
-{
-   void *ret;
-
-   if (dev == NULL || (*dev->dma_mask < 0x))
-   gfp |= GFP_DMA;
-   ret = (void *)__get_free_pages(gfp, get_order(size));
-
-   if (ret != NULL) {
-   memset(ret, 0, size);
-   *dma_handle = virt_to_phys(ret);
-   }
-   return ret;
-}
-
-static void dma_free(struct device *dev, size_t size,
-void *vaddr, dma_addr_t dma_handle,
-unsigned long attrs)
-
-{
-   free_pages((unsigned long)vaddr, get_order(size));
-}
-
-static dma_addr_t map_page(struct device *dev, struct page *page,
- unsigned long offset, size_t size,
- enum dma_data_direction direction,
- unsigned long attrs)
-{
-   return page_to_phys(page) + offset;
-}
-
-static int map_sg(struct device *dev, struct scatterlist *sgl,
- int nents, enum dma_data_direction direction,
- unsigned long attrs)
-{
-   struct scatterlist *sg;
-   int i;
-
-   for_each_sg(sgl, sg, nents, i) {
-   sg->dma_address = sg_phys(sg);
-   }
-
-   return nents;
-}
-
-const struct dma_map_ops h8300_dma_map_ops = {
-   .alloc = dma_alloc,
-   .free = dma_free,
-   .map_page = map_page,
-   .map_sg = map_sg,
-   .is_phys = true,
-};
-EXPORT_SYMBOL(h8300_dma_map_ops);
-- 
2.14.2

[PATCH 34/67] cris: use dma-direct

[Christoph Hellwig]

cris currently has an incomplete direct mapping dma_map_ops implementation
is PCI support is enabled.  Replace it with the fully feature generic
dma-direct implementation.

Signed-off-by: Christoph Hellwig 
---
 arch/cris/Kconfig   |  4 ++
 arch/cris/arch-v32/drivers/pci/Makefile |  2 +-
 arch/cris/arch-v32/drivers/pci/dma.c| 78 -
 arch/cris/include/asm/Kbuild|  1 +
 arch/cris/include/asm/dma-mapping.h | 20 -
 5 files changed, 6 insertions(+), 99 deletions(-)
 delete mode 100644 arch/cris/arch-v32/drivers/pci/dma.c
 delete mode 100644 arch/cris/include/asm/dma-mapping.h

diff --git a/arch/cris/Kconfig b/arch/cris/Kconfig
index 54d3f426763b..cd5a0865c97f 100644
--- a/arch/cris/Kconfig
+++ b/arch/cris/Kconfig
@@ -33,6 +33,9 @@ config GENERIC_CALIBRATE_DELAY
 config NO_IOPORT_MAP
def_bool y if !PCI
 
+config NO_DMA
+   def_bool y if !PCI
+
 config FORCE_MAX_ZONEORDER
int
default 6
@@ -72,6 +75,7 @@ config CRIS
select GENERIC_SCHED_CLOCK if ETRAX_ARCH_V32
select HAVE_DEBUG_BUGVERBOSE if ETRAX_ARCH_V32
select HAVE_NMI
+   select DMA_DIRECT_OPS if PCI
 
 config HZ
int
diff --git a/arch/cris/arch-v32/drivers/pci/Makefile 
b/arch/cris/arch-v32/drivers/pci/Makefile
index bff7482f2444..93c8be6170b1 100644
--- a/arch/cris/arch-v32/drivers/pci/Makefile
+++ b/arch/cris/arch-v32/drivers/pci/Makefile
@@ -2,4 +2,4 @@
 # Makefile for Etrax cardbus driver
 #
 
-obj-$(CONFIG_ETRAX_CARDBUS)+= bios.o dma.o
+obj-$(CONFIG_ETRAX_CARDBUS)+= bios.o
diff --git a/arch/cris/arch-v32/drivers/pci/dma.c 
b/arch/cris/arch-v32/drivers/pci/dma.c
deleted file mode 100644
index c7e3056885d3..
--- a/arch/cris/arch-v32/drivers/pci/dma.c
+++ /dev/null
@@ -1,78 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/*
- * Dynamic DMA mapping support.
- *
- * On cris there is no hardware dynamic DMA address translation,
- * so consistent alloc/free are merely page allocation/freeing.
- * The rest of the dynamic DMA mapping interface is implemented
- * in asm/pci.h.
- *
- * Borrowed from i386.
- */
-
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-
-static void *v32_dma_alloc(struct device *dev, size_t size,
-   dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs)
-{
-   void *ret;
-
-   if (dev == NULL || (dev->coherent_dma_mask < 0x))
-   gfp |= GFP_DMA;
-
-   ret = (void *)__get_free_pages(gfp,  get_order(size));
-
-   if (ret != NULL) {
-   memset(ret, 0, size);
-   *dma_handle = virt_to_phys(ret);
-   }
-   return ret;
-}
-
-static void v32_dma_free(struct device *dev, size_t size, void *vaddr,
-   dma_addr_t dma_handle, unsigned long attrs)
-{
-   free_pages((unsigned long)vaddr, get_order(size));
-}
-
-static inline dma_addr_t v32_dma_map_page(struct device *dev,
-   struct page *page, unsigned long offset, size_t size,
-   enum dma_data_direction direction, unsigned long attrs)
-{
-   return page_to_phys(page) + offset;
-}
-
-static inline int v32_dma_map_sg(struct device *dev, struct scatterlist *sg,
-   int nents, enum dma_data_direction direction,
-   unsigned long attrs)
-{
-   printk("Map sg\n");
-   return nents;
-}
-
-static inline int v32_dma_supported(struct device *dev, u64 mask)
-{
-/*
- * we fall back to GFP_DMA when the mask isn't all 1s,
- * so we can't guarantee allocations that must be
- * within a tighter range than GFP_DMA..
- */
-if (mask < 0x00ff)
-return 0;
-   return 1;
-}
-
-const struct dma_map_ops v32_dma_ops = {
-   .alloc  = v32_dma_alloc,
-   .free   = v32_dma_free,
-   .map_page   = v32_dma_map_page,
-   .map_sg = v32_dma_map_sg,
-   .dma_supported  = v32_dma_supported,
-   .is_phys= true,
-};
-EXPORT_SYMBOL(v32_dma_ops);
diff --git a/arch/cris/include/asm/Kbuild b/arch/cris/include/asm/Kbuild
index 460349cb147f..8cf45ac30c1b 100644
--- a/arch/cris/include/asm/Kbuild
+++ b/arch/cris/include/asm/Kbuild
@@ -5,6 +5,7 @@ generic-y += cmpxchg.h
 generic-y += current.h
 generic-y += device.h
 generic-y += div64.h
+generic-y += dma-mapping.h
 generic-y += emergency-restart.h
 generic-y += exec.h
 generic-y += extable.h
diff --git a/arch/cris/include/asm/dma-mapping.h 
b/arch/cris/include/asm/dma-mapping.h
deleted file mode 100644
index 1553bdb30a0c..
--- a/arch/cris/include/asm/dma-mapping.h
+++ /dev/null
@@ -1,20 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _ASM_CRIS_DMA_MAPPING_H
-#define _ASM_CRIS_DMA_MAPPING_H
-
-#ifdef CONFIG_PCI
-extern const struct dma_map_ops v32_dma_ops;
-
-static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
-{
-   

[PATCH 33/67] dma-direct: reject too small dma masks

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 include/linux/dma-direct.h |  1 +
 lib/dma-direct.c   | 19 +++
 2 files changed, 20 insertions(+)

diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h
index 4788bf0bf683..bcdb1a3e4b1f 100644
--- a/include/linux/dma-direct.h
+++ b/include/linux/dma-direct.h
@@ -42,5 +42,6 @@ void *dma_direct_alloc(struct device *dev, size_t size, 
dma_addr_t *dma_handle,
gfp_t gfp, unsigned long attrs);
 void dma_direct_free(struct device *dev, size_t size, void *cpu_addr,
dma_addr_t dma_addr, unsigned long attrs);
+int dma_direct_supported(struct device *dev, u64 mask);
 
 #endif /* _LINUX_DMA_DIRECT_H */
diff --git a/lib/dma-direct.c b/lib/dma-direct.c
index 2e9b9494610c..5bb289483efc 100644
--- a/lib/dma-direct.c
+++ b/lib/dma-direct.c
@@ -123,6 +123,24 @@ static int dma_direct_map_sg(struct device *dev, struct 
scatterlist *sgl,
return nents;
 }
 
+int dma_direct_supported(struct device *dev, u64 mask)
+{
+#ifdef CONFIG_ZONE_DMA
+   if (mask < DMA_BIT_MASK(ARCH_ZONE_DMA_BITS))
+   return 0;
+#else
+   /*
+* Because 32-bit DMA masks are so common we expect every architecture
+* to be able to satisfy them - either by not supporting more physical
+* memory, or by providing a ZONE_DMA32.  If neither is the case, the
+* architecture needs to use an IOMMU instead of the direct mapping.
+*/
+   if (mask < DMA_BIT_MASK(32))
+   return 0;
+#endif
+   return 1;
+}
+
 static int dma_direct_mapping_error(struct device *dev, dma_addr_t dma_addr)
 {
return dma_addr == DIRECT_MAPPING_ERROR;
@@ -133,6 +151,7 @@ const struct dma_map_ops dma_direct_ops = {
.free   = dma_direct_free,
.map_page   = dma_direct_map_page,
.map_sg = dma_direct_map_sg,
+   .dma_supported  = dma_direct_supported,
.mapping_error  = dma_direct_mapping_error,
.is_phys= true,
 };
-- 
2.14.2

[PATCH 32/67] dma-direct: add support for allocation from ZONE_DMA and ZONE_DMA32

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 lib/dma-direct.c | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/lib/dma-direct.c b/lib/dma-direct.c
index 7e913728e099..2e9b9494610c 100644
--- a/lib/dma-direct.c
+++ b/lib/dma-direct.c
@@ -12,6 +12,14 @@
 
 #define DIRECT_MAPPING_ERROR   0
 
+/*
+ * Most architectures use ZONE_DMA for the first 16 Megabytes, but
+ * some use it for entirely different regions:
+ */
+#ifndef ARCH_ZONE_DMA_BITS
+#define ARCH_ZONE_DMA_BITS 24
+#endif
+
 static bool
 check_addr(struct device *dev, dma_addr_t dma_addr, size_t size,
const char *caller)
@@ -40,6 +48,12 @@ void *dma_direct_alloc(struct device *dev, size_t size, 
dma_addr_t *dma_handle,
int page_order = get_order(size);
struct page *page = NULL;
 
+   /* GFP_DMA32 and GFP_DMA are no ops without the corresponding zones: */
+   if (dev->coherent_dma_mask < DMA_BIT_MASK(32))
+   gfp |= GFP_DMA32;
+   else if (dev->coherent_dma_mask < DMA_BIT_MASK(ARCH_ZONE_DMA_BITS))
+   gfp |= GFP_DMA;
+
 again:
/* CMA can be used only in the context which permits sleeping */
if (gfpflags_allow_blocking(gfp)) {
-- 
2.14.2

[PATCH 31/67] dma-direct: make dma_direct_{alloc, free} available to other implementations

[Christoph Hellwig]

So that they don't need to indirect through the operation vector.

Signed-off-by: Christoph Hellwig 
---
 arch/arm/mm/dma-mapping-nommu.c | 9 +++--
 include/linux/dma-direct.h  | 5 +
 lib/dma-direct.c| 6 +++---
 3 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/arch/arm/mm/dma-mapping-nommu.c b/arch/arm/mm/dma-mapping-nommu.c
index 49e9831dc0f1..b4cf3e4e9d4a 100644
--- a/arch/arm/mm/dma-mapping-nommu.c
+++ b/arch/arm/mm/dma-mapping-nommu.c
@@ -11,7 +11,7 @@
 
 #include 
 #include 
-#include 
+#include 
 #include 
 
 #include 
@@ -39,7 +39,6 @@ static void *arm_nommu_dma_alloc(struct device *dev, size_t 
size,
 unsigned long attrs)
 
 {
-   const struct dma_map_ops *ops = _direct_ops;
void *ret;
 
/*
@@ -48,7 +47,7 @@ static void *arm_nommu_dma_alloc(struct device *dev, size_t 
size,
 */
 
if (attrs & DMA_ATTR_NON_CONSISTENT)
-   return ops->alloc(dev, size, dma_handle, gfp, attrs);
+   return dma_direct_alloc(dev, size, dma_handle, gfp, attrs);
 
ret = dma_alloc_from_global_coherent(size, dma_handle);
 
@@ -70,10 +69,8 @@ static void arm_nommu_dma_free(struct device *dev, size_t 
size,
   void *cpu_addr, dma_addr_t dma_addr,
   unsigned long attrs)
 {
-   const struct dma_map_ops *ops = _direct_ops;
-
if (attrs & DMA_ATTR_NON_CONSISTENT) {
-   ops->free(dev, size, cpu_addr, dma_addr, attrs);
+   dma_direct_free(dev, size, cpu_addr, dma_addr, attrs);
} else {
int ret = dma_release_from_global_coherent(get_order(size),
   cpu_addr);
diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h
index 10e924b7cba7..4788bf0bf683 100644
--- a/include/linux/dma-direct.h
+++ b/include/linux/dma-direct.h
@@ -38,4 +38,9 @@ static inline void dma_mark_clean(void *addr, size_t size)
 }
 #endif /* CONFIG_ARCH_HAS_DMA_MARK_CLEAN */
 
+void *dma_direct_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle,
+   gfp_t gfp, unsigned long attrs);
+void dma_direct_free(struct device *dev, size_t size, void *cpu_addr,
+   dma_addr_t dma_addr, unsigned long attrs);
+
 #endif /* _LINUX_DMA_DIRECT_H */
diff --git a/lib/dma-direct.c b/lib/dma-direct.c
index f8467cb3d89a..7e913728e099 100644
--- a/lib/dma-direct.c
+++ b/lib/dma-direct.c
@@ -33,8 +33,8 @@ static bool dma_coherent_ok(struct device *dev, phys_addr_t 
phys, size_t size)
return phys_to_dma(dev, phys) + size <= dev->coherent_dma_mask;
 }
 
-static void *dma_direct_alloc(struct device *dev, size_t size,
-   dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs)
+void *dma_direct_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle,
+   gfp_t gfp, unsigned long attrs)
 {
unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
int page_order = get_order(size);
@@ -71,7 +71,7 @@ static void *dma_direct_alloc(struct device *dev, size_t size,
return page_address(page);
 }
 
-static void dma_direct_free(struct device *dev, size_t size, void *cpu_addr,
+void dma_direct_free(struct device *dev, size_t size, void *cpu_addr,
dma_addr_t dma_addr, unsigned long attrs)
 {
unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
-- 
2.14.2

[PATCH 30/67] dma-direct: retry allocations using GFP_DMA for small masks

[Christoph Hellwig]

If we got back an allocation that wasn't inside the support coherent mask,
retry the allocation using GFP_DMA.

Based on the x86 code.

Signed-off-by: Christoph Hellwig 
---
 lib/dma-direct.c | 25 -
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/lib/dma-direct.c b/lib/dma-direct.c
index ab81de3ac1d3..f8467cb3d89a 100644
--- a/lib/dma-direct.c
+++ b/lib/dma-direct.c
@@ -28,6 +28,11 @@ check_addr(struct device *dev, dma_addr_t dma_addr, size_t 
size,
return true;
 }
 
+static bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)
+{
+   return phys_to_dma(dev, phys) + size <= dev->coherent_dma_mask;
+}
+
 static void *dma_direct_alloc(struct device *dev, size_t size,
dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs)
 {
@@ -35,11 +40,29 @@ static void *dma_direct_alloc(struct device *dev, size_t 
size,
int page_order = get_order(size);
struct page *page = NULL;
 
+again:
/* CMA can be used only in the context which permits sleeping */
-   if (gfpflags_allow_blocking(gfp))
+   if (gfpflags_allow_blocking(gfp)) {
page = dma_alloc_from_contiguous(dev, count, page_order, gfp);
+   if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) {
+   dma_release_from_contiguous(dev, page, count);
+   page = NULL;
+   }
+   }
if (!page)
page = alloc_pages_node(dev_to_node(dev), gfp, page_order);
+
+   if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) {
+   __free_pages(page, page_order);
+   page = NULL;
+
+   if (dev->coherent_dma_mask < DMA_BIT_MASK(32) &&
+   !(gfp & GFP_DMA)) {
+   gfp = (gfp & ~GFP_DMA32) | GFP_DMA;
+   goto again;
+   }
+   }
+
if (!page)
return NULL;
 
-- 
2.14.2

[PATCH 29/67] dma-direct: use node local allocations for coherent memory

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 lib/dma-direct.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/dma-direct.c b/lib/dma-direct.c
index d0266b39788b..ab81de3ac1d3 100644
--- a/lib/dma-direct.c
+++ b/lib/dma-direct.c
@@ -39,7 +39,7 @@ static void *dma_direct_alloc(struct device *dev, size_t size,
if (gfpflags_allow_blocking(gfp))
page = dma_alloc_from_contiguous(dev, count, page_order, gfp);
if (!page)
-   page = alloc_pages(gfp, page_order);
+   page = alloc_pages_node(dev_to_node(dev), gfp, page_order);
if (!page)
return NULL;
 
-- 
2.14.2

[PATCH 28/67] dma-direct: add support for CMA allocation

[Christoph Hellwig]

Try the CMA allocator for coherent allocations if supported.

Roughly modelled after the x86 code.

Signed-off-by: Christoph Hellwig 
---
 lib/dma-direct.c | 24 ++--
 1 file changed, 18 insertions(+), 6 deletions(-)

diff --git a/lib/dma-direct.c b/lib/dma-direct.c
index ddd9dcf4e663..d0266b39788b 100644
--- a/lib/dma-direct.c
+++ b/lib/dma-direct.c
@@ -7,6 +7,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 
 #define DIRECT_MAPPING_ERROR   0
@@ -30,19 +31,30 @@ check_addr(struct device *dev, dma_addr_t dma_addr, size_t 
size,
 static void *dma_direct_alloc(struct device *dev, size_t size,
dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs)
 {
-   void *ret;
+   unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+   int page_order = get_order(size);
+   struct page *page = NULL;
 
-   ret = (void *)__get_free_pages(gfp, get_order(size));
-   if (ret)
-   *dma_handle = phys_to_dma(dev, virt_to_phys(ret));
+   /* CMA can be used only in the context which permits sleeping */
+   if (gfpflags_allow_blocking(gfp))
+   page = dma_alloc_from_contiguous(dev, count, page_order, gfp);
+   if (!page)
+   page = alloc_pages(gfp, page_order);
+   if (!page)
+   return NULL;
 
-   return ret;
+   *dma_handle = phys_to_dma(dev, page_to_phys(page));
+   memset(page_address(page), 0, size);
+   return page_address(page);
 }
 
 static void dma_direct_free(struct device *dev, size_t size, void *cpu_addr,
dma_addr_t dma_addr, unsigned long attrs)
 {
-   free_pages((unsigned long)cpu_addr, get_order(size));
+   unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+
+   if (!dma_release_from_contiguous(dev, virt_to_page(cpu_addr), count))
+   free_pages((unsigned long)cpu_addr, get_order(size));
 }
 
 static dma_addr_t dma_direct_map_page(struct device *dev, struct page *page,
-- 
2.14.2

[PATCH 27/67] dma-direct: add dma address sanity checks

[Christoph Hellwig]

Roughly based on the x86 pci-nommu implementation.

Signed-off-by: Christoph Hellwig 
---
 lib/dma-direct.c | 32 +++-
 1 file changed, 31 insertions(+), 1 deletion(-)

diff --git a/lib/dma-direct.c b/lib/dma-direct.c
index 0e087650e86b..ddd9dcf4e663 100644
--- a/lib/dma-direct.c
+++ b/lib/dma-direct.c
@@ -9,6 +9,24 @@
 #include 
 #include 
 
+#define DIRECT_MAPPING_ERROR   0
+
+static bool
+check_addr(struct device *dev, dma_addr_t dma_addr, size_t size,
+   const char *caller)
+{
+   if (unlikely(dev && !dma_capable(dev, dma_addr, size))) {
+   if (*dev->dma_mask >= DMA_BIT_MASK(32)) {
+   dev_err(dev,
+   "%s: overflow %llx+%zu of device mask %llx\n",
+   caller, (long long)dma_addr, size,
+   (long long)*dev->dma_mask);
+   }
+   return false;
+   }
+   return true;
+}
+
 static void *dma_direct_alloc(struct device *dev, size_t size,
dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs)
 {
@@ -31,7 +49,11 @@ static dma_addr_t dma_direct_map_page(struct device *dev, 
struct page *page,
unsigned long offset, size_t size, enum dma_data_direction dir,
unsigned long attrs)
 {
-   return phys_to_dma(dev, page_to_phys(page)) + offset;
+   dma_addr_t dma_addr = phys_to_dma(dev, page_to_phys(page)) + offset;
+
+   if (!check_addr(dev, dma_addr, size, __func__))
+   return DIRECT_MAPPING_ERROR;
+   return dma_addr;
 }
 
 static int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl,
@@ -44,17 +66,25 @@ static int dma_direct_map_sg(struct device *dev, struct 
scatterlist *sgl,
BUG_ON(!sg_page(sg));
 
sg_dma_address(sg) = phys_to_dma(dev, sg_phys(sg));
+   if (!check_addr(dev, sg_dma_address(sg), sg->length, __func__))
+   return 0;
sg_dma_len(sg) = sg->length;
}
 
return nents;
 }
 
+static int dma_direct_mapping_error(struct device *dev, dma_addr_t dma_addr)
+{
+   return dma_addr == DIRECT_MAPPING_ERROR;
+}
+
 const struct dma_map_ops dma_direct_ops = {
.alloc  = dma_direct_alloc,
.free   = dma_direct_free,
.map_page   = dma_direct_map_page,
.map_sg = dma_direct_map_sg,
+   .mapping_error  = dma_direct_mapping_error,
.is_phys= true,
 };
 EXPORT_SYMBOL(dma_direct_ops);
-- 
2.14.2

[PATCH 26/67] dma-direct: use phys_to_dma

[Christoph Hellwig]

This means it uses whatever linear remapping scheme that the architecture
provides is used in the generic dma_direct ops.

Signed-off-by: Christoph Hellwig 
---
 lib/dma-direct.c | 18 +++---
 1 file changed, 7 insertions(+), 11 deletions(-)

diff --git a/lib/dma-direct.c b/lib/dma-direct.c
index 439db40854b7..0e087650e86b 100644
--- a/lib/dma-direct.c
+++ b/lib/dma-direct.c
@@ -1,12 +1,11 @@
 // SPDX-License-Identifier: GPL-2.0
 /*
- * lib/dma-noop.c
- *
- * DMA operations that map to physical addresses without flushing memory.
+ * DMA operations that map physical memory directly without using an IOMMU or
+ * flushing caches.
  */
 #include 
 #include 
-#include 
+#include 
 #include 
 #include 
 
@@ -17,7 +16,7 @@ static void *dma_direct_alloc(struct device *dev, size_t size,
 
ret = (void *)__get_free_pages(gfp, get_order(size));
if (ret)
-   *dma_handle = virt_to_phys(ret) - PFN_PHYS(dev->dma_pfn_offset);
+   *dma_handle = phys_to_dma(dev, virt_to_phys(ret));
 
return ret;
 }
@@ -32,7 +31,7 @@ static dma_addr_t dma_direct_map_page(struct device *dev, 
struct page *page,
unsigned long offset, size_t size, enum dma_data_direction dir,
unsigned long attrs)
 {
-   return page_to_phys(page) + offset - PFN_PHYS(dev->dma_pfn_offset);
+   return phys_to_dma(dev, page_to_phys(page)) + offset;
 }
 
 static int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl,
@@ -42,12 +41,9 @@ static int dma_direct_map_sg(struct device *dev, struct 
scatterlist *sgl,
struct scatterlist *sg;
 
for_each_sg(sgl, sg, nents, i) {
-   dma_addr_t offset = PFN_PHYS(dev->dma_pfn_offset);
-   void *va;
-
BUG_ON(!sg_page(sg));
-   va = sg_virt(sg);
-   sg_dma_address(sg) = (dma_addr_t)virt_to_phys(va) - offset;
+
+   sg_dma_address(sg) = phys_to_dma(dev, sg_phys(sg));
sg_dma_len(sg) = sg->length;
}
 
-- 
2.14.2

[PATCH 25/67] dma-direct: rename dma_noop to dma_direct

[Christoph Hellwig]

The trivial direct mapping implementation already does a virtual to
physical translation which isn't strictly a noop, and will soon learn
to do non-direct but linear physical to dma translations through the
device offset and a few small tricks.  Rename it to a better fitting
name.

Signed-off-by: Christoph Hellwig 
---
 MAINTAINERS|  2 +-
 arch/arm/Kconfig   |  2 +-
 arch/arm/include/asm/dma-mapping.h |  2 +-
 arch/arm/mm/dma-mapping-nommu.c|  8 
 arch/m32r/Kconfig  |  2 +-
 arch/riscv/Kconfig |  2 +-
 arch/s390/Kconfig  |  2 +-
 include/asm-generic/dma-mapping.h  |  2 +-
 include/linux/dma-mapping.h|  2 +-
 lib/Kconfig|  2 +-
 lib/Makefile   |  2 +-
 lib/{dma-noop.c => dma-direct.c}   | 35 +++
 12 files changed, 29 insertions(+), 34 deletions(-)
 rename lib/{dma-noop.c => dma-direct.c} (53%)

diff --git a/MAINTAINERS b/MAINTAINERS
index a8b35d9f41b2..b4005fe06e4c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4336,7 +4336,7 @@ T:git 
git://git.infradead.org/users/hch/dma-mapping.git
 W: http://git.infradead.org/users/hch/dma-mapping.git
 S: Supported
 F: lib/dma-debug.c
-F: lib/dma-noop.c
+F: lib/dma-direct.c
 F: lib/dma-virt.c
 F: drivers/base/dma-mapping.c
 F: drivers/base/dma-coherent.c
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 00d889a37965..430a0aa710d6 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -25,7 +25,7 @@ config ARM
select CLONE_BACKWARDS
select CPU_PM if (SUSPEND || CPU_IDLE)
select DCACHE_WORD_ACCESS if HAVE_EFFICIENT_UNALIGNED_ACCESS
-   select DMA_NOOP_OPS if !MMU
+   select DMA_DIRECT_OPS if !MMU
select EDAC_SUPPORT
select EDAC_ATOMIC_SCRUB
select GENERIC_ALLOCATOR
diff --git a/arch/arm/include/asm/dma-mapping.h 
b/arch/arm/include/asm/dma-mapping.h
index e5d9020c9ee1..8436f6ade57d 100644
--- a/arch/arm/include/asm/dma-mapping.h
+++ b/arch/arm/include/asm/dma-mapping.h
@@ -18,7 +18,7 @@ extern const struct dma_map_ops arm_coherent_dma_ops;
 
 static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
 {
-   return IS_ENABLED(CONFIG_MMU) ? _dma_ops : _noop_ops;
+   return IS_ENABLED(CONFIG_MMU) ? _dma_ops : _direct_ops;
 }
 
 #ifdef __arch_page_to_dma
diff --git a/arch/arm/mm/dma-mapping-nommu.c b/arch/arm/mm/dma-mapping-nommu.c
index 1cced700e45a..49e9831dc0f1 100644
--- a/arch/arm/mm/dma-mapping-nommu.c
+++ b/arch/arm/mm/dma-mapping-nommu.c
@@ -22,7 +22,7 @@
 #include "dma.h"
 
 /*
- *  dma_noop_ops is used if
+ *  dma_direct_ops is used if
  *   - MMU/MPU is off
  *   - cpu is v7m w/o cache support
  *   - device is coherent
@@ -39,7 +39,7 @@ static void *arm_nommu_dma_alloc(struct device *dev, size_t 
size,
 unsigned long attrs)
 
 {
-   const struct dma_map_ops *ops = _noop_ops;
+   const struct dma_map_ops *ops = _direct_ops;
void *ret;
 
/*
@@ -70,7 +70,7 @@ static void arm_nommu_dma_free(struct device *dev, size_t 
size,
   void *cpu_addr, dma_addr_t dma_addr,
   unsigned long attrs)
 {
-   const struct dma_map_ops *ops = _noop_ops;
+   const struct dma_map_ops *ops = _direct_ops;
 
if (attrs & DMA_ATTR_NON_CONSISTENT) {
ops->free(dev, size, cpu_addr, dma_addr, attrs);
@@ -214,7 +214,7 @@ EXPORT_SYMBOL(arm_nommu_dma_ops);
 
 static const struct dma_map_ops *arm_nommu_get_dma_map_ops(bool coherent)
 {
-   return coherent ? _noop_ops : _nommu_dma_ops;
+   return coherent ? _direct_ops : _nommu_dma_ops;
 }
 
 void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
diff --git a/arch/m32r/Kconfig b/arch/m32r/Kconfig
index 498398d915c1..dd84ee194579 100644
--- a/arch/m32r/Kconfig
+++ b/arch/m32r/Kconfig
@@ -19,7 +19,7 @@ config M32R
select MODULES_USE_ELF_RELA
select HAVE_DEBUG_STACKOVERFLOW
select CPU_NO_EFFICIENT_FFS
-   select DMA_NOOP_OPS
+   select DMA_DIRECT_OPS
select ARCH_NO_COHERENT_DMA_MMAP if !MMU
 
 config SBUS
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 2c6adf12713a..865e14f50c14 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -83,7 +83,7 @@ config PGTABLE_LEVELS
 config HAVE_KPROBES
def_bool n
 
-config DMA_NOOP_OPS
+config DMA_DIRECT_OPS
def_bool y
 
 menu "Platform type"
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 829c67986db7..9376637229c9 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -140,7 +140,7 @@ config S390
select HAVE_DEBUG_KMEMLEAK
select HAVE_DMA_API_DEBUG
select HAVE_DMA_CONTIGUOUS
-   select DMA_NOOP_OPS
+   select DMA_DIRECT_OPS
select HAVE_DYNAMIC_FTRACE
select HAVE_DYNAMIC_FTRACE_WITH_REGS
select 

[PATCH 24/67] dma-mapping: provide a generic asm/dma-mapping.h

[Christoph Hellwig]

For architectures that just use the generic dma_noop_ops we can provide
a generic version of dma-mapping.h.

Signed-off-by: Christoph Hellwig 
---
 MAINTAINERS  |  1 +
 arch/m32r/include/asm/Kbuild |  1 +
 arch/m32r/include/asm/dma-mapping.h  | 17 -
 arch/riscv/include/asm/Kbuild|  1 +
 arch/riscv/include/asm/dma-mapping.h | 30 --
 arch/s390/include/asm/Kbuild |  1 +
 arch/s390/include/asm/dma-mapping.h  | 17 -
 include/asm-generic/dma-mapping.h| 10 ++
 8 files changed, 14 insertions(+), 64 deletions(-)
 delete mode 100644 arch/m32r/include/asm/dma-mapping.h
 delete mode 100644 arch/riscv/include/asm/dma-mapping.h
 delete mode 100644 arch/s390/include/asm/dma-mapping.h
 create mode 100644 include/asm-generic/dma-mapping.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 7521b063b499..a8b35d9f41b2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4340,6 +4340,7 @@ F:lib/dma-noop.c
 F: lib/dma-virt.c
 F: drivers/base/dma-mapping.c
 F: drivers/base/dma-coherent.c
+F: include/asm-generic/dma-mapping.h
 F: include/linux/dma-direct.h
 F: include/linux/dma-mapping.h
 
diff --git a/arch/m32r/include/asm/Kbuild b/arch/m32r/include/asm/Kbuild
index 7e11b125c35e..ca83fda8177b 100644
--- a/arch/m32r/include/asm/Kbuild
+++ b/arch/m32r/include/asm/Kbuild
@@ -1,5 +1,6 @@
 generic-y += clkdev.h
 generic-y += current.h
+generic-y += dma-mapping.h
 generic-y += exec.h
 generic-y += extable.h
 generic-y += irq_work.h
diff --git a/arch/m32r/include/asm/dma-mapping.h 
b/arch/m32r/include/asm/dma-mapping.h
deleted file mode 100644
index 8967fb659691..
--- a/arch/m32r/include/asm/dma-mapping.h
+++ /dev/null
@@ -1,17 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _ASM_M32R_DMA_MAPPING_H
-#define _ASM_M32R_DMA_MAPPING_H
-
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-
-static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
-{
-   return _noop_ops;
-}
-
-#endif /* _ASM_M32R_DMA_MAPPING_H */
diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild
index 970460a0b492..197460ccbf21 100644
--- a/arch/riscv/include/asm/Kbuild
+++ b/arch/riscv/include/asm/Kbuild
@@ -7,6 +7,7 @@ generic-y += device.h
 generic-y += div64.h
 generic-y += dma.h
 generic-y += dma-contiguous.h
+generic-y += dma-mapping.h
 generic-y += emergency-restart.h
 generic-y += errno.h
 generic-y += exec.h
diff --git a/arch/riscv/include/asm/dma-mapping.h 
b/arch/riscv/include/asm/dma-mapping.h
deleted file mode 100644
index 73849e2cc761..
--- a/arch/riscv/include/asm/dma-mapping.h
+++ /dev/null
@@ -1,30 +0,0 @@
-/*
- * Copyright (C) 2003-2004 Hewlett-Packard Co
- * David Mosberger-Tang 
- * Copyright (C) 2012 ARM Ltd.
- * Copyright (C) 2016 SiFive, Inc.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program.  If not, see .
- */
-#ifndef __ASM_RISCV_DMA_MAPPING_H
-#define __ASM_RISCV_DMA_MAPPING_H
-
-/* Use ops->dma_mapping_error (if it exists) or assume success */
-// #undef DMA_ERROR_CODE
-
-static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
-{
-   return _noop_ops;
-}
-
-#endif /* __ASM_RISCV_DMA_MAPPING_H */
diff --git a/arch/s390/include/asm/Kbuild b/arch/s390/include/asm/Kbuild
index 048450869328..dade72be127b 100644
--- a/arch/s390/include/asm/Kbuild
+++ b/arch/s390/include/asm/Kbuild
@@ -4,6 +4,7 @@ generic-y += cacheflush.h
 generic-y += clkdev.h
 generic-y += device.h
 generic-y += dma-contiguous.h
+generic-y += dma-mapping.h
 generic-y += div64.h
 generic-y += emergency-restart.h
 generic-y += export.h
diff --git a/arch/s390/include/asm/dma-mapping.h 
b/arch/s390/include/asm/dma-mapping.h
deleted file mode 100644
index bdc2455483f6..
--- a/arch/s390/include/asm/dma-mapping.h
+++ /dev/null
@@ -1,17 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _ASM_S390_DMA_MAPPING_H
-#define _ASM_S390_DMA_MAPPING_H
-
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-
-static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
-{
-   return _noop_ops;
-}
-
-#endif /* _ASM_S390_DMA_MAPPING_H */
diff --git a/include/asm-generic/dma-mapping.h 
b/include/asm-generic/dma-mapping.h
new file mode 100644
index ..164031531d85
--- /dev/null
+++ b/include/asm-generic/dma-mapping.h
@@ -0,0 

[PATCH 23/67] dma-mapping: add an arch_dma_supported hook

[Christoph Hellwig]

To implement the x86 forbid_dac and iommu_sac_force we want an arch hook
so that it can apply the global options across all dma_map_ops
implementations.

Signed-off-by: Christoph Hellwig 
---
 arch/x86/include/asm/dma-mapping.h |  3 +++
 arch/x86/kernel/pci-dma.c  | 19 ---
 include/linux/dma-mapping.h| 11 +++
 3 files changed, 26 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/dma-mapping.h 
b/arch/x86/include/asm/dma-mapping.h
index dfdc9357a349..6277c83c0eb1 100644
--- a/arch/x86/include/asm/dma-mapping.h
+++ b/arch/x86/include/asm/dma-mapping.h
@@ -30,6 +30,9 @@ static inline const struct dma_map_ops 
*get_arch_dma_ops(struct bus_type *bus)
return dma_ops;
 }
 
+int arch_dma_supported(struct device *dev, u64 mask);
+#define arch_dma_supported arch_dma_supported
+
 bool arch_dma_alloc_attrs(struct device **dev, gfp_t *gfp);
 #define arch_dma_alloc_attrs arch_dma_alloc_attrs
 
diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
index 61a8f1cb3829..df7ab02f959f 100644
--- a/arch/x86/kernel/pci-dma.c
+++ b/arch/x86/kernel/pci-dma.c
@@ -215,7 +215,7 @@ static __init int iommu_setup(char *p)
 }
 early_param("iommu", iommu_setup);
 
-int x86_dma_supported(struct device *dev, u64 mask)
+int arch_dma_supported(struct device *dev, u64 mask)
 {
 #ifdef CONFIG_PCI
if (mask > 0x && forbid_dac > 0) {
@@ -224,12 +224,6 @@ int x86_dma_supported(struct device *dev, u64 mask)
}
 #endif
 
-   /* Copied from i386. Doesn't make much sense, because it will
-  only work for pci_alloc_coherent.
-  The caller just has to use GFP_DMA in this case. */
-   if (mask < DMA_BIT_MASK(24))
-   return 0;
-
/* Tell the device to use SAC when IOMMU force is on.  This
   allows the driver to use cheaper accesses in some cases.
 
@@ -249,6 +243,17 @@ int x86_dma_supported(struct device *dev, u64 mask)
 
return 1;
 }
+EXPORT_SYMBOL(arch_dma_supported);
+
+int x86_dma_supported(struct device *dev, u64 mask)
+{
+   /* Copied from i386. Doesn't make much sense, because it will
+  only work for pci_alloc_coherent.
+  The caller just has to use GFP_DMA in this case. */
+   if (mask < DMA_BIT_MASK(24))
+   return 0;
+   return 1;
+}
 
 static int __init pci_iommu_init(void)
 {
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index fd5197af882a..72568bf4fc12 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -583,6 +583,14 @@ static inline int dma_mapping_error(struct device *dev, 
dma_addr_t dma_addr)
return 0;
 }
 
+/*
+ * This is a hack for the legacy x86 forbid_dac and iommu_sac_force. Please
+ * don't use this is new code.
+ */
+#ifndef arch_dma_supported
+#define arch_dma_supported(dev, mask)  (1)
+#endif
+
 static inline void dma_check_mask(struct device *dev, u64 mask)
 {
if (sme_active() && (mask < (((u64)sme_get_me_mask() << 1) - 1)))
@@ -595,6 +603,9 @@ static inline int dma_supported(struct device *dev, u64 
mask)
 
if (!ops)
return 0;
+   if (!arch_dma_supported(dev, mask))
+   return 0;
+
if (!ops->dma_supported)
return 1;
return ops->dma_supported(dev, mask);
-- 
2.14.2

[PATCH 22/67] dma-mapping: clear harmful GFP_* flags in common code

[Christoph Hellwig]

Life the code from x86 so that we behave consistently.  In the future we
should probably warn if any of these is set.

Signed-off-by: Christoph Hellwig 
---
 arch/cris/arch-v32/drivers/pci/dma.c  | 3 ---
 arch/h8300/kernel/dma.c   | 3 ---
 arch/m68k/kernel/dma.c| 2 --
 arch/mips/cavium-octeon/dma-octeon.c  | 3 ---
 arch/mips/loongson64/common/dma-swiotlb.c | 3 ---
 arch/mips/mm/dma-default.c| 3 ---
 arch/mips/netlogic/common/nlm-dma.c   | 3 ---
 arch/mn10300/mm/dma-alloc.c   | 3 ---
 arch/nios2/mm/dma-mapping.c   | 3 ---
 arch/powerpc/kernel/dma.c | 3 ---
 arch/x86/kernel/pci-dma.c | 2 --
 include/linux/dma-mapping.h   | 7 +++
 12 files changed, 7 insertions(+), 31 deletions(-)

diff --git a/arch/cris/arch-v32/drivers/pci/dma.c 
b/arch/cris/arch-v32/drivers/pci/dma.c
index aa16ce27e036..c7e3056885d3 100644
--- a/arch/cris/arch-v32/drivers/pci/dma.c
+++ b/arch/cris/arch-v32/drivers/pci/dma.c
@@ -22,9 +22,6 @@ static void *v32_dma_alloc(struct device *dev, size_t size,
 {
void *ret;
 
-   /* ignore region specifiers */
-   gfp &= ~(__GFP_DMA | __GFP_HIGHMEM);
-
if (dev == NULL || (dev->coherent_dma_mask < 0x))
gfp |= GFP_DMA;
 
diff --git a/arch/h8300/kernel/dma.c b/arch/h8300/kernel/dma.c
index 0e92214310c4..4e27b74df973 100644
--- a/arch/h8300/kernel/dma.c
+++ b/arch/h8300/kernel/dma.c
@@ -16,9 +16,6 @@ static void *dma_alloc(struct device *dev, size_t size,
 {
void *ret;
 
-   /* ignore region specifiers */
-   gfp &= ~(__GFP_DMA | __GFP_HIGHMEM);
-
if (dev == NULL || (*dev->dma_mask < 0x))
gfp |= GFP_DMA;
ret = (void *)__get_free_pages(gfp, get_order(size));
diff --git a/arch/m68k/kernel/dma.c b/arch/m68k/kernel/dma.c
index e0167418072b..2f3492e8295c 100644
--- a/arch/m68k/kernel/dma.c
+++ b/arch/m68k/kernel/dma.c
@@ -76,8 +76,6 @@ static void *m68k_dma_alloc(struct device *dev, size_t size,
dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs)
 {
void *ret;
-   /* ignore region specifiers */
-   gfp &= ~(__GFP_DMA | __GFP_HIGHMEM);
 
if (dev == NULL || (*dev->dma_mask < 0x))
gfp |= GFP_DMA;
diff --git a/arch/mips/cavium-octeon/dma-octeon.c 
b/arch/mips/cavium-octeon/dma-octeon.c
index c64bd87f0b6e..5baf79fce643 100644
--- a/arch/mips/cavium-octeon/dma-octeon.c
+++ b/arch/mips/cavium-octeon/dma-octeon.c
@@ -161,9 +161,6 @@ static void *octeon_dma_alloc_coherent(struct device *dev, 
size_t size,
 {
void *ret;
 
-   /* ignore region specifiers */
-   gfp &= ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM);
-
if (IS_ENABLED(CONFIG_ZONE_DMA) && dev == NULL)
gfp |= __GFP_DMA;
else if (IS_ENABLED(CONFIG_ZONE_DMA) &&
diff --git a/arch/mips/loongson64/common/dma-swiotlb.c 
b/arch/mips/loongson64/common/dma-swiotlb.c
index ef07740cee61..15388c24a504 100644
--- a/arch/mips/loongson64/common/dma-swiotlb.c
+++ b/arch/mips/loongson64/common/dma-swiotlb.c
@@ -15,9 +15,6 @@ static void *loongson_dma_alloc_coherent(struct device *dev, 
size_t size,
 {
void *ret;
 
-   /* ignore region specifiers */
-   gfp &= ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM);
-
if ((IS_ENABLED(CONFIG_ISA) && dev == NULL) ||
(IS_ENABLED(CONFIG_ZONE_DMA) &&
 dev->coherent_dma_mask < DMA_BIT_MASK(32)))
diff --git a/arch/mips/mm/dma-default.c b/arch/mips/mm/dma-default.c
index 3cd93e0c7a29..6f6b1399e98e 100644
--- a/arch/mips/mm/dma-default.c
+++ b/arch/mips/mm/dma-default.c
@@ -93,9 +93,6 @@ static gfp_t massage_gfp_flags(const struct device *dev, 
gfp_t gfp)
 {
gfp_t dma_flag;
 
-   /* ignore region specifiers */
-   gfp &= ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM);
-
 #ifdef CONFIG_ISA
if (dev == NULL)
dma_flag = __GFP_DMA;
diff --git a/arch/mips/netlogic/common/nlm-dma.c 
b/arch/mips/netlogic/common/nlm-dma.c
index 0ec9d9da6d51..49c975b6aa28 100644
--- a/arch/mips/netlogic/common/nlm-dma.c
+++ b/arch/mips/netlogic/common/nlm-dma.c
@@ -47,9 +47,6 @@ static char *nlm_swiotlb;
 static void *nlm_dma_alloc_coherent(struct device *dev, size_t size,
dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs)
 {
-   /* ignore region specifiers */
-   gfp &= ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM);
-
 #ifdef CONFIG_ZONE_DMA32
if (dev->coherent_dma_mask <= DMA_BIT_MASK(32))
gfp |= __GFP_DMA32;
diff --git a/arch/mn10300/mm/dma-alloc.c b/arch/mn10300/mm/dma-alloc.c
index 55876a87c247..2629f1f4b04e 100644
--- a/arch/mn10300/mm/dma-alloc.c
+++ b/arch/mn10300/mm/dma-alloc.c
@@ -37,9 +37,6 @@ static void *mn10300_dma_alloc(struct device *dev, size_t 
size,
goto done;
}
 
-   /* ignore region specifiers */
-   gfp &= ~(__GFP_DMA | __GFP_HIGHMEM);
-

[PATCH 21/67] dma-mapping: warn when there is no coherent_dma_mask

[Christoph Hellwig]

These days all devices should have a DMA coherent mask, and most dma_ops
implementations rely on that fact.  But just to be sure add an assert to
ring the warning bell if that is not the case.

Signed-off-by: Christoph Hellwig 
---
 include/linux/dma-mapping.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index e77e2dec4723..2779d544485c 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -520,6 +520,7 @@ static inline void *dma_alloc_attrs(struct device *dev, 
size_t size,
void *cpu_addr;
 
BUG_ON(!ops);
+   WARN_ON_ONCE(!dev->coherent_dma_mask);
 
if (dma_alloc_from_dev_coherent(dev, size, dma_handle, _addr))
return cpu_addr;
-- 
2.14.2

[PATCH 20/67] s390: move s390_pci_dma_ops to asm/pci_dma.h

[Christoph Hellwig]

This is not needed in drivers, so move it to a private header.

Signed-off-by: Christoph Hellwig 
---
 arch/s390/include/asm/dma-mapping.h | 2 --
 arch/s390/include/asm/pci_dma.h | 3 +++
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/s390/include/asm/dma-mapping.h 
b/arch/s390/include/asm/dma-mapping.h
index 2ec7240c1ada..bdc2455483f6 100644
--- a/arch/s390/include/asm/dma-mapping.h
+++ b/arch/s390/include/asm/dma-mapping.h
@@ -9,8 +9,6 @@
 #include 
 #include 
 
-extern const struct dma_map_ops s390_pci_dma_ops;
-
 static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
 {
return _noop_ops;
diff --git a/arch/s390/include/asm/pci_dma.h b/arch/s390/include/asm/pci_dma.h
index e8d9161fa17a..419fac7a62c0 100644
--- a/arch/s390/include/asm/pci_dma.h
+++ b/arch/s390/include/asm/pci_dma.h
@@ -201,4 +201,7 @@ void dma_cleanup_tables(unsigned long *);
 unsigned long *dma_walk_cpu_trans(unsigned long *rto, dma_addr_t dma_addr);
 void dma_update_cpu_trans(unsigned long *entry, void *page_addr, int flags);
 
+extern const struct dma_map_ops s390_pci_dma_ops;
+
+
 #endif
-- 
2.14.2

[PATCH 19/67] microblaze: remove the dead !NOT_COHERENT_CACHE dma code

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 arch/microblaze/kernel/dma.c | 28 
 1 file changed, 28 deletions(-)

diff --git a/arch/microblaze/kernel/dma.c b/arch/microblaze/kernel/dma.c
index 49b09648679b..031d889670f5 100644
--- a/arch/microblaze/kernel/dma.c
+++ b/arch/microblaze/kernel/dma.c
@@ -15,42 +15,18 @@
 #include 
 #include 
 
-#define NOT_COHERENT_CACHE
-
 static void *dma_nommu_alloc_coherent(struct device *dev, size_t size,
   dma_addr_t *dma_handle, gfp_t flag,
   unsigned long attrs)
 {
-#ifdef NOT_COHERENT_CACHE
return consistent_alloc(flag, size, dma_handle);
-#else
-   void *ret;
-   struct page *page;
-   int node = dev_to_node(dev);
-
-   /* ignore region specifiers */
-   flag  &= ~(__GFP_HIGHMEM);
-
-   page = alloc_pages_node(node, flag, get_order(size));
-   if (page == NULL)
-   return NULL;
-   ret = page_address(page);
-   memset(ret, 0, size);
-   *dma_handle = virt_to_phys(ret);
-
-   return ret;
-#endif
 }
 
 static void dma_nommu_free_coherent(struct device *dev, size_t size,
 void *vaddr, dma_addr_t dma_handle,
 unsigned long attrs)
 {
-#ifdef NOT_COHERENT_CACHE
consistent_free(size, vaddr);
-#else
-   free_pages((unsigned long)vaddr, get_order(size));
-#endif
 }
 
 static inline void __dma_sync(unsigned long paddr,
@@ -186,12 +162,8 @@ int dma_nommu_mmap_coherent(struct device *dev, struct 
vm_area_struct *vma,
if (off >= count || user_count > (count - off))
return -ENXIO;
 
-#ifdef NOT_COHERENT_CACHE
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
pfn = consistent_virt_to_pfn(cpu_addr);
-#else
-   pfn = virt_to_pfn(cpu_addr);
-#endif
return remap_pfn_range(vma, vma->vm_start, pfn + off,
   vma->vm_end - vma->vm_start, vma->vm_page_prot);
 #else
-- 
2.14.2

[PATCH 18/67] microblaze: remove dma_nommu_dma_supported

[Christoph Hellwig]

Always returning 1 is the same behavior as not supplying a method at all.

Signed-off-by: Christoph Hellwig 
---
 arch/microblaze/kernel/dma.c | 6 --
 arch/parisc/kernel/pci-dma.c | 7 ---
 2 files changed, 13 deletions(-)

diff --git a/arch/microblaze/kernel/dma.c b/arch/microblaze/kernel/dma.c
index 364b0ac41452..49b09648679b 100644
--- a/arch/microblaze/kernel/dma.c
+++ b/arch/microblaze/kernel/dma.c
@@ -89,11 +89,6 @@ static int dma_nommu_map_sg(struct device *dev, struct 
scatterlist *sgl,
return nents;
 }
 
-static int dma_nommu_dma_supported(struct device *dev, u64 mask)
-{
-   return 1;
-}
-
 static inline dma_addr_t dma_nommu_map_page(struct device *dev,
 struct page *page,
 unsigned long offset,
@@ -209,7 +204,6 @@ const struct dma_map_ops dma_nommu_ops = {
.free   = dma_nommu_free_coherent,
.mmap   = dma_nommu_mmap_coherent,
.map_sg = dma_nommu_map_sg,
-   .dma_supported  = dma_nommu_dma_supported,
.map_page   = dma_nommu_map_page,
.unmap_page = dma_nommu_unmap_page,
.sync_single_for_cpu= dma_nommu_sync_single_for_cpu,
diff --git a/arch/parisc/kernel/pci-dma.c b/arch/parisc/kernel/pci-dma.c
index 6ad9aed3d025..2a05457f7aab 100644
--- a/arch/parisc/kernel/pci-dma.c
+++ b/arch/parisc/kernel/pci-dma.c
@@ -75,11 +75,6 @@ void dump_resmap(void)
 static inline void dump_resmap(void) {;}
 #endif
 
-static int pa11_dma_supported( struct device *dev, u64 mask)
-{
-   return 1;
-}
-
 static inline int map_pte_uncached(pte_t * pte,
unsigned long vaddr,
unsigned long size, unsigned long *paddr_ptr)
@@ -579,7 +574,6 @@ static void pa11_dma_cache_sync(struct device *dev, void 
*vaddr, size_t size,
 }
 
 const struct dma_map_ops pcxl_dma_ops = {
-   .dma_supported =pa11_dma_supported,
.alloc =pa11_dma_alloc,
.free = pa11_dma_free,
.map_page = pa11_dma_map_page,
@@ -617,7 +611,6 @@ static void pcx_dma_free(struct device *dev, size_t size, 
void *vaddr,
 }
 
 const struct dma_map_ops pcx_dma_ops = {
-   .dma_supported =pa11_dma_supported,
.alloc =pcx_dma_alloc,
.free = pcx_dma_free,
.map_page = pa11_dma_map_page,
-- 
2.14.2

[PATCH 17/67] microblaze: rename dma_direct to dma_microblaze

[Christoph Hellwig]

This frees the dma_direct_* namespace for a generic implementation.

Signed-off-by: Christoph Hellwig 
---
 arch/microblaze/include/asm/dma-mapping.h |  4 +--
 arch/microblaze/kernel/dma.c  | 50 +++
 2 files changed, 27 insertions(+), 27 deletions(-)

diff --git a/arch/microblaze/include/asm/dma-mapping.h 
b/arch/microblaze/include/asm/dma-mapping.h
index 6b9ea39405b8..add50c1373bf 100644
--- a/arch/microblaze/include/asm/dma-mapping.h
+++ b/arch/microblaze/include/asm/dma-mapping.h
@@ -18,11 +18,11 @@
 /*
  * Available generic sets of operations
  */
-extern const struct dma_map_ops dma_direct_ops;
+extern const struct dma_map_ops dma_nommu_ops;
 
 static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
 {
-   return _direct_ops;
+   return _nommu_ops;
 }
 
 #endif /* _ASM_MICROBLAZE_DMA_MAPPING_H */
diff --git a/arch/microblaze/kernel/dma.c b/arch/microblaze/kernel/dma.c
index 2a9a0ec14c46..364b0ac41452 100644
--- a/arch/microblaze/kernel/dma.c
+++ b/arch/microblaze/kernel/dma.c
@@ -17,7 +17,7 @@
 
 #define NOT_COHERENT_CACHE
 
-static void *dma_direct_alloc_coherent(struct device *dev, size_t size,
+static void *dma_nommu_alloc_coherent(struct device *dev, size_t size,
   dma_addr_t *dma_handle, gfp_t flag,
   unsigned long attrs)
 {
@@ -42,7 +42,7 @@ static void *dma_direct_alloc_coherent(struct device *dev, 
size_t size,
 #endif
 }
 
-static void dma_direct_free_coherent(struct device *dev, size_t size,
+static void dma_nommu_free_coherent(struct device *dev, size_t size,
 void *vaddr, dma_addr_t dma_handle,
 unsigned long attrs)
 {
@@ -69,7 +69,7 @@ static inline void __dma_sync(unsigned long paddr,
}
 }
 
-static int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl,
+static int dma_nommu_map_sg(struct device *dev, struct scatterlist *sgl,
 int nents, enum dma_data_direction direction,
 unsigned long attrs)
 {
@@ -89,12 +89,12 @@ static int dma_direct_map_sg(struct device *dev, struct 
scatterlist *sgl,
return nents;
 }
 
-static int dma_direct_dma_supported(struct device *dev, u64 mask)
+static int dma_nommu_dma_supported(struct device *dev, u64 mask)
 {
return 1;
 }
 
-static inline dma_addr_t dma_direct_map_page(struct device *dev,
+static inline dma_addr_t dma_nommu_map_page(struct device *dev,
 struct page *page,
 unsigned long offset,
 size_t size,
@@ -106,7 +106,7 @@ static inline dma_addr_t dma_direct_map_page(struct device 
*dev,
return page_to_phys(page) + offset;
 }
 
-static inline void dma_direct_unmap_page(struct device *dev,
+static inline void dma_nommu_unmap_page(struct device *dev,
 dma_addr_t dma_address,
 size_t size,
 enum dma_data_direction direction,
@@ -122,7 +122,7 @@ static inline void dma_direct_unmap_page(struct device *dev,
 }
 
 static inline void
-dma_direct_sync_single_for_cpu(struct device *dev,
+dma_nommu_sync_single_for_cpu(struct device *dev,
   dma_addr_t dma_handle, size_t size,
   enum dma_data_direction direction)
 {
@@ -136,7 +136,7 @@ dma_direct_sync_single_for_cpu(struct device *dev,
 }
 
 static inline void
-dma_direct_sync_single_for_device(struct device *dev,
+dma_nommu_sync_single_for_device(struct device *dev,
  dma_addr_t dma_handle, size_t size,
  enum dma_data_direction direction)
 {
@@ -150,7 +150,7 @@ dma_direct_sync_single_for_device(struct device *dev,
 }
 
 static inline void
-dma_direct_sync_sg_for_cpu(struct device *dev,
+dma_nommu_sync_sg_for_cpu(struct device *dev,
   struct scatterlist *sgl, int nents,
   enum dma_data_direction direction)
 {
@@ -164,7 +164,7 @@ dma_direct_sync_sg_for_cpu(struct device *dev,
 }
 
 static inline void
-dma_direct_sync_sg_for_device(struct device *dev,
+dma_nommu_sync_sg_for_device(struct device *dev,
  struct scatterlist *sgl, int nents,
  enum dma_data_direction direction)
 {
@@ -178,7 +178,7 @@ dma_direct_sync_sg_for_device(struct device *dev,
 }
 
 static
-int dma_direct_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
+int dma_nommu_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
 void *cpu_addr, dma_addr_t handle, size_t size,
 unsigned long attrs)
 {
@@ -204,21 +204,21 @@ int dma_direct_mmap_coherent(struct device *dev, struct 

[PATCH 16/67] powerpc: rename dma_direct_ to dma_nommu_

[Christoph Hellwig]

We want to use the dma_direct_ namespace for a generic implementation,
so rename powerpc to the second best choice: dma_nommu_.

Signed-off-by: Christoph Hellwig 
---
 arch/powerpc/include/asm/dma-mapping.h|  8 ++--
 arch/powerpc/kernel/dma-iommu.c   |  2 +-
 arch/powerpc/kernel/dma-swiotlb.c |  6 +--
 arch/powerpc/kernel/dma.c | 68 +++
 arch/powerpc/kernel/pci-common.c  |  2 +-
 arch/powerpc/kernel/setup-common.c|  2 +-
 arch/powerpc/platforms/cell/iommu.c   | 28 ++---
 arch/powerpc/platforms/pasemi/iommu.c |  2 +-
 arch/powerpc/platforms/pasemi/setup.c |  2 +-
 arch/powerpc/platforms/powernv/pci-ioda.c |  4 +-
 arch/powerpc/platforms/pseries/iommu.c|  2 +-
 arch/powerpc/platforms/pseries/vio.c  |  2 +-
 arch/powerpc/sysdev/dart_iommu.c  |  4 +-
 arch/powerpc/sysdev/fsl_pci.c |  2 +-
 drivers/misc/cxl/vphb.c   |  2 +-
 15 files changed, 68 insertions(+), 68 deletions(-)

diff --git a/arch/powerpc/include/asm/dma-mapping.h 
b/arch/powerpc/include/asm/dma-mapping.h
index f6ab51205a85..8fa394520af6 100644
--- a/arch/powerpc/include/asm/dma-mapping.h
+++ b/arch/powerpc/include/asm/dma-mapping.h
@@ -19,13 +19,13 @@
 #include 
 
 /* Some dma direct funcs must be visible for use in other dma_ops */
-extern void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+extern void *__dma_nommu_alloc_coherent(struct device *dev, size_t size,
 dma_addr_t *dma_handle, gfp_t flag,
 unsigned long attrs);
-extern void __dma_direct_free_coherent(struct device *dev, size_t size,
+extern void __dma_nommu_free_coherent(struct device *dev, size_t size,
   void *vaddr, dma_addr_t dma_handle,
   unsigned long attrs);
-extern int dma_direct_mmap_coherent(struct device *dev,
+extern int dma_nommu_mmap_coherent(struct device *dev,
struct vm_area_struct *vma,
void *cpu_addr, dma_addr_t handle,
size_t size, unsigned long attrs);
@@ -73,7 +73,7 @@ static inline unsigned long device_to_mask(struct device *dev)
 #ifdef CONFIG_PPC64
 extern struct dma_map_ops dma_iommu_ops;
 #endif
-extern const struct dma_map_ops dma_direct_ops;
+extern const struct dma_map_ops dma_nommu_ops;
 
 static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
 {
diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
index 66f33e7f8d40..f9fe2080ceb9 100644
--- a/arch/powerpc/kernel/dma-iommu.c
+++ b/arch/powerpc/kernel/dma-iommu.c
@@ -114,7 +114,7 @@ int dma_iommu_mapping_error(struct device *dev, dma_addr_t 
dma_addr)
 struct dma_map_ops dma_iommu_ops = {
.alloc  = dma_iommu_alloc_coherent,
.free   = dma_iommu_free_coherent,
-   .mmap   = dma_direct_mmap_coherent,
+   .mmap   = dma_nommu_mmap_coherent,
.map_sg = dma_iommu_map_sg,
.unmap_sg   = dma_iommu_unmap_sg,
.dma_supported  = dma_iommu_dma_supported,
diff --git a/arch/powerpc/kernel/dma-swiotlb.c 
b/arch/powerpc/kernel/dma-swiotlb.c
index d0ea7860e02b..f1e99b9cee97 100644
--- a/arch/powerpc/kernel/dma-swiotlb.c
+++ b/arch/powerpc/kernel/dma-swiotlb.c
@@ -47,9 +47,9 @@ static u64 swiotlb_powerpc_get_required(struct device *dev)
  * for everything else.
  */
 const struct dma_map_ops swiotlb_dma_ops = {
-   .alloc = __dma_direct_alloc_coherent,
-   .free = __dma_direct_free_coherent,
-   .mmap = dma_direct_mmap_coherent,
+   .alloc = __dma_nommu_alloc_coherent,
+   .free = __dma_nommu_free_coherent,
+   .mmap = dma_nommu_mmap_coherent,
.map_sg = swiotlb_map_sg_attrs,
.unmap_sg = swiotlb_unmap_sg_attrs,
.dma_supported = swiotlb_dma_supported,
diff --git a/arch/powerpc/kernel/dma.c b/arch/powerpc/kernel/dma.c
index df0e7bb97ab5..5d49da094a93 100644
--- a/arch/powerpc/kernel/dma.c
+++ b/arch/powerpc/kernel/dma.c
@@ -40,7 +40,7 @@ static u64 __maybe_unused get_pfn_limit(struct device *dev)
return pfn;
 }
 
-static int dma_direct_dma_supported(struct device *dev, u64 mask)
+static int dma_nommu_dma_supported(struct device *dev, u64 mask)
 {
 #ifdef CONFIG_PPC64
u64 limit = get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
@@ -62,7 +62,7 @@ static int dma_direct_dma_supported(struct device *dev, u64 
mask)
 #endif
 }
 
-void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+void *__dma_nommu_alloc_coherent(struct device *dev, size_t size,
  dma_addr_t *dma_handle, gfp_t flag,
  unsigned long attrs)
 {
@@ -119,7 +119,7 @@ void *__dma_direct_alloc_coherent(struct 

[PATCH 15/67] hexagon: use the generic dma_capable helper

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 arch/hexagon/include/asm/dma-mapping.h | 7 ---
 arch/hexagon/kernel/dma.c  | 1 +
 2 files changed, 1 insertion(+), 7 deletions(-)

diff --git a/arch/hexagon/include/asm/dma-mapping.h 
b/arch/hexagon/include/asm/dma-mapping.h
index 5208de242e79..263f6acbfb0f 100644
--- a/arch/hexagon/include/asm/dma-mapping.h
+++ b/arch/hexagon/include/asm/dma-mapping.h
@@ -37,11 +37,4 @@ static inline const struct dma_map_ops 
*get_arch_dma_ops(struct bus_type *bus)
return dma_ops;
 }
 
-static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t 
size)
-{
-   if (!dev->dma_mask)
-   return 0;
-   return addr + size - 1 <= *dev->dma_mask;
-}
-
 #endif
diff --git a/arch/hexagon/kernel/dma.c b/arch/hexagon/kernel/dma.c
index 3683bb9c05a2..c1d24e37807c 100644
--- a/arch/hexagon/kernel/dma.c
+++ b/arch/hexagon/kernel/dma.c
@@ -19,6 +19,7 @@
  */
 
 #include 
+#include 
 #include 
 #include 
 #include 
-- 
2.14.2

[PATCH 14/67] dma-mapping: move dma_mark_clean to dma-direct.h

[Christoph Hellwig]

And unlike the other helpers we don't require a  as
this helper is a special case for ia64 only, and this keeps it as
simple as possible.

Signed-off-by: Christoph Hellwig 
---
 arch/arm/include/asm/dma-mapping.h   | 2 --
 arch/arm64/include/asm/dma-mapping.h | 4 
 arch/ia64/Kconfig| 1 +
 arch/ia64/include/asm/dma.h  | 2 --
 arch/mips/include/asm/dma-mapping.h  | 2 --
 arch/powerpc/include/asm/swiotlb.h   | 2 --
 arch/tile/include/asm/dma-mapping.h  | 2 --
 arch/unicore32/include/asm/dma-mapping.h | 2 --
 arch/x86/include/asm/swiotlb.h   | 2 --
 include/linux/dma-direct.h   | 9 +
 10 files changed, 10 insertions(+), 18 deletions(-)

diff --git a/arch/arm/include/asm/dma-mapping.h 
b/arch/arm/include/asm/dma-mapping.h
index 5fb1b7fbdfbe..e5d9020c9ee1 100644
--- a/arch/arm/include/asm/dma-mapping.h
+++ b/arch/arm/include/asm/dma-mapping.h
@@ -109,8 +109,6 @@ static inline bool is_device_dma_coherent(struct device 
*dev)
return dev->archdata.dma_coherent;
 }
 
-static inline void dma_mark_clean(void *addr, size_t size) { }
-
 /**
  * arm_dma_alloc - allocate consistent memory for DMA
  * @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
diff --git a/arch/arm64/include/asm/dma-mapping.h 
b/arch/arm64/include/asm/dma-mapping.h
index 400fa67d3b5a..b7847eb8a7bb 100644
--- a/arch/arm64/include/asm/dma-mapping.h
+++ b/arch/arm64/include/asm/dma-mapping.h
@@ -50,9 +50,5 @@ static inline bool is_device_dma_coherent(struct device *dev)
return dev->archdata.dma_coherent;
 }
 
-static inline void dma_mark_clean(void *addr, size_t size)
-{
-}
-
 #endif /* __KERNEL__ */
 #endif /* __ASM_DMA_MAPPING_H */
diff --git a/arch/ia64/Kconfig b/arch/ia64/Kconfig
index 49583c5a5d44..4d18fca885ee 100644
--- a/arch/ia64/Kconfig
+++ b/arch/ia64/Kconfig
@@ -33,6 +33,7 @@ config IA64
select HAVE_MEMBLOCK
select HAVE_MEMBLOCK_NODE_MAP
select HAVE_VIRT_CPU_ACCOUNTING
+   select ARCH_HAS_DMA_MARK_CLEAN
select ARCH_HAS_SG_CHAIN
select VIRT_TO_BUS
select ARCH_DISCARD_MEMBLOCK
diff --git a/arch/ia64/include/asm/dma.h b/arch/ia64/include/asm/dma.h
index 186850eec934..23604d6a2cb2 100644
--- a/arch/ia64/include/asm/dma.h
+++ b/arch/ia64/include/asm/dma.h
@@ -20,6 +20,4 @@ extern unsigned long MAX_DMA_ADDRESS;
 
 #define free_dma(x)
 
-void dma_mark_clean(void *addr, size_t size);
-
 #endif /* _ASM_IA64_DMA_H */
diff --git a/arch/mips/include/asm/dma-mapping.h 
b/arch/mips/include/asm/dma-mapping.h
index 676c14cfc580..886e75a383f2 100644
--- a/arch/mips/include/asm/dma-mapping.h
+++ b/arch/mips/include/asm/dma-mapping.h
@@ -17,8 +17,6 @@ static inline const struct dma_map_ops 
*get_arch_dma_ops(struct bus_type *bus)
return mips_dma_map_ops;
 }
 
-static inline void dma_mark_clean(void *addr, size_t size) {}
-
 #define arch_setup_dma_ops arch_setup_dma_ops
 static inline void arch_setup_dma_ops(struct device *dev, u64 dma_base,
  u64 size, const struct iommu_ops *iommu,
diff --git a/arch/powerpc/include/asm/swiotlb.h 
b/arch/powerpc/include/asm/swiotlb.h
index 01d45a5fd00b..9341ee804d19 100644
--- a/arch/powerpc/include/asm/swiotlb.h
+++ b/arch/powerpc/include/asm/swiotlb.h
@@ -15,8 +15,6 @@
 
 extern const struct dma_map_ops swiotlb_dma_ops;
 
-static inline void dma_mark_clean(void *addr, size_t size) {}
-
 extern unsigned int ppc_swiotlb_enable;
 int __init swiotlb_setup_bus_notifier(void);
 
diff --git a/arch/tile/include/asm/dma-mapping.h 
b/arch/tile/include/asm/dma-mapping.h
index 75b8aaa4e70b..d25fce101fc0 100644
--- a/arch/tile/include/asm/dma-mapping.h
+++ b/arch/tile/include/asm/dma-mapping.h
@@ -44,8 +44,6 @@ static inline void set_dma_offset(struct device *dev, 
dma_addr_t off)
dev->archdata.dma_offset = off;
 }
 
-static inline void dma_mark_clean(void *addr, size_t size) {}
-
 #define HAVE_ARCH_DMA_SET_MASK 1
 int dma_set_mask(struct device *dev, u64 mask);
 
diff --git a/arch/unicore32/include/asm/dma-mapping.h 
b/arch/unicore32/include/asm/dma-mapping.h
index 5cb250bf2d8c..f2bfec273aa7 100644
--- a/arch/unicore32/include/asm/dma-mapping.h
+++ b/arch/unicore32/include/asm/dma-mapping.h
@@ -25,7 +25,5 @@ static inline const struct dma_map_ops 
*get_arch_dma_ops(struct bus_type *bus)
return _dma_map_ops;
 }
 
-static inline void dma_mark_clean(void *addr, size_t size) {}
-
 #endif /* __KERNEL__ */
 #endif
diff --git a/arch/x86/include/asm/swiotlb.h b/arch/x86/include/asm/swiotlb.h
index bdf9aed40403..1c6a6cb230ff 100644
--- a/arch/x86/include/asm/swiotlb.h
+++ b/arch/x86/include/asm/swiotlb.h
@@ -28,8 +28,6 @@ static inline void pci_swiotlb_late_init(void)
 }
 #endif
 
-static inline void dma_mark_clean(void *addr, size_t size) {}
-
 extern void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size,
dma_addr_t *dma_handle, gfp_t flags,
  

[PATCH 13/67] dma-mapping: move swiotlb arch helpers to a new header

[Christoph Hellwig]

phys_to_dma, dma_to_phys and dma_capable are helpers published by
architecture code for use of swiotlb and xen-swiotlb only.  Drivers are
not supposed to use these directly, but use the DMA API instead.

Move these to a new asm/dma-direct.h helper, included by a
linux/dma-direct.h wrapper that provides the default linear mapping
unless the architecture wants to override it.

Signed-off-by: Christoph Hellwig 
---
 MAINTAINERS|  1 +
 arch/Kconfig   |  4 +++
 arch/arm/Kconfig   |  1 +
 arch/arm/include/asm/dma-direct.h  | 36 ++
 arch/arm/include/asm/dma-mapping.h | 31 ---
 arch/arm64/include/asm/dma-mapping.h   | 22 -
 arch/arm64/mm/dma-mapping.c|  2 +-
 arch/ia64/include/asm/dma-mapping.h| 18 ---
 arch/mips/Kconfig  |  2 ++
 arch/mips/include/asm/dma-direct.h |  1 +
 arch/mips/include/asm/dma-mapping.h|  8 -
 .../include/asm/mach-cavium-octeon/dma-coherence.h |  8 +
 arch/mips/include/asm/mach-generic/dma-coherence.h | 12 
 .../include/asm/mach-loongson64/dma-coherence.h|  8 +
 arch/powerpc/Kconfig   |  1 +
 arch/powerpc/include/asm/dma-direct.h  | 29 +
 arch/powerpc/include/asm/dma-mapping.h | 25 ---
 arch/tile/include/asm/dma-mapping.h| 18 ---
 arch/unicore32/include/asm/dma-mapping.h   | 18 ---
 arch/x86/Kconfig   |  1 +
 arch/x86/include/asm/dma-direct.h  | 30 ++
 arch/x86/include/asm/dma-mapping.h | 26 
 arch/x86/kernel/amd_gart_64.c  |  1 +
 arch/x86/kernel/pci-dma.c  |  2 +-
 arch/x86/kernel/pci-nommu.c|  2 +-
 arch/x86/kernel/pci-swiotlb.c  |  2 +-
 arch/x86/mm/mem_encrypt.c  |  2 +-
 arch/x86/pci/sta2x11-fixup.c   |  1 +
 arch/xtensa/include/asm/dma-mapping.h  | 10 --
 drivers/crypto/marvell/cesa.c  |  1 +
 drivers/mtd/nand/qcom_nandc.c  |  1 +
 drivers/xen/swiotlb-xen.c  |  2 +-
 include/linux/dma-direct.h | 32 +++
 lib/swiotlb.c  |  2 +-
 34 files changed, 165 insertions(+), 195 deletions(-)
 create mode 100644 arch/arm/include/asm/dma-direct.h
 create mode 100644 arch/mips/include/asm/dma-direct.h
 create mode 100644 arch/powerpc/include/asm/dma-direct.h
 create mode 100644 arch/x86/include/asm/dma-direct.h
 create mode 100644 include/linux/dma-direct.h

diff --git a/MAINTAINERS b/MAINTAINERS
index a6e86e20761e..7521b063b499 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4340,6 +4340,7 @@ F:lib/dma-noop.c
 F: lib/dma-virt.c
 F: drivers/base/dma-mapping.c
 F: drivers/base/dma-coherent.c
+F: include/linux/dma-direct.h
 F: include/linux/dma-mapping.h
 
 DME1737 HARDWARE MONITOR DRIVER
diff --git a/arch/Kconfig b/arch/Kconfig
index 400b9e1b2f27..3edf118ad777 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -938,6 +938,10 @@ config STRICT_MODULE_RWX
  and non-text memory will be made non-executable. This provides
  protection against certain security exploits (e.g. writing to text)
 
+# select if the architecture provides an asm/dma-direct.h header
+config ARCH_HAS_PHYS_TO_DMA
+   bool
+
 config ARCH_HAS_REFCOUNT
bool
help
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 51c8df561077..00d889a37965 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -8,6 +8,7 @@ config ARM
select ARCH_HAS_DEVMEM_IS_ALLOWED
select ARCH_HAS_ELF_RANDOMIZE
select ARCH_HAS_SET_MEMORY
+   select ARCH_HAS_PHYS_TO_DMA
select ARCH_HAS_STRICT_KERNEL_RWX if MMU && !XIP_KERNEL
select ARCH_HAS_STRICT_MODULE_RWX if MMU
select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST
diff --git a/arch/arm/include/asm/dma-direct.h 
b/arch/arm/include/asm/dma-direct.h
new file mode 100644
index ..5b0a8a421894
--- /dev/null
+++ b/arch/arm/include/asm/dma-direct.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef ASM_ARM_DMA_DIRECT_H
+#define ASM_ARM_DMA_DIRECT_H 1
+
+static inline dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr)
+{
+   unsigned int offset = paddr & ~PAGE_MASK;
+   return pfn_to_dma(dev, __phys_to_pfn(paddr)) + offset;
+}
+
+static inline phys_addr_t dma_to_phys(struct device *dev, dma_addr_t dev_addr)
+{
+   unsigned int offset = dev_addr & ~PAGE_MASK;
+   return __pfn_to_phys(dma_to_pfn(dev, dev_addr)) + 

[PATCH 12/67] s390: remove the unused dma_capable helper

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 arch/s390/include/asm/dma-mapping.h | 7 ---
 1 file changed, 7 deletions(-)

diff --git a/arch/s390/include/asm/dma-mapping.h 
b/arch/s390/include/asm/dma-mapping.h
index eaf490f9c5bc..2ec7240c1ada 100644
--- a/arch/s390/include/asm/dma-mapping.h
+++ b/arch/s390/include/asm/dma-mapping.h
@@ -16,11 +16,4 @@ static inline const struct dma_map_ops 
*get_arch_dma_ops(struct bus_type *bus)
return _noop_ops;
 }
 
-static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t 
size)
-{
-   if (!dev->dma_mask)
-   return false;
-   return addr + size - 1 <= *dev->dma_mask;
-}
-
 #endif /* _ASM_S390_DMA_MAPPING_H */
-- 
2.14.2

[PATCH 11/67] riscv: remove the unused dma_capable helper

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 arch/riscv/include/asm/dma-mapping.h | 8 
 1 file changed, 8 deletions(-)

diff --git a/arch/riscv/include/asm/dma-mapping.h 
b/arch/riscv/include/asm/dma-mapping.h
index 3eec1000196d..73849e2cc761 100644
--- a/arch/riscv/include/asm/dma-mapping.h
+++ b/arch/riscv/include/asm/dma-mapping.h
@@ -27,12 +27,4 @@ static inline const struct dma_map_ops 
*get_arch_dma_ops(struct bus_type *bus)
return _noop_ops;
 }
 
-static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t 
size)
-{
-   if (!dev->dma_mask)
-   return false;
-
-   return addr + size - 1 <= *dev->dma_mask;
-}
-
 #endif /* __ASM_RISCV_DMA_MAPPING_H */
-- 
2.14.2

[PATCH 10/67] m32r: remove the unused dma_capable helper

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 arch/m32r/include/asm/dma-mapping.h | 7 ---
 1 file changed, 7 deletions(-)

diff --git a/arch/m32r/include/asm/dma-mapping.h 
b/arch/m32r/include/asm/dma-mapping.h
index 336ffe60814b..8967fb659691 100644
--- a/arch/m32r/include/asm/dma-mapping.h
+++ b/arch/m32r/include/asm/dma-mapping.h
@@ -14,11 +14,4 @@ static inline const struct dma_map_ops 
*get_arch_dma_ops(struct bus_type *bus)
return _noop_ops;
 }
 
-static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t 
size)
-{
-   if (!dev->dma_mask)
-   return false;
-   return addr + size - 1 <= *dev->dma_mask;
-}
-
 #endif /* _ASM_M32R_DMA_MAPPING_H */
-- 
2.14.2

[PATCH 09/67] arc: remove CONFIG_ARC_PLAT_NEEDS_PHYS_TO_DMA

[Christoph Hellwig]

We always use the stub definitions, so remove the unused other code.

Signed-off-by: Christoph Hellwig 
---
 arch/arc/Kconfig   |  3 ---
 arch/arc/include/asm/dma-mapping.h |  7 ---
 arch/arc/mm/dma.c  | 14 +++---
 3 files changed, 7 insertions(+), 17 deletions(-)

diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig
index 9d5fd00d9e91..f3a80cf164cc 100644
--- a/arch/arc/Kconfig
+++ b/arch/arc/Kconfig
@@ -463,9 +463,6 @@ config ARCH_PHYS_ADDR_T_64BIT
 config ARCH_DMA_ADDR_T_64BIT
bool
 
-config ARC_PLAT_NEEDS_PHYS_TO_DMA
-   bool
-
 config ARC_KVADDR_SIZE
int "Kernel Virtual Address Space size (MB)"
range 0 512
diff --git a/arch/arc/include/asm/dma-mapping.h 
b/arch/arc/include/asm/dma-mapping.h
index 94285031c4fb..7a16824bfe98 100644
--- a/arch/arc/include/asm/dma-mapping.h
+++ b/arch/arc/include/asm/dma-mapping.h
@@ -11,13 +11,6 @@
 #ifndef ASM_ARC_DMA_MAPPING_H
 #define ASM_ARC_DMA_MAPPING_H
 
-#ifndef CONFIG_ARC_PLAT_NEEDS_PHYS_TO_DMA
-#define plat_dma_to_phys(dev, dma_handle) ((phys_addr_t)(dma_handle))
-#define plat_phys_to_dma(dev, paddr) ((dma_addr_t)(paddr))
-#else
-#include 
-#endif
-
 extern const struct dma_map_ops arc_dma_ops;
 
 static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
diff --git a/arch/arc/mm/dma.c b/arch/arc/mm/dma.c
index fad18261ef6a..1d405b86250c 100644
--- a/arch/arc/mm/dma.c
+++ b/arch/arc/mm/dma.c
@@ -60,7 +60,7 @@ static void *arc_dma_alloc(struct device *dev, size_t size,
/* This is linear addr (0x8000_ based) */
paddr = page_to_phys(page);
 
-   *dma_handle = plat_phys_to_dma(dev, paddr);
+   *dma_handle = paddr;
 
/* This is kernel Virtual address (0x7000_ based) */
if (need_kvaddr) {
@@ -92,7 +92,7 @@ static void *arc_dma_alloc(struct device *dev, size_t size,
 static void arc_dma_free(struct device *dev, size_t size, void *vaddr,
dma_addr_t dma_handle, unsigned long attrs)
 {
-   phys_addr_t paddr = plat_dma_to_phys(dev, dma_handle);
+   phys_addr_t paddr = dma_handle;
struct page *page = virt_to_page(paddr);
int is_non_coh = 1;
 
@@ -111,7 +111,7 @@ static int arc_dma_mmap(struct device *dev, struct 
vm_area_struct *vma,
 {
unsigned long user_count = vma_pages(vma);
unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT;
-   unsigned long pfn = __phys_to_pfn(plat_dma_to_phys(dev, dma_addr));
+   unsigned long pfn = __phys_to_pfn(dma_addr);
unsigned long off = vma->vm_pgoff;
int ret = -ENXIO;
 
@@ -175,7 +175,7 @@ static dma_addr_t arc_dma_map_page(struct device *dev, 
struct page *page,
if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC))
_dma_cache_sync(paddr, size, dir);
 
-   return plat_phys_to_dma(dev, paddr);
+   return paddr;
 }
 
 /*
@@ -190,7 +190,7 @@ static void arc_dma_unmap_page(struct device *dev, 
dma_addr_t handle,
   size_t size, enum dma_data_direction dir,
   unsigned long attrs)
 {
-   phys_addr_t paddr = plat_dma_to_phys(dev, handle);
+   phys_addr_t paddr = handle;
 
if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC))
_dma_cache_sync(paddr, size, dir);
@@ -224,13 +224,13 @@ static void arc_dma_unmap_sg(struct device *dev, struct 
scatterlist *sg,
 static void arc_dma_sync_single_for_cpu(struct device *dev,
dma_addr_t dma_handle, size_t size, enum dma_data_direction dir)
 {
-   _dma_cache_sync(plat_dma_to_phys(dev, dma_handle), size, 
DMA_FROM_DEVICE);
+   _dma_cache_sync(dma_handle, size, DMA_FROM_DEVICE);
 }
 
 static void arc_dma_sync_single_for_device(struct device *dev,
dma_addr_t dma_handle, size_t size, enum dma_data_direction dir)
 {
-   _dma_cache_sync(plat_dma_to_phys(dev, dma_handle), size, DMA_TO_DEVICE);
+   _dma_cache_sync(dma_handle, size, DMA_TO_DEVICE);
 }
 
 static void arc_dma_sync_sg_for_cpu(struct device *dev,
-- 
2.14.2

[PATCH 08/67] powerpc: remove unused flush_write_buffers definition

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 arch/powerpc/include/asm/dma-mapping.h | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/arch/powerpc/include/asm/dma-mapping.h 
b/arch/powerpc/include/asm/dma-mapping.h
index 5a6cbe11db6f..592c7f418aa0 100644
--- a/arch/powerpc/include/asm/dma-mapping.h
+++ b/arch/powerpc/include/asm/dma-mapping.h
@@ -107,9 +107,6 @@ static inline void set_dma_offset(struct device *dev, 
dma_addr_t off)
dev->archdata.dma_offset = off;
 }
 
-/* this will be removed soon */
-#define flush_write_buffers()
-
 #define HAVE_ARCH_DMA_SET_MASK 1
 extern int dma_set_mask(struct device *dev, u64 dma_mask);
 
-- 
2.14.2

[PATCH 06/67] hexagon: remove unused flush_write_buffers definition

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 arch/hexagon/include/asm/io.h | 2 --
 1 file changed, 2 deletions(-)

diff --git a/arch/hexagon/include/asm/io.h b/arch/hexagon/include/asm/io.h
index 66f5e9a61efc..9e8621d94ee9 100644
--- a/arch/hexagon/include/asm/io.h
+++ b/arch/hexagon/include/asm/io.h
@@ -330,8 +330,6 @@ static inline void outsl(unsigned long port, const void 
*buffer, int count)
}
 }
 
-#define flush_write_buffers() do { } while (0)
-
 #endif /* __KERNEL__ */
 
 #endif
-- 
2.14.2

[PATCH 05/67] dma-mapping: replace PCI_DMA_BUS_IS_PHYS with a flag in struct dma_map_ops

[Christoph Hellwig]

The current PCI_DMA_BUS_IS_PHYS decided if a dma implementation is bound
by the dma mask in the device because it directly maps to a physical
address range (modulo an offset in the device), or if it is virtualized
by an iommu and can map any address (that includes virtual iommus like
swiotlb).  The problem with this scheme is that it is per-architecture and
not per dma_ops instance, and we are growing more and more setups that
have multiple different dma operations in use on a single system, for
which this scheme can't provide a correct answer.  Depending on the
architecture that means we either get a false positive or false negative
at the moment.

This patch instead extents the is_phys flag in struct dma_map_ops that
is currently only used by a few architectures to be used tree wide.

Note that this means that we now need a struct device parent in the
Scsi_Host or netdevice.  Every modern driver has these, but there might
still be a few outdated legacy drivers out there, which now won't make
an intelligent decision.

Signed-off-by: Christoph Hellwig 
---
 arch/alpha/include/asm/pci.h  |  5 -
 arch/alpha/kernel/pci-noop.c  |  1 +
 arch/arc/include/asm/pci.h|  6 --
 arch/arc/mm/dma.c |  1 +
 arch/arm/include/asm/pci.h|  7 ---
 arch/arm/mm/dma-mapping-nommu.c   |  1 +
 arch/arm/mm/dma-mapping.c |  2 ++
 arch/arm64/include/asm/pci.h  |  5 -
 arch/blackfin/kernel/dma-mapping.c|  2 ++
 arch/c6x/kernel/dma.c |  1 +
 arch/cris/arch-v32/drivers/pci/dma.c  |  1 +
 arch/cris/include/asm/pci.h   |  6 --
 arch/frv/mb93090-mb00/pci-dma-nommu.c |  1 +
 arch/frv/mb93090-mb00/pci-dma.c   |  1 +
 arch/h8300/include/asm/pci.h  |  2 --
 arch/h8300/kernel/dma.c   |  1 +
 arch/hexagon/kernel/dma.c |  2 +-
 arch/ia64/hp/common/sba_iommu.c   |  3 ---
 arch/ia64/include/asm/pci.h   | 17 -
 arch/ia64/kernel/setup.c  | 12 
 arch/ia64/sn/kernel/io_common.c   |  5 -
 arch/m68k/include/asm/pci.h   |  6 --
 arch/m68k/kernel/dma.c|  1 +
 arch/metag/kernel/dma.c   |  1 +
 arch/microblaze/include/asm/pci.h |  6 --
 arch/microblaze/kernel/dma.c  |  1 +
 arch/mips/include/asm/pci.h   |  7 ---
 arch/mips/mm/dma-default.c|  1 +
 arch/mn10300/include/asm/pci.h|  6 --
 arch/mn10300/mm/dma-alloc.c   |  1 +
 arch/nios2/mm/dma-mapping.c   |  1 +
 arch/openrisc/kernel/dma.c|  1 +
 arch/parisc/include/asm/pci.h | 23 ---
 arch/parisc/kernel/pci-dma.c  |  2 ++
 arch/parisc/kernel/setup.c|  5 -
 arch/powerpc/include/asm/pci.h| 18 --
 arch/powerpc/kernel/dma.c |  1 +
 arch/riscv/include/asm/pci.h  |  3 ---
 arch/s390/include/asm/pci.h   |  2 --
 arch/s390/pci/pci_dma.c   |  3 ---
 arch/sh/include/asm/pci.h |  6 --
 arch/sh/kernel/dma-nommu.c|  2 +-
 arch/sparc/include/asm/pci_32.h   |  4 
 arch/sparc/include/asm/pci_64.h   |  6 --
 arch/sparc/kernel/ioport.c|  1 +
 arch/tile/include/asm/pci.h   | 14 --
 arch/tile/kernel/pci-dma.c|  2 ++
 arch/x86/include/asm/pci.h|  2 --
 arch/x86/kernel/pci-nommu.c   |  2 +-
 arch/xtensa/include/asm/pci.h |  7 ---
 arch/xtensa/kernel/pci-dma.c  |  1 +
 drivers/ide/ide-lib.c |  5 ++---
 drivers/ide/ide-probe.c   |  2 +-
 drivers/parisc/ccio-dma.c |  2 --
 drivers/parisc/sba_iommu.c|  2 --
 drivers/pci/host/vmd.c|  1 +
 drivers/scsi/scsi_lib.c   | 14 ++
 include/asm-generic/pci.h |  8 
 include/linux/dma-mapping.h   | 23 ++-
 lib/dma-noop.c|  1 +
 net/core/dev.c| 18 --
 tools/virtio/linux/dma-mapping.h  |  2 --
 62 files changed, 70 insertions(+), 226 deletions(-)

diff --git a/arch/alpha/include/asm/pci.h b/arch/alpha/include/asm/pci.h
index b9ec55351924..cf6bc1e64d66 100644
--- a/arch/alpha/include/asm/pci.h
+++ b/arch/alpha/include/asm/pci.h
@@ -56,11 +56,6 @@ struct pci_controller {
 
 /* IOMMU controls.  */
 
-/* The PCI address space does not equal the physical memory address space.
-   The networking and block device layers use this boolean for bounce buffer
-   decisions.  */
-#define PCI_DMA_BUS_IS_PHYS  0
-
 /* TODO: integrate with include/asm-generic/pci.h ? */
 static inline int pci_get_legacy_ide_irq(struct pci_dev *dev, int channel)
 {
diff --git a/arch/alpha/kernel/pci-noop.c b/arch/alpha/kernel/pci-noop.c
index b995987b1557..d3208254b269 100644
--- a/arch/alpha/kernel/pci-noop.c
+++ 

[PATCH 07/67] m32r: remove unused flush_write_buffers definition

[Christoph Hellwig]

Signed-off-by: Christoph Hellwig 
---
 arch/m32r/include/asm/io.h | 2 --
 1 file changed, 2 deletions(-)

diff --git a/arch/m32r/include/asm/io.h b/arch/m32r/include/asm/io.h
index 1b653bb16f9a..a4272d8f0d9c 100644
--- a/arch/m32r/include/asm/io.h
+++ b/arch/m32r/include/asm/io.h
@@ -191,8 +191,6 @@ static inline void _writel(unsigned long l, unsigned long 
addr)
 
 #define mmiowb()
 
-#define flush_write_buffers() do { } while (0)  /* M32R_FIXME */
-
 static inline void
 memset_io(volatile void __iomem *addr, unsigned char val, int count)
 {
-- 
2.14.2

[PATCH 04/67] arm64: don't override dma_max_pfn

[Christoph Hellwig]

The generic version now takes dma_pfn_offset into account, so there is no
more need for an architecture override.

Signed-off-by: Christoph Hellwig 
---
 arch/arm64/include/asm/dma-mapping.h | 9 -
 1 file changed, 9 deletions(-)

diff --git a/arch/arm64/include/asm/dma-mapping.h 
b/arch/arm64/include/asm/dma-mapping.h
index 0df756b24863..eada887a93bf 100644
--- a/arch/arm64/include/asm/dma-mapping.h
+++ b/arch/arm64/include/asm/dma-mapping.h
@@ -76,14 +76,5 @@ static inline void dma_mark_clean(void *addr, size_t size)
 {
 }
 
-/* Override for dma_max_pfn() */
-static inline unsigned long dma_max_pfn(struct device *dev)
-{
-   dma_addr_t dma_max = (dma_addr_t)*dev->dma_mask;
-
-   return (ulong)dma_to_phys(dev, dma_max) >> PAGE_SHIFT;
-}
-#define dma_max_pfn(dev) dma_max_pfn(dev)
-
 #endif /* __KERNEL__ */
 #endif /* __ASM_DMA_MAPPING_H */
-- 
2.14.2

[PATCH 03/67] dma-mapping: take dma_pfn_offset into account in dma_max_pfn

[Christoph Hellwig]

This makes sure the generic version can be used with architectures /
devices that have a DMA offset in the direct mapping.

Signed-off-by: Christoph Hellwig 
---
 include/linux/dma-mapping.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index 81ed9b2d84dc..d84951865be7 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -692,7 +692,7 @@ static inline int dma_set_seg_boundary(struct device *dev, 
unsigned long mask)
 #ifndef dma_max_pfn
 static inline unsigned long dma_max_pfn(struct device *dev)
 {
-   return *dev->dma_mask >> PAGE_SHIFT;
+   return (*dev->dma_mask >> PAGE_SHIFT) + dev->dma_pfn_offset;
 }
 #endif
 
-- 
2.14.2

[PATCH 02/67] alpha: mark jensen as broken

[Christoph Hellwig]

CONFIG_ALPHA_JENSEN has failed to compile since commit aca05038
("alpha/dma: use common noop dma ops"), so mark it as broken.

Signed-off-by: Christoph Hellwig 
---
 arch/alpha/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/alpha/Kconfig b/arch/alpha/Kconfig
index b31b974a03cb..e96adcbcab41 100644
--- a/arch/alpha/Kconfig
+++ b/arch/alpha/Kconfig
@@ -209,6 +209,7 @@ config ALPHA_EIGER
 
 config ALPHA_JENSEN
bool "Jensen"
+   depends on BROKEN
help
  DEC PC 150 AXP (aka Jensen): This is a very old Digital system - one
  of the first-generation Alpha systems. A number of these systems
-- 
2.14.2

[PATCH 01/67] x86: remove X86_PPRO_FENCE

[Christoph Hellwig]

There were only a few Pentium Pro multiprocessors systems where this
errata applied. They are more than 20 years old now, and we've slowly
dropped places where put the workarounds in and discuraged anyone
from enabling the workaround.

Get rid of it for good.

Signed-off-by: Christoph Hellwig 
---
 arch/x86/Kconfig.cpu| 13 -
 arch/x86/entry/vdso/vdso32/vclock_gettime.c |  2 --
 arch/x86/include/asm/barrier.h  | 30 -
 arch/x86/include/asm/io.h   | 15 ---
 arch/x86/kernel/pci-nommu.c | 19 --
 arch/x86/um/asm/barrier.h   |  4 
 6 files changed, 83 deletions(-)

diff --git a/arch/x86/Kconfig.cpu b/arch/x86/Kconfig.cpu
index 65a9a4716e34..f0c5ef578153 100644
--- a/arch/x86/Kconfig.cpu
+++ b/arch/x86/Kconfig.cpu
@@ -315,19 +315,6 @@ config X86_L1_CACHE_SHIFT
default "4" if MELAN || M486 || MGEODEGX1
default "5" if MWINCHIP3D || MWINCHIPC6 || MCRUSOE || MEFFICEON || 
MCYRIXIII || MK6 || MPENTIUMIII || MPENTIUMII || M686 || M586MMX || M586TSC || 
M586 || MVIAC3_2 || MGEODE_LX
 
-config X86_PPRO_FENCE
-   bool "PentiumPro memory ordering errata workaround"
-   depends on M686 || M586MMX || M586TSC || M586 || M486 || MGEODEGX1
-   ---help---
- Old PentiumPro multiprocessor systems had errata that could cause
- memory operations to violate the x86 ordering standard in rare cases.
- Enabling this option will attempt to work around some (but not all)
- occurrences of this problem, at the cost of much heavier spinlock and
- memory barrier operations.
-
- If unsure, say n here. Even distro kernels should think twice before
- enabling this: there are few systems, and an unlikely bug.
-
 config X86_F00F_BUG
def_bool y
depends on M586MMX || M586TSC || M586 || M486
diff --git a/arch/x86/entry/vdso/vdso32/vclock_gettime.c 
b/arch/x86/entry/vdso/vdso32/vclock_gettime.c
index 7780bbfb06ef..9242b28418d5 100644
--- a/arch/x86/entry/vdso/vdso32/vclock_gettime.c
+++ b/arch/x86/entry/vdso/vdso32/vclock_gettime.c
@@ -5,8 +5,6 @@
 #undef CONFIG_OPTIMIZE_INLINING
 #endif
 
-#undef CONFIG_X86_PPRO_FENCE
-
 #ifdef CONFIG_X86_64
 
 /*
diff --git a/arch/x86/include/asm/barrier.h b/arch/x86/include/asm/barrier.h
index 7fb336210e1b..aa0f7449d4a4 100644
--- a/arch/x86/include/asm/barrier.h
+++ b/arch/x86/include/asm/barrier.h
@@ -24,11 +24,7 @@
 #define wmb()  asm volatile("sfence" ::: "memory")
 #endif
 
-#ifdef CONFIG_X86_PPRO_FENCE
-#define dma_rmb()  rmb()
-#else
 #define dma_rmb()  barrier()
-#endif
 #define dma_wmb()  barrier()
 
 #ifdef CONFIG_X86_32
@@ -40,30 +36,6 @@
 #define __smp_wmb()barrier()
 #define __smp_store_mb(var, value) do { (void)xchg(, value); } while (0)
 
-#if defined(CONFIG_X86_PPRO_FENCE)
-
-/*
- * For this option x86 doesn't have a strong TSO memory
- * model and we should fall back to full barriers.
- */
-
-#define __smp_store_release(p, v)  \
-do {   \
-   compiletime_assert_atomic_type(*p); \
-   __smp_mb(); \
-   WRITE_ONCE(*p, v);  \
-} while (0)
-
-#define __smp_load_acquire(p)  \
-({ \
-   typeof(*p) ___p1 = READ_ONCE(*p);   \
-   compiletime_assert_atomic_type(*p); \
-   __smp_mb(); \
-   ___p1;  \
-})
-
-#else /* regular x86 TSO memory ordering */
-
 #define __smp_store_release(p, v)  \
 do {   \
compiletime_assert_atomic_type(*p); \
@@ -79,8 +51,6 @@ do {  
\
___p1;  \
 })
 
-#endif
-
 /* Atomic operations are already serializing on x86 */
 #define __smp_mb__before_atomic()  barrier()
 #define __smp_mb__after_atomic()   barrier()
diff --git a/arch/x86/include/asm/io.h b/arch/x86/include/asm/io.h
index 95e948627fd0..f6e5b9375d8c 100644
--- a/arch/x86/include/asm/io.h
+++ b/arch/x86/include/asm/io.h
@@ -232,21 +232,6 @@ extern void set_iounmap_nonlazy(void);
  */
 #define __ISA_IO_base ((char __iomem *)(PAGE_OFFSET))
 
-/*
- * Cache management
- *
- * This needed for two cases
- * 1. Out of order aware processors
- * 2. Accidentally out of order processors (PPro errata #51)
- */
-
-static inline void flush_write_buffers(void)
-{

consolidate direct dma mapping and swiotlb support

[Christoph Hellwig]

Almost every architecture supports a direct dma mapping implementation,
where no iommu is used and the device dma address is a 1:1 mapping to
the physical address or has a simple linear offset.  Currently the
code for this implementation is most duplicated over the architectures,
and the duplicated again in the swiotlb code, and then duplicated again
for special cases like the x86 memory encryption DMA ops.

This series takes the existing very simple dma-noop dma mapping
implementation, enhances it with all the x86 features and quirks, and
creates a common set of architecture hooks for it and the swiotlb code.

It then switches a large number of architectures to this generic
direct map implement and the new generic swiotlb dma_map ops.

Note that for now this only handles architectures that do cache coherent
DMA, but a similar consolidation for non-coherent architectures is in the
work for later merge windows.

[PATCH 17/17] memremap: merge find_dev_pagemap into get_dev_pagemap

[Christoph Hellwig]

There is only one caller of the trivial function find_dev_pagemap left,
so just merge it into the caller.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Dan Williams 
---
 kernel/memremap.c | 10 +-
 1 file changed, 1 insertion(+), 9 deletions(-)

diff --git a/kernel/memremap.c b/kernel/memremap.c
index a9a948cd3d7f..ada31b0d76d4 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -306,14 +306,6 @@ static void devm_memremap_pages_release(void *data)
  "%s: failed to free all reserved pages\n", __func__);
 }
 
-/* assumes rcu_read_lock() held at entry */
-static struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
-{
-   WARN_ON_ONCE(!rcu_read_lock_held());
-
-   return radix_tree_lookup(_radix, PHYS_PFN(phys));
-}
-
 /**
  * devm_memremap_pages - remap and provide memmap backing for the given 
resource
  * @dev: hosting device for @res
@@ -466,7 +458,7 @@ struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
 
/* fall back to slow path lookup */
rcu_read_lock();
-   pgmap = find_dev_pagemap(phys);
+   pgmap = radix_tree_lookup(_radix, PHYS_PFN(phys));
if (pgmap && !percpu_ref_tryget_live(pgmap->ref))
pgmap = NULL;
rcu_read_unlock();
-- 
2.14.2

[PATCH 16/17] memremap: change devm_memremap_pages interface to use struct dev_pagemap

[Christoph Hellwig]

This new interface is similar to how struct device (and many others)
work. The caller initializes a 'struct dev_pagemap' as required
and calls 'devm_memremap_pages'. This allows the pagemap structure to
be embedded in another structure and thus container_of can be used. In
this way application specific members can be stored in a containing
struct.

This will be used by the P2P infrastructure and HMM could probably
be cleaned up to use it as well (instead of having it's own, similar
'hmm_devmem_pages_create' function).

Signed-off-by: Logan Gunthorpe 
Signed-off-by: Christoph Hellwig 
Reviewed-by: Dan Williams 
---
 drivers/dax/pmem.c| 20 ---
 drivers/nvdimm/nd.h   |  9 +++
 drivers/nvdimm/pfn_devs.c | 27 -
 drivers/nvdimm/pmem.c | 37 +++-
 drivers/nvdimm/pmem.h |  1 +
 include/linux/memremap.h  |  6 ++---
 kernel/memremap.c | 51 ---
 tools/testing/nvdimm/test/iomap.c |  7 +++---
 8 files changed, 77 insertions(+), 81 deletions(-)

diff --git a/drivers/dax/pmem.c b/drivers/dax/pmem.c
index 8d8c852ba8f2..31b6ecce4c64 100644
--- a/drivers/dax/pmem.c
+++ b/drivers/dax/pmem.c
@@ -21,6 +21,7 @@
 struct dax_pmem {
struct device *dev;
struct percpu_ref ref;
+   struct dev_pagemap pgmap;
struct completion cmp;
 };
 
@@ -69,20 +70,23 @@ static int dax_pmem_probe(struct device *dev)
struct nd_namespace_common *ndns;
struct nd_dax *nd_dax = to_nd_dax(dev);
struct nd_pfn *nd_pfn = _dax->nd_pfn;
-   struct vmem_altmap __altmap, *altmap = NULL;
 
ndns = nvdimm_namespace_common_probe(dev);
if (IS_ERR(ndns))
return PTR_ERR(ndns);
nsio = to_nd_namespace_io(>dev);
 
+   dax_pmem = devm_kzalloc(dev, sizeof(*dax_pmem), GFP_KERNEL);
+   if (!dax_pmem)
+   return -ENOMEM;
+
/* parse the 'pfn' info block via ->rw_bytes */
rc = devm_nsio_enable(dev, nsio);
if (rc)
return rc;
-   altmap = nvdimm_setup_pfn(nd_pfn, , &__altmap);
-   if (IS_ERR(altmap))
-   return PTR_ERR(altmap);
+   rc = nvdimm_setup_pfn(nd_pfn, _pmem->pgmap);
+   if (rc)
+   return rc;
devm_nsio_disable(dev, nsio);
 
pfn_sb = nd_pfn->pfn_sb;
@@ -94,10 +98,6 @@ static int dax_pmem_probe(struct device *dev)
return -EBUSY;
}
 
-   dax_pmem = devm_kzalloc(dev, sizeof(*dax_pmem), GFP_KERNEL);
-   if (!dax_pmem)
-   return -ENOMEM;
-
dax_pmem->dev = dev;
init_completion(_pmem->cmp);
rc = percpu_ref_init(_pmem->ref, dax_pmem_percpu_release, 0,
@@ -110,7 +110,8 @@ static int dax_pmem_probe(struct device *dev)
if (rc)
return rc;
 
-   addr = devm_memremap_pages(dev, , _pmem->ref, altmap);
+   dax_pmem->pgmap.ref = _pmem->ref;
+   addr = devm_memremap_pages(dev, _pmem->pgmap);
if (IS_ERR(addr))
return PTR_ERR(addr);
 
@@ -120,6 +121,7 @@ static int dax_pmem_probe(struct device *dev)
return rc;
 
/* adjust the dax_region resource to the start of data */
+   memcpy(, _pmem->pgmap.res, sizeof(res));
res.start += le64_to_cpu(pfn_sb->dataoff);
 
rc = sscanf(dev_name(>dev), "namespace%d.%d", _id, );
diff --git a/drivers/nvdimm/nd.h b/drivers/nvdimm/nd.h
index e958f3724c41..8d6375ee0fda 100644
--- a/drivers/nvdimm/nd.h
+++ b/drivers/nvdimm/nd.h
@@ -368,15 +368,14 @@ unsigned int pmem_sector_size(struct nd_namespace_common 
*ndns);
 void nvdimm_badblocks_populate(struct nd_region *nd_region,
struct badblocks *bb, const struct resource *res);
 #if IS_ENABLED(CONFIG_ND_CLAIM)
-struct vmem_altmap *nvdimm_setup_pfn(struct nd_pfn *nd_pfn,
-   struct resource *res, struct vmem_altmap *altmap);
+int nvdimm_setup_pfn(struct nd_pfn *nd_pfn, struct dev_pagemap *pgmap);
 int devm_nsio_enable(struct device *dev, struct nd_namespace_io *nsio);
 void devm_nsio_disable(struct device *dev, struct nd_namespace_io *nsio);
 #else
-static inline struct vmem_altmap *nvdimm_setup_pfn(struct nd_pfn *nd_pfn,
-   struct resource *res, struct vmem_altmap *altmap)
+static inline int nvdimm_setup_pfn(struct nd_pfn *nd_pfn,
+  struct dev_pagemap *pgmap)
 {
-   return ERR_PTR(-ENXIO);
+   return -ENXIO;
 }
 static inline int devm_nsio_enable(struct device *dev,
struct nd_namespace_io *nsio)
diff --git a/drivers/nvdimm/pfn_devs.c b/drivers/nvdimm/pfn_devs.c
index 2adada1a5855..f5c4e8c6e29d 100644
--- a/drivers/nvdimm/pfn_devs.c
+++ b/drivers/nvdimm/pfn_devs.c
@@ -542,9 +542,10 @@ static unsigned long init_altmap_reserve(resource_size_t 
base)
return reserve;
 }
 
-static struct vmem_altmap 

[PATCH 15/17] memremap: drop private struct page_map

[Christoph Hellwig]

From: Logan Gunthorpe 

'struct page_map' is a private structure of 'struct dev_pagemap' but the
latter replicates all the same fields as the former so there isn't much
value in it. Thus drop it in favour of a completely public struct.

This is a clean up in preperation for a more generally useful
'devm_memeremap_pages' interface.

Signed-off-by: Logan Gunthorpe 
Signed-off-by: Christoph Hellwig 
Reviewed-by: Dan Williams 
---
 include/linux/memremap.h |  5 ++--
 kernel/memremap.c| 68 ++--
 mm/hmm.c |  2 +-
 3 files changed, 30 insertions(+), 45 deletions(-)

diff --git a/include/linux/memremap.h b/include/linux/memremap.h
index 3fddcfe57bb0..1cb5f39d25c1 100644
--- a/include/linux/memremap.h
+++ b/include/linux/memremap.h
@@ -113,8 +113,9 @@ typedef void (*dev_page_free_t)(struct page *page, void 
*data);
 struct dev_pagemap {
dev_page_fault_t page_fault;
dev_page_free_t page_free;
-   struct vmem_altmap *altmap;
-   const struct resource *res;
+   struct vmem_altmap altmap;
+   bool altmap_valid;
+   struct resource res;
struct percpu_ref *ref;
struct device *dev;
void *data;
diff --git a/kernel/memremap.c b/kernel/memremap.c
index 12e78528fea4..9207c44cce20 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -188,13 +188,6 @@ static RADIX_TREE(pgmap_radix, GFP_KERNEL);
 #define SECTION_MASK ~((1UL << PA_SECTION_SHIFT) - 1)
 #define SECTION_SIZE (1UL << PA_SECTION_SHIFT)
 
-struct page_map {
-   struct resource res;
-   struct percpu_ref *ref;
-   struct dev_pagemap pgmap;
-   struct vmem_altmap altmap;
-};
-
 static unsigned long order_at(struct resource *res, unsigned long pgoff)
 {
unsigned long phys_pgoff = PHYS_PFN(res->start) + pgoff;
@@ -260,22 +253,21 @@ static void pgmap_radix_release(struct resource *res)
synchronize_rcu();
 }
 
-static unsigned long pfn_first(struct page_map *page_map)
+static unsigned long pfn_first(struct dev_pagemap *pgmap)
 {
-   struct dev_pagemap *pgmap = _map->pgmap;
-   const struct resource *res = _map->res;
-   struct vmem_altmap *altmap = pgmap->altmap;
+   const struct resource *res = >res;
+   struct vmem_altmap *altmap = >altmap;
unsigned long pfn;
 
pfn = res->start >> PAGE_SHIFT;
-   if (altmap)
+   if (pgmap->altmap_valid)
pfn += vmem_altmap_offset(altmap);
return pfn;
 }
 
-static unsigned long pfn_end(struct page_map *page_map)
+static unsigned long pfn_end(struct dev_pagemap *pgmap)
 {
-   const struct resource *res = _map->res;
+   const struct resource *res = >res;
 
return (res->start + resource_size(res)) >> PAGE_SHIFT;
 }
@@ -285,13 +277,12 @@ static unsigned long pfn_end(struct page_map *page_map)
 
 static void devm_memremap_pages_release(struct device *dev, void *data)
 {
-   struct page_map *page_map = data;
-   struct resource *res = _map->res;
+   struct dev_pagemap *pgmap = data;
+   struct resource *res = >res;
resource_size_t align_start, align_size;
-   struct dev_pagemap *pgmap = _map->pgmap;
unsigned long pfn;
 
-   for_each_device_pfn(pfn, page_map)
+   for_each_device_pfn(pfn, pgmap)
put_page(pfn_to_page(pfn));
 
if (percpu_ref_tryget_live(pgmap->ref)) {
@@ -304,24 +295,22 @@ static void devm_memremap_pages_release(struct device 
*dev, void *data)
align_size = ALIGN(resource_size(res), SECTION_SIZE);
 
mem_hotplug_begin();
-   arch_remove_memory(align_start, align_size, pgmap->altmap);
+   arch_remove_memory(align_start, align_size, pgmap->altmap_valid ?
+   >altmap : NULL);
mem_hotplug_done();
 
untrack_pfn(NULL, PHYS_PFN(align_start), align_size);
pgmap_radix_release(res);
-   dev_WARN_ONCE(dev, pgmap->altmap && pgmap->altmap->alloc,
-   "%s: failed to free all reserved pages\n", __func__);
+   dev_WARN_ONCE(dev, pgmap->altmap.alloc,
+ "%s: failed to free all reserved pages\n", __func__);
 }
 
 /* assumes rcu_read_lock() held at entry */
 static struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
 {
-   struct page_map *page_map;
-
WARN_ON_ONCE(!rcu_read_lock_held());
 
-   page_map = radix_tree_lookup(_radix, PHYS_PFN(phys));
-   return page_map ? _map->pgmap : NULL;
+   return radix_tree_lookup(_radix, PHYS_PFN(phys));
 }
 
 /**
@@ -349,7 +338,6 @@ void *devm_memremap_pages(struct device *dev, struct 
resource *res,
unsigned long pfn, pgoff, order;
pgprot_t pgprot = PAGE_KERNEL;
struct dev_pagemap *pgmap;
-   struct page_map *page_map;
int error, nid, is_ram, i = 0;
 
align_start = res->start & ~(SECTION_SIZE - 1);
@@ -370,22 +358,20 @@ void 

[PATCH 14/17] memremap: simplify duplicate region handling in devm_memremap_pages

[Christoph Hellwig]

__radix_tree_insert already checks for duplicates and returns -EEXIST in
that case, so remove the duplicate (and racy) duplicates check.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Logan Gunthorpe 
---
 kernel/memremap.c | 11 ---
 1 file changed, 11 deletions(-)

diff --git a/kernel/memremap.c b/kernel/memremap.c
index b09517439dec..12e78528fea4 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -396,17 +396,6 @@ void *devm_memremap_pages(struct device *dev, struct 
resource *res,
align_end = align_start + align_size - 1;
 
foreach_order_pgoff(res, order, pgoff) {
-   struct dev_pagemap *dup;
-
-   rcu_read_lock();
-   dup = find_dev_pagemap(res->start + PFN_PHYS(pgoff));
-   rcu_read_unlock();
-   if (dup) {
-   dev_err(dev, "%s: %pr collides with mapping for %s\n",
-   __func__, res, dev_name(dup->dev));
-   error = -EBUSY;
-   break;
-   }
error = __radix_tree_insert(_radix,
PHYS_PFN(res->start) + pgoff, order, page_map);
if (error) {
-- 
2.14.2

[PATCH 13/17] memremap: remove to_vmem_altmap

[Christoph Hellwig]

All callers are gone now.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Dan Williams 
---
 include/linux/memremap.h |  9 -
 kernel/memremap.c| 26 --
 2 files changed, 35 deletions(-)

diff --git a/include/linux/memremap.h b/include/linux/memremap.h
index 26e8aaba27d5..3fddcfe57bb0 100644
--- a/include/linux/memremap.h
+++ b/include/linux/memremap.h
@@ -26,15 +26,6 @@ struct vmem_altmap {
unsigned long alloc;
 };
 
-#ifdef CONFIG_ZONE_DEVICE
-struct vmem_altmap *to_vmem_altmap(unsigned long memmap_start);
-#else
-static inline struct vmem_altmap *to_vmem_altmap(unsigned long memmap_start)
-{
-   return NULL;
-}
-#endif
-
 /*
  * Specialize ZONE_DEVICE memory into multiple types each having differents
  * usage.
diff --git a/kernel/memremap.c b/kernel/memremap.c
index 891c77487a6a..b09517439dec 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -476,32 +476,6 @@ void vmem_altmap_free(struct vmem_altmap *altmap, unsigned 
long nr_pfns)
altmap->alloc -= nr_pfns;
 }
 
-struct vmem_altmap *to_vmem_altmap(unsigned long memmap_start)
-{
-   /*
-* 'memmap_start' is the virtual address for the first "struct
-* page" in this range of the vmemmap array.  In the case of
-* CONFIG_SPARSEMEM_VMEMMAP a page_to_pfn conversion is simple
-* pointer arithmetic, so we can perform this to_vmem_altmap()
-* conversion without concern for the initialization state of
-* the struct page fields.
-*/
-   struct page *page = (struct page *) memmap_start;
-   struct dev_pagemap *pgmap;
-
-   /*
-* Unconditionally retrieve a dev_pagemap associated with the
-* given physical address, this is only for use in the
-* arch_{add|remove}_memory() for setting up and tearing down
-* the memmap.
-*/
-   rcu_read_lock();
-   pgmap = find_dev_pagemap(__pfn_to_phys(page_to_pfn(page)));
-   rcu_read_unlock();
-
-   return pgmap ? pgmap->altmap : NULL;
-}
-
 /**
  * get_dev_pagemap() - take a new live reference on the dev_pagemap for @pfn
  * @pfn: page frame number to lookup page_map
-- 
2.14.2

[PATCH 12/17] mm: optimize dev_pagemap reference counting around get_dev_pagemap

[Christoph Hellwig]

Change the calling convention so that get_dev_pagemap always consumes the
previous reference instead of doing this using an explicit earlier call to
put_dev_pagemap in the callers.

The callers will still need to put the final reference after finishing the
loop over the pages.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Logan Gunthorpe 
Reviewed-by: Dan Williams 
---
 kernel/memremap.c | 17 +
 mm/gup.c  |  7 +--
 2 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/kernel/memremap.c b/kernel/memremap.c
index 3df6cd4ffb40..891c77487a6a 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -507,22 +507,23 @@ struct vmem_altmap *to_vmem_altmap(unsigned long 
memmap_start)
  * @pfn: page frame number to lookup page_map
  * @pgmap: optional known pgmap that already has a reference
  *
- * @pgmap allows the overhead of a lookup to be bypassed when @pfn lands in the
- * same mapping.
+ * If @pgmap is non-NULL and covers @pfn it will be returned as-is.  If @pgmap
+ * is non-NULL but does not cover @pfn the reference to it will be released.
  */
 struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
struct dev_pagemap *pgmap)
 {
-   const struct resource *res = pgmap ? pgmap->res : NULL;
resource_size_t phys = PFN_PHYS(pfn);
 
/*
-* In the cached case we're already holding a live reference so
-* we can simply do a blind increment
+* In the cached case we're already holding a live reference.
 */
-   if (res && phys >= res->start && phys <= res->end) {
-   percpu_ref_get(pgmap->ref);
-   return pgmap;
+   if (pgmap) {
+   const struct resource *res = pgmap ? pgmap->res : NULL;
+
+   if (res && phys >= res->start && phys <= res->end)
+   return pgmap;
+   put_dev_pagemap(pgmap);
}
 
/* fall back to slow path lookup */
diff --git a/mm/gup.c b/mm/gup.c
index e0d82b6706d7..3affe7544b0c 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1410,7 +1410,6 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, 
unsigned long end,
 
VM_BUG_ON_PAGE(compound_head(page) != head, page);
 
-   put_dev_pagemap(pgmap);
SetPageReferenced(page);
pages[*nr] = page;
(*nr)++;
@@ -1420,6 +1419,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, 
unsigned long end,
ret = 1;
 
 pte_unmap:
+   if (pgmap)
+   put_dev_pagemap(pgmap);
pte_unmap(ptem);
return ret;
 }
@@ -1459,10 +1460,12 @@ static int __gup_device_huge(unsigned long pfn, 
unsigned long addr,
SetPageReferenced(page);
pages[*nr] = page;
get_page(page);
-   put_dev_pagemap(pgmap);
(*nr)++;
pfn++;
} while (addr += PAGE_SIZE, addr != end);
+
+   if (pgmap)
+   put_dev_pagemap(pgmap);
return 1;
 }
 
-- 
2.14.2

[PATCH 11/17] mm: move get_dev_pagemap out of line

[Christoph Hellwig]

This is a pretty big function, which should be out of line in general,
and a no-op stub if CONFIG_ZONE_DEVICЕ is not set.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Logan Gunthorpe 
Reviewed-by: Dan Williams 
---
 include/linux/memremap.h | 39 ---
 kernel/memremap.c| 36 ++--
 2 files changed, 38 insertions(+), 37 deletions(-)

diff --git a/include/linux/memremap.h b/include/linux/memremap.h
index d5a6736d9737..26e8aaba27d5 100644
--- a/include/linux/memremap.h
+++ b/include/linux/memremap.h
@@ -133,7 +133,8 @@ struct dev_pagemap {
 #ifdef CONFIG_ZONE_DEVICE
 void *devm_memremap_pages(struct device *dev, struct resource *res,
struct percpu_ref *ref, struct vmem_altmap *altmap);
-struct dev_pagemap *find_dev_pagemap(resource_size_t phys);
+struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
+   struct dev_pagemap *pgmap);
 
 unsigned long vmem_altmap_offset(struct vmem_altmap *altmap);
 void vmem_altmap_free(struct vmem_altmap *altmap, unsigned long nr_pfns);
@@ -153,7 +154,8 @@ static inline void *devm_memremap_pages(struct device *dev,
return ERR_PTR(-ENXIO);
 }
 
-static inline struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
+static inline struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
+   struct dev_pagemap *pgmap)
 {
return NULL;
 }
@@ -183,39 +185,6 @@ static inline bool is_device_public_page(const struct page 
*page)
 }
 #endif /* CONFIG_DEVICE_PRIVATE || CONFIG_DEVICE_PUBLIC */
 
-/**
- * get_dev_pagemap() - take a new live reference on the dev_pagemap for @pfn
- * @pfn: page frame number to lookup page_map
- * @pgmap: optional known pgmap that already has a reference
- *
- * @pgmap allows the overhead of a lookup to be bypassed when @pfn lands in the
- * same mapping.
- */
-static inline struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
-   struct dev_pagemap *pgmap)
-{
-   const struct resource *res = pgmap ? pgmap->res : NULL;
-   resource_size_t phys = PFN_PHYS(pfn);
-
-   /*
-* In the cached case we're already holding a live reference so
-* we can simply do a blind increment
-*/
-   if (res && phys >= res->start && phys <= res->end) {
-   percpu_ref_get(pgmap->ref);
-   return pgmap;
-   }
-
-   /* fall back to slow path lookup */
-   rcu_read_lock();
-   pgmap = find_dev_pagemap(phys);
-   if (pgmap && !percpu_ref_tryget_live(pgmap->ref))
-   pgmap = NULL;
-   rcu_read_unlock();
-
-   return pgmap;
-}
-
 static inline void put_dev_pagemap(struct dev_pagemap *pgmap)
 {
if (pgmap)
diff --git a/kernel/memremap.c b/kernel/memremap.c
index 64b12c806cc5..3df6cd4ffb40 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -314,7 +314,7 @@ static void devm_memremap_pages_release(struct device *dev, 
void *data)
 }
 
 /* assumes rcu_read_lock() held at entry */
-struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
+static struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
 {
struct page_map *page_map;
 
@@ -501,8 +501,40 @@ struct vmem_altmap *to_vmem_altmap(unsigned long 
memmap_start)
 
return pgmap ? pgmap->altmap : NULL;
 }
-#endif /* CONFIG_ZONE_DEVICE */
 
+/**
+ * get_dev_pagemap() - take a new live reference on the dev_pagemap for @pfn
+ * @pfn: page frame number to lookup page_map
+ * @pgmap: optional known pgmap that already has a reference
+ *
+ * @pgmap allows the overhead of a lookup to be bypassed when @pfn lands in the
+ * same mapping.
+ */
+struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
+   struct dev_pagemap *pgmap)
+{
+   const struct resource *res = pgmap ? pgmap->res : NULL;
+   resource_size_t phys = PFN_PHYS(pfn);
+
+   /*
+* In the cached case we're already holding a live reference so
+* we can simply do a blind increment
+*/
+   if (res && phys >= res->start && phys <= res->end) {
+   percpu_ref_get(pgmap->ref);
+   return pgmap;
+   }
+
+   /* fall back to slow path lookup */
+   rcu_read_lock();
+   pgmap = find_dev_pagemap(phys);
+   if (pgmap && !percpu_ref_tryget_live(pgmap->ref))
+   pgmap = NULL;
+   rcu_read_unlock();
+
+   return pgmap;
+}
+#endif /* CONFIG_ZONE_DEVICE */
 
 #if IS_ENABLED(CONFIG_DEVICE_PRIVATE) ||  IS_ENABLED(CONFIG_DEVICE_PUBLIC)
 void put_zone_device_private_or_public_page(struct page *page)
-- 
2.14.2

[PATCH 10/17] mm: merge vmem_altmap_alloc into altmap_alloc_block_buf

[Christoph Hellwig]

There is no clear separation between the two, so merge them.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Logan Gunthorpe 
Reviewed-by: Dan Williams 
---
 mm/sparse-vmemmap.c | 45 -
 1 file changed, 16 insertions(+), 29 deletions(-)

diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index d012c9e2811b..bd0276d5f66b 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -107,33 +107,16 @@ static unsigned long __meminit vmem_altmap_nr_free(struct 
vmem_altmap *altmap)
 }
 
 /**
- * vmem_altmap_alloc - allocate pages from the vmem_altmap reservation
- * @altmap - reserved page pool for the allocation
- * @nr_pfns - size (in pages) of the allocation
+ * altmap_alloc_block_buf - allocate pages from the device page map
+ * @altmap:device page map
+ * @size:  size (in bytes) of the allocation
  *
- * Allocations are aligned to the size of the request
+ * Allocations are aligned to the size of the request.
  */
-static unsigned long __meminit vmem_altmap_alloc(struct vmem_altmap *altmap,
-   unsigned long nr_pfns)
-{
-   unsigned long pfn = vmem_altmap_next_pfn(altmap);
-   unsigned long nr_align;
-
-   nr_align = 1UL << find_first_bit(_pfns, BITS_PER_LONG);
-   nr_align = ALIGN(pfn, nr_align) - pfn;
-
-   if (nr_pfns + nr_align > vmem_altmap_nr_free(altmap))
-   return ULONG_MAX;
-   altmap->alloc += nr_pfns;
-   altmap->align += nr_align;
-   return pfn + nr_align;
-}
-
 void * __meminit altmap_alloc_block_buf(unsigned long size,
struct vmem_altmap *altmap)
 {
-   unsigned long pfn, nr_pfns;
-   void *ptr;
+   unsigned long pfn, nr_pfns, nr_align;
 
if (size & ~PAGE_MASK) {
pr_warn_once("%s: allocations must be multiple of PAGE_SIZE 
(%ld)\n",
@@ -141,16 +124,20 @@ void * __meminit altmap_alloc_block_buf(unsigned long 
size,
return NULL;
}
 
+   pfn = vmem_altmap_next_pfn(altmap);
nr_pfns = size >> PAGE_SHIFT;
-   pfn = vmem_altmap_alloc(altmap, nr_pfns);
-   if (pfn < ULONG_MAX)
-   ptr = __va(__pfn_to_phys(pfn));
-   else
-   ptr = NULL;
+   nr_align = 1UL << find_first_bit(_pfns, BITS_PER_LONG);
+   nr_align = ALIGN(pfn, nr_align) - pfn;
+   if (nr_pfns + nr_align > vmem_altmap_nr_free(altmap))
+   return NULL;
+
+   altmap->alloc += nr_pfns;
+   altmap->align += nr_align;
+   pfn += nr_align;
+
pr_debug("%s: pfn: %#lx alloc: %ld align: %ld nr: %#lx\n",
__func__, pfn, altmap->alloc, altmap->align, nr_pfns);
-
-   return ptr;
+   return __va(__pfn_to_phys(pfn));
 }
 
 void __meminit vmemmap_verify(pte_t *pte, int node,
-- 
2.14.2

[PATCH 09/17] mm: split altmap memory map allocation from normal case

[Christoph Hellwig]

No functional changes, just untangling the call chain and document
why the altmap is passed around the hotplug code.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Logan Gunthorpe 
Reviewed-by: Dan Williams 
---
 arch/powerpc/mm/init_64.c |  5 -
 arch/x86/mm/init_64.c |  5 -
 include/linux/mm.h|  9 ++---
 mm/sparse-vmemmap.c   | 15 +++
 4 files changed, 13 insertions(+), 21 deletions(-)

diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c
index db7d4e092157..7a2251d99ed3 100644
--- a/arch/powerpc/mm/init_64.c
+++ b/arch/powerpc/mm/init_64.c
@@ -200,7 +200,10 @@ int __meminit vmemmap_populate(unsigned long start, 
unsigned long end, int node,
if (vmemmap_populated(start, page_size))
continue;
 
-   p =  __vmemmap_alloc_block_buf(page_size, node, altmap);
+   if (altmap)
+   p = altmap_alloc_block_buf(page_size, altmap);
+   else
+   p = vmemmap_alloc_block_buf(page_size, node);
if (!p)
return -ENOMEM;
 
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 0cab4b5b59ba..1ab42c852069 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1385,7 +1385,10 @@ static int __meminit vmemmap_populate_hugepages(unsigned 
long start,
if (pmd_none(*pmd)) {
void *p;
 
-   p = __vmemmap_alloc_block_buf(PMD_SIZE, node, altmap);
+   if (altmap)
+   p = altmap_alloc_block_buf(PMD_SIZE, altmap);
+   else
+   p = vmemmap_alloc_block_buf(PMD_SIZE, node);
if (p) {
pte_t entry;
 
diff --git a/include/linux/mm.h b/include/linux/mm.h
index fd01135324b6..09637c353de0 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2547,13 +2547,8 @@ pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long 
addr, int node);
 pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node);
 void *vmemmap_alloc_block(unsigned long size, int node);
 struct vmem_altmap;
-void *__vmemmap_alloc_block_buf(unsigned long size, int node,
-   struct vmem_altmap *altmap);
-static inline void *vmemmap_alloc_block_buf(unsigned long size, int node)
-{
-   return __vmemmap_alloc_block_buf(size, node, NULL);
-}
-
+void *vmemmap_alloc_block_buf(unsigned long size, int node);
+void *altmap_alloc_block_buf(unsigned long size, struct vmem_altmap *altmap);
 void vmemmap_verify(pte_t *, int, unsigned long, unsigned long);
 int vmemmap_populate_basepages(unsigned long start, unsigned long end,
   int node);
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index 376dcf05a39c..d012c9e2811b 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -74,7 +74,7 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int 
node)
 }
 
 /* need to make sure size is all the same during early stage */
-static void * __meminit alloc_block_buf(unsigned long size, int node)
+void * __meminit vmemmap_alloc_block_buf(unsigned long size, int node)
 {
void *ptr;
 
@@ -129,7 +129,7 @@ static unsigned long __meminit vmem_altmap_alloc(struct 
vmem_altmap *altmap,
return pfn + nr_align;
 }
 
-static void * __meminit altmap_alloc_block_buf(unsigned long size,
+void * __meminit altmap_alloc_block_buf(unsigned long size,
struct vmem_altmap *altmap)
 {
unsigned long pfn, nr_pfns;
@@ -153,15 +153,6 @@ static void * __meminit altmap_alloc_block_buf(unsigned 
long size,
return ptr;
 }
 
-/* need to make sure size is all the same during early stage */
-void * __meminit __vmemmap_alloc_block_buf(unsigned long size, int node,
-   struct vmem_altmap *altmap)
-{
-   if (altmap)
-   return altmap_alloc_block_buf(size, altmap);
-   return alloc_block_buf(size, node);
-}
-
 void __meminit vmemmap_verify(pte_t *pte, int node,
unsigned long start, unsigned long end)
 {
@@ -178,7 +169,7 @@ pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned 
long addr, int node)
pte_t *pte = pte_offset_kernel(pmd, addr);
if (pte_none(*pte)) {
pte_t entry;
-   void *p = alloc_block_buf(PAGE_SIZE, node);
+   void *p = vmemmap_alloc_block_buf(PAGE_SIZE, node);
if (!p)
return NULL;
entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL);
-- 
2.14.2

[PATCH 08/17] mm: pass the vmem_altmap to memmap_init_zone

[Christoph Hellwig]

Pass the vmem_altmap two levels down instead of needing a lookup.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Dan Williams 
---
 arch/ia64/mm/init.c| 9 +
 include/linux/memory_hotplug.h | 2 +-
 include/linux/mm.h | 4 ++--
 kernel/memremap.c  | 2 +-
 mm/hmm.c   | 2 +-
 mm/memory_hotplug.c| 9 +
 mm/page_alloc.c| 6 +++---
 7 files changed, 18 insertions(+), 16 deletions(-)

diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
index 6a8ce9e1536e..18278b448530 100644
--- a/arch/ia64/mm/init.c
+++ b/arch/ia64/mm/init.c
@@ -501,7 +501,7 @@ virtual_memmap_init(u64 start, u64 end, void *arg)
if (map_start < map_end)
memmap_init_zone((unsigned long)(map_end - map_start),
 args->nid, args->zone, page_to_pfn(map_start),
-MEMMAP_EARLY);
+MEMMAP_EARLY, NULL);
return 0;
 }
 
@@ -509,9 +509,10 @@ void __meminit
 memmap_init (unsigned long size, int nid, unsigned long zone,
 unsigned long start_pfn)
 {
-   if (!vmem_map)
-   memmap_init_zone(size, nid, zone, start_pfn, MEMMAP_EARLY);
-   else {
+   if (!vmem_map) {
+   memmap_init_zone(size, nid, zone, start_pfn, MEMMAP_EARLY,
+   NULL);
+   } else {
struct page *start;
struct memmap_init_callback_data args;
 
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 20dd98ad44a0..aba5f86eb038 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -324,7 +324,7 @@ extern int add_memory_resource(int nid, struct resource 
*resource, bool online);
 extern int arch_add_memory(int nid, u64 start, u64 size,
struct vmem_altmap *altmap, bool want_memblock);
 extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,
-   unsigned long nr_pages);
+   unsigned long nr_pages, struct vmem_altmap *altmap);
 extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
 extern bool is_memblock_offlined(struct memory_block *mem);
 extern void remove_memory(int nid, u64 start, u64 size);
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 9d4cd4c1dc6d..fd01135324b6 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2069,8 +2069,8 @@ static inline void zero_resv_unavail(void) {}
 #endif
 
 extern void set_dma_reserve(unsigned long new_dma_reserve);
-extern void memmap_init_zone(unsigned long, int, unsigned long,
-   unsigned long, enum memmap_context);
+extern void memmap_init_zone(unsigned long, int, unsigned long, unsigned long,
+   enum memmap_context, struct vmem_altmap *);
 extern void setup_per_zone_wmarks(void);
 extern int __meminit init_per_zone_wmark_min(void);
 extern void mem_init(void);
diff --git a/kernel/memremap.c b/kernel/memremap.c
index 380fca1c4a02..64b12c806cc5 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -432,7 +432,7 @@ void *devm_memremap_pages(struct device *dev, struct 
resource *res,
if (!error)
move_pfn_range_to_zone(_DATA(nid)->node_zones[ZONE_DEVICE],
align_start >> PAGE_SHIFT,
-   align_size >> PAGE_SHIFT);
+   align_size >> PAGE_SHIFT, altmap);
mem_hotplug_done();
if (error)
goto err_add_memory;
diff --git a/mm/hmm.c b/mm/hmm.c
index 5d17ba89062f..2f2e13c61040 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -942,7 +942,7 @@ static int hmm_devmem_pages_create(struct hmm_devmem 
*devmem)
}
move_pfn_range_to_zone(_DATA(nid)->node_zones[ZONE_DEVICE],
align_start >> PAGE_SHIFT,
-   align_size >> PAGE_SHIFT);
+   align_size >> PAGE_SHIFT, NULL);
mem_hotplug_done();
 
for (pfn = devmem->pfn_first; pfn < devmem->pfn_last; pfn++) {
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index a8dde9734120..12df8a5fadcc 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -798,8 +798,8 @@ static void __meminit resize_pgdat_range(struct pglist_data 
*pgdat, unsigned lon
pgdat->node_spanned_pages = max(start_pfn + nr_pages, old_end_pfn) - 
pgdat->node_start_pfn;
 }
 
-void __ref move_pfn_range_to_zone(struct zone *zone,
-   unsigned long start_pfn, unsigned long nr_pages)
+void __ref move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,
+   unsigned long nr_pages, struct vmem_altmap *altmap)
 {
struct pglist_data *pgdat = zone->zone_pgdat;
int nid = pgdat->node_id;
@@ -824,7 +824,8 @@ void __ref move_pfn_range_to_zone(struct zone *zone,
 * expects 

[PATCH 07/17] mm: pass the vmem_altmap to vmemmap_free

[Christoph Hellwig]

We can just pass this on instead of having to do a radix tree lookup
without proper locking a few levels into the callchain.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Dan Williams 
---
 arch/arm64/mm/mmu.c|  3 +-
 arch/ia64/mm/discontig.c   |  3 +-
 arch/powerpc/mm/init_64.c  |  5 ++--
 arch/s390/mm/vmem.c|  3 +-
 arch/sparc/mm/init_64.c|  3 +-
 arch/x86/mm/init_64.c  | 67 --
 include/linux/memory_hotplug.h |  2 +-
 include/linux/mm.h |  3 +-
 mm/memory_hotplug.c|  7 +++--
 mm/sparse.c| 23 ---
 10 files changed, 68 insertions(+), 51 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index ec8952ff13be..0b1f13e0b4b3 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -696,7 +696,8 @@ int __meminit vmemmap_populate(unsigned long start, 
unsigned long end, int node,
return 0;
 }
 #endif /* CONFIG_ARM64_64K_PAGES */
-void vmemmap_free(unsigned long start, unsigned long end)
+void vmemmap_free(unsigned long start, unsigned long end,
+   struct vmem_altmap *altmap)
 {
 }
 #endif /* CONFIG_SPARSEMEM_VMEMMAP */
diff --git a/arch/ia64/mm/discontig.c b/arch/ia64/mm/discontig.c
index 1555aecaaf85..5ea0d8d0968b 100644
--- a/arch/ia64/mm/discontig.c
+++ b/arch/ia64/mm/discontig.c
@@ -760,7 +760,8 @@ int __meminit vmemmap_populate(unsigned long start, 
unsigned long end, int node,
return vmemmap_populate_basepages(start, end, node);
 }
 
-void vmemmap_free(unsigned long start, unsigned long end)
+void vmemmap_free(unsigned long start, unsigned long end,
+   struct vmem_altmap *altmap)
 {
 }
 #endif
diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c
index 779b74a96b8f..db7d4e092157 100644
--- a/arch/powerpc/mm/init_64.c
+++ b/arch/powerpc/mm/init_64.c
@@ -254,7 +254,8 @@ static unsigned long vmemmap_list_free(unsigned long start)
return vmem_back->phys;
 }
 
-void __ref vmemmap_free(unsigned long start, unsigned long end)
+void __ref vmemmap_free(unsigned long start, unsigned long end,
+   struct vmem_altmap *altmap)
 {
unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift;
unsigned long page_order = get_order(page_size);
@@ -265,7 +266,6 @@ void __ref vmemmap_free(unsigned long start, unsigned long 
end)
 
for (; start < end; start += page_size) {
unsigned long nr_pages, addr;
-   struct vmem_altmap *altmap;
struct page *section_base;
struct page *page;
 
@@ -285,7 +285,6 @@ void __ref vmemmap_free(unsigned long start, unsigned long 
end)
section_base = pfn_to_page(vmemmap_section_start(start));
nr_pages = 1 << page_order;
 
-   altmap = to_vmem_altmap((unsigned long) section_base);
if (altmap) {
vmem_altmap_free(altmap, nr_pages);
} else if (PageReserved(page)) {
diff --git a/arch/s390/mm/vmem.c b/arch/s390/mm/vmem.c
index c44ef0e7c466..db55561c5981 100644
--- a/arch/s390/mm/vmem.c
+++ b/arch/s390/mm/vmem.c
@@ -297,7 +297,8 @@ int __meminit vmemmap_populate(unsigned long start, 
unsigned long end, int node,
return ret;
 }
 
-void vmemmap_free(unsigned long start, unsigned long end)
+void vmemmap_free(unsigned long start, unsigned long end,
+   struct vmem_altmap *altmap)
 {
 }
 
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index 42d27a1a042a..995f9490334d 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2671,7 +2671,8 @@ int __meminit vmemmap_populate(unsigned long vstart, 
unsigned long vend,
return 0;
 }
 
-void vmemmap_free(unsigned long start, unsigned long end)
+void vmemmap_free(unsigned long start, unsigned long end,
+   struct vmem_altmap *altmap)
 {
 }
 #endif /* CONFIG_SPARSEMEM_VMEMMAP */
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 3c046618cc7e..0cab4b5b59ba 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -800,11 +800,11 @@ int arch_add_memory(int nid, u64 start, u64 size, struct 
vmem_altmap *altmap,
 
 #define PAGE_INUSE 0xFD
 
-static void __meminit free_pagetable(struct page *page, int order)
+static void __meminit free_pagetable(struct page *page, int order,
+   struct vmem_altmap *altmap)
 {
unsigned long magic;
unsigned int nr_pages = 1 << order;
-   struct vmem_altmap *altmap = to_vmem_altmap((unsigned long) page);
 
if (altmap) {
vmem_altmap_free(altmap, nr_pages);
@@ -826,7 +826,8 @@ static void __meminit free_pagetable(struct page *page, int 
order)
free_pages((unsigned long)page_address(page), order);
 }
 
-static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd)
+static void __meminit free_pte_table(pte_t 

[PATCH 06/17] mm: pass the vmem_altmap to arch_remove_memory and __remove_pages

[Christoph Hellwig]

We can just pass this on instead of having to do a radix tree lookup
without proper locking 2 levels into the callchain.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Dan Williams 
---
 arch/ia64/mm/init.c| 4 ++--
 arch/powerpc/mm/mem.c  | 6 ++
 arch/s390/mm/init.c| 2 +-
 arch/sh/mm/init.c  | 4 ++--
 arch/x86/mm/init_32.c  | 4 ++--
 arch/x86/mm/init_64.c  | 6 ++
 include/linux/memory_hotplug.h | 5 +++--
 kernel/memremap.c  | 2 +-
 mm/hmm.c   | 4 ++--
 mm/memory_hotplug.c| 8 ++--
 10 files changed, 19 insertions(+), 26 deletions(-)

diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
index 2e2e4f532204..6a8ce9e1536e 100644
--- a/arch/ia64/mm/init.c
+++ b/arch/ia64/mm/init.c
@@ -663,7 +663,7 @@ int arch_add_memory(int nid, u64 start, u64 size, struct 
vmem_altmap *altmap,
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
-int arch_remove_memory(u64 start, u64 size)
+int arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
 {
unsigned long start_pfn = start >> PAGE_SHIFT;
unsigned long nr_pages = size >> PAGE_SHIFT;
@@ -671,7 +671,7 @@ int arch_remove_memory(u64 start, u64 size)
int ret;
 
zone = page_zone(pfn_to_page(start_pfn));
-   ret = __remove_pages(zone, start_pfn, nr_pages);
+   ret = __remove_pages(zone, start_pfn, nr_pages, altmap);
if (ret)
pr_warn("%s: Problem encountered in __remove_pages() as"
" ret=%d\n", __func__,  ret);
diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
index e670cfc2766e..22aa528b78a2 100644
--- a/arch/powerpc/mm/mem.c
+++ b/arch/powerpc/mm/mem.c
@@ -149,11 +149,10 @@ int arch_add_memory(int nid, u64 start, u64 size, struct 
vmem_altmap *altmap,
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
-int arch_remove_memory(u64 start, u64 size)
+int arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
 {
unsigned long start_pfn = start >> PAGE_SHIFT;
unsigned long nr_pages = size >> PAGE_SHIFT;
-   struct vmem_altmap *altmap;
struct page *page;
int ret;
 
@@ -162,11 +161,10 @@ int arch_remove_memory(u64 start, u64 size)
 * when querying the zone.
 */
page = pfn_to_page(start_pfn);
-   altmap = to_vmem_altmap((unsigned long) page);
if (altmap)
page += vmem_altmap_offset(altmap);
 
-   ret = __remove_pages(page_zone(page), start_pfn, nr_pages);
+   ret = __remove_pages(page_zone(page), start_pfn, nr_pages, altmap);
if (ret)
return ret;
 
diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
index e12c5af50cd7..3fa3e5323612 100644
--- a/arch/s390/mm/init.c
+++ b/arch/s390/mm/init.c
@@ -240,7 +240,7 @@ int arch_add_memory(int nid, u64 start, u64 size, struct 
vmem_altmap *altmap,
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
-int arch_remove_memory(u64 start, u64 size)
+int arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
 {
/*
 * There is no hardware or firmware interface which could trigger a
diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
index 552afbf55bad..ce0bbaa7e404 100644
--- a/arch/sh/mm/init.c
+++ b/arch/sh/mm/init.c
@@ -510,7 +510,7 @@ EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
 #endif
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
-int arch_remove_memory(u64 start, u64 size)
+int arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
 {
unsigned long start_pfn = PFN_DOWN(start);
unsigned long nr_pages = size >> PAGE_SHIFT;
@@ -518,7 +518,7 @@ int arch_remove_memory(u64 start, u64 size)
int ret;
 
zone = page_zone(pfn_to_page(start_pfn));
-   ret = __remove_pages(zone, start_pfn, nr_pages);
+   ret = __remove_pages(zone, start_pfn, nr_pages, altmap);
if (unlikely(ret))
pr_warn("%s: Failed, __remove_pages() == %d\n", __func__,
ret);
diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
index 8a3091511a71..79cb066f40c0 100644
--- a/arch/x86/mm/init_32.c
+++ b/arch/x86/mm/init_32.c
@@ -839,14 +839,14 @@ int arch_add_memory(int nid, u64 start, u64 size, struct 
vmem_altmap *altmap,
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
-int arch_remove_memory(u64 start, u64 size)
+int arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
 {
unsigned long start_pfn = start >> PAGE_SHIFT;
unsigned long nr_pages = size >> PAGE_SHIFT;
struct zone *zone;
 
zone = page_zone(pfn_to_page(start_pfn));
-   return __remove_pages(zone, start_pfn, nr_pages);
+   return __remove_pages(zone, start_pfn, nr_pages, altmap);
 }
 #endif
 #endif
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 594902ef56ef..3c046618cc7e 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1132,21 +1132,19 @@ kernel_physical_mapping_remove(unsigned 

[PATCH 05/17] mm: pass the vmem_altmap to vmemmap_populate

[Christoph Hellwig]

We can just pass this on instead of having to do a radix tree lookup
without proper locking a few levels into the callchain.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Dan Williams 
---
 arch/arm64/mm/mmu.c|  6 --
 arch/ia64/mm/discontig.c   |  3 ++-
 arch/powerpc/mm/init_64.c  |  7 ++-
 arch/s390/mm/vmem.c|  3 ++-
 arch/sparc/mm/init_64.c|  2 +-
 arch/x86/mm/init_64.c  |  4 ++--
 include/linux/memory_hotplug.h |  3 ++-
 include/linux/mm.h |  6 --
 mm/memory_hotplug.c|  7 ---
 mm/sparse-vmemmap.c|  7 ---
 mm/sparse.c| 20 
 11 files changed, 39 insertions(+), 29 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 267d2b79d52d..ec8952ff13be 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -654,12 +654,14 @@ int kern_addr_valid(unsigned long addr)
 }
 #ifdef CONFIG_SPARSEMEM_VMEMMAP
 #if !ARM64_SWAPPER_USES_SECTION_MAPS
-int __meminit vmemmap_populate(unsigned long start, unsigned long end, int 
node)
+int __meminit vmemmap_populate(unsigned long start, unsigned long end, int 
node,
+   struct vmem_altmap *altmap)
 {
return vmemmap_populate_basepages(start, end, node);
 }
 #else  /* !ARM64_SWAPPER_USES_SECTION_MAPS */
-int __meminit vmemmap_populate(unsigned long start, unsigned long end, int 
node)
+int __meminit vmemmap_populate(unsigned long start, unsigned long end, int 
node,
+   struct vmem_altmap *altmap)
 {
unsigned long addr = start;
unsigned long next;
diff --git a/arch/ia64/mm/discontig.c b/arch/ia64/mm/discontig.c
index 9b2d994cddf6..1555aecaaf85 100644
--- a/arch/ia64/mm/discontig.c
+++ b/arch/ia64/mm/discontig.c
@@ -754,7 +754,8 @@ void arch_refresh_nodedata(int update_node, pg_data_t 
*update_pgdat)
 #endif
 
 #ifdef CONFIG_SPARSEMEM_VMEMMAP
-int __meminit vmemmap_populate(unsigned long start, unsigned long end, int 
node)
+int __meminit vmemmap_populate(unsigned long start, unsigned long end, int 
node,
+   struct vmem_altmap *altmap)
 {
return vmemmap_populate_basepages(start, end, node);
 }
diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c
index a07722531b32..779b74a96b8f 100644
--- a/arch/powerpc/mm/init_64.c
+++ b/arch/powerpc/mm/init_64.c
@@ -183,7 +183,8 @@ static __meminit void vmemmap_list_populate(unsigned long 
phys,
vmemmap_list = vmem_back;
 }
 
-int __meminit vmemmap_populate(unsigned long start, unsigned long end, int 
node)
+int __meminit vmemmap_populate(unsigned long start, unsigned long end, int 
node,
+   struct vmem_altmap *altmap)
 {
unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift;
 
@@ -193,16 +194,12 @@ int __meminit vmemmap_populate(unsigned long start, 
unsigned long end, int node)
pr_debug("vmemmap_populate %lx..%lx, node %d\n", start, end, node);
 
for (; start < end; start += page_size) {
-   struct vmem_altmap *altmap;
void *p;
int rc;
 
if (vmemmap_populated(start, page_size))
continue;
 
-   /* altmap lookups only work at section boundaries */
-   altmap = to_vmem_altmap(SECTION_ALIGN_DOWN(start));
-
p =  __vmemmap_alloc_block_buf(page_size, node, altmap);
if (!p)
return -ENOMEM;
diff --git a/arch/s390/mm/vmem.c b/arch/s390/mm/vmem.c
index 3316d463fc29..c44ef0e7c466 100644
--- a/arch/s390/mm/vmem.c
+++ b/arch/s390/mm/vmem.c
@@ -211,7 +211,8 @@ static void vmem_remove_range(unsigned long start, unsigned 
long size)
 /*
  * Add a backed mem_map array to the virtual mem_map array.
  */
-int __meminit vmemmap_populate(unsigned long start, unsigned long end, int 
node)
+int __meminit vmemmap_populate(unsigned long start, unsigned long end, int 
node,
+   struct vmem_altmap *altmap)
 {
unsigned long pgt_prot, sgt_prot;
unsigned long address = start;
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index 55ba62957e64..42d27a1a042a 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2628,7 +2628,7 @@ EXPORT_SYMBOL(_PAGE_CACHE);
 
 #ifdef CONFIG_SPARSEMEM_VMEMMAP
 int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend,
-  int node)
+  int node, struct vmem_altmap *altmap)
 {
unsigned long pte_base;
 
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index e80bb4189254..594902ef56ef 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1411,9 +1411,9 @@ static int __meminit vmemmap_populate_hugepages(unsigned 
long start,
return 0;
 }
 
-int __meminit vmemmap_populate(unsigned long start, unsigned long end, int 
node)
+int __meminit vmemmap_populate(unsigned long start, 

revamp vmem_altmap / dev_pagemap handling V3

[Christoph Hellwig]

Hi all,

this series started with two patches from Logan that now are in the
middle of the series to kill the memremap-internal pgmap structure
and to redo the dev_memreamp_pages interface to be better suitable
for future PCI P2P uses.  I reviewed them and noticed that there
isn't really any good reason to keep struct vmem_altmap either,
and that a lot of these alternative device page map access should
be better abstracted out instead of being sprinkled all over the
mm code.  But when we got the RCU warnings in V1 I went for yet
another approach, and now struct vmem_altmap is kept for now,
but passed explicitly through the memory hotplug code instead of
having to do unprotected lookups through the radix tree.  The
end result is that only the get_user_pages path ever looks up
struct dev_pagemap, and struct vmem_altmap is now always embedded
into struct dev_pagemap, and explicitly passed where needed.

Please review carefully, this has only been tested with my legacy
e820 NVDIMM system.

Chances since V2:
 - properly pass altmap from dev_devm_memremap_pages through add_pages
   (Dan Williams)
 - small changelog updates
 - a comment type fix
 - dropped the patch to just rely on the radix_tree_insert return value
 - initialize pgmap->type

[PATCH 03/17] mm: don't export __add_pages

[Christoph Hellwig]

This function isn't used by any modules, and is only to be called
from core MM code.  This includes the calls for the add_pages wrapper
that might be inlined.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Dan Williams 
---
 mm/memory_hotplug.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index c52aa05b106c..5c6f96e6b334 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -334,7 +334,6 @@ int __ref __add_pages(int nid, unsigned long phys_start_pfn,
 out:
return err;
 }
-EXPORT_SYMBOL_GPL(__add_pages);
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
 /* find the smallest valid pfn in the range [start_pfn, end_pfn) */
-- 
2.14.2

[PATCH 04/17] mm: pass the vmem_altmap to arch_add_memory and __add_pages

[Christoph Hellwig]

We can just pass this on instead of having to do a radix tree lookup
without proper locking 2 levels into the callchain.

Signed-off-by: Christoph Hellwig 
---
 arch/ia64/mm/init.c|  5 +++--
 arch/powerpc/mm/mem.c  |  5 +++--
 arch/s390/mm/init.c|  5 +++--
 arch/sh/mm/init.c  |  5 +++--
 arch/x86/mm/init_32.c  |  5 +++--
 arch/x86/mm/init_64.c  | 11 ++-
 include/linux/memory_hotplug.h | 17 ++---
 kernel/memremap.c  |  3 ++-
 mm/hmm.c   |  5 +++--
 mm/memory_hotplug.c|  7 +++
 10 files changed, 39 insertions(+), 29 deletions(-)

diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
index 7af4e05bb61e..2e2e4f532204 100644
--- a/arch/ia64/mm/init.c
+++ b/arch/ia64/mm/init.c
@@ -647,13 +647,14 @@ mem_init (void)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG
-int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
+int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap,
+   bool want_memblock)
 {
unsigned long start_pfn = start >> PAGE_SHIFT;
unsigned long nr_pages = size >> PAGE_SHIFT;
int ret;
 
-   ret = __add_pages(nid, start_pfn, nr_pages, want_memblock);
+   ret = __add_pages(nid, start_pfn, nr_pages, altmap, want_memblock);
if (ret)
printk("%s: Problem encountered in __add_pages() as ret=%d\n",
   __func__,  ret);
diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
index 4362b86ef84c..e670cfc2766e 100644
--- a/arch/powerpc/mm/mem.c
+++ b/arch/powerpc/mm/mem.c
@@ -127,7 +127,8 @@ int __weak remove_section_mapping(unsigned long start, 
unsigned long end)
return -ENODEV;
 }
 
-int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
+int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap,
+   bool want_memblock)
 {
unsigned long start_pfn = start >> PAGE_SHIFT;
unsigned long nr_pages = size >> PAGE_SHIFT;
@@ -144,7 +145,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool 
want_memblock)
return -EFAULT;
}
 
-   return __add_pages(nid, start_pfn, nr_pages, want_memblock);
+   return __add_pages(nid, start_pfn, nr_pages, altmap, want_memblock);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
index 671535e64aba..e12c5af50cd7 100644
--- a/arch/s390/mm/init.c
+++ b/arch/s390/mm/init.c
@@ -222,7 +222,8 @@ device_initcall(s390_cma_mem_init);
 
 #endif /* CONFIG_CMA */
 
-int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
+int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap,
+   bool want_memblock)
 {
unsigned long start_pfn = PFN_DOWN(start);
unsigned long size_pages = PFN_DOWN(size);
@@ -232,7 +233,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool 
want_memblock)
if (rc)
return rc;
 
-   rc = __add_pages(nid, start_pfn, size_pages, want_memblock);
+   rc = __add_pages(nid, start_pfn, size_pages, altmap, want_memblock);
if (rc)
vmem_remove_mapping(start, size);
return rc;
diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
index afc54d593a26..552afbf55bad 100644
--- a/arch/sh/mm/init.c
+++ b/arch/sh/mm/init.c
@@ -485,14 +485,15 @@ void free_initrd_mem(unsigned long start, unsigned long 
end)
 #endif
 
 #ifdef CONFIG_MEMORY_HOTPLUG
-int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
+int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap,
+   bool want_memblock)
 {
unsigned long start_pfn = PFN_DOWN(start);
unsigned long nr_pages = size >> PAGE_SHIFT;
int ret;
 
/* We only have ZONE_NORMAL, so this is easy.. */
-   ret = __add_pages(nid, start_pfn, nr_pages, want_memblock);
+   ret = __add_pages(nid, start_pfn, nr_pages, altmap, want_memblock);
if (unlikely(ret))
printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
 
diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
index 135c9a7898c7..8a3091511a71 100644
--- a/arch/x86/mm/init_32.c
+++ b/arch/x86/mm/init_32.c
@@ -829,12 +829,13 @@ void __init mem_init(void)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG
-int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
+int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap,
+   bool want_memblock)
 {
unsigned long start_pfn = start >> PAGE_SHIFT;
unsigned long nr_pages = size >> PAGE_SHIFT;
 
-   return __add_pages(nid, start_pfn, nr_pages, want_memblock);
+   return __add_pages(nid, start_pfn, nr_pages, altmap, want_memblock);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 8acdc35c2dfa..e80bb4189254 100644
---