Re: [RFC 0/5] Support non-coherent DMA on RISC-V using a global pool
On Fri, 23 Jul 2021 14:40:26 PDT (-0700), Atish Patra wrote: RISC-V privilege specification doesn't define an IOMMU or any method modify PMA attributes or PTE entries to allow non-coherent mappings yet. In the beginning, this approach was adopted assuming that most of the RISC-V platforms would support full cache-coherent IO. Here is excerpt from the priv spec section 3.6.5 "In RISC-V platforms, the use of hardware-incoherent regions is discouraged due to software complexity, performance, and energy impacts." While some of the RISC-V platforms adhere to the above suggestion, not all platforms can afford to build to fully cache-coherent I/O devices. To address DMA for non-coherent I/O devices, we need to mark a region of memory as non-cacheable. Some of the platforms rely on a fixed region of uncached memory that is remapped to the system memory while some other modify the PTE entries to achieve that. The patch3 solves the issue for the fist use case by using a global pool of memory that is reserved for DMA. The device access the reserved area of the region while corresponding CPU address will be from uncached region As the uncached region is remapped to the beginning of the system ram, both CPU and device get the same view. To facilitate streaming DMA APIs, patch 1 introduces a set of generic cache operations. Any platform can use the generic ops to provide platform specific cache management operations. Once the standard RISC-V CMO extension is available, it will also use these generic ops. To address the second use case, Page Based Memory Attribute (PBMT) extension is proposed. Once the extension is in good shape, we can leverage that using CONFIG_DIRECT_REMAP. Currently, it is selected via a compile time config option. We will probably need another arch specific hooks to know if the platform supports direct remap at runtime. For RISC-V, it will check the presence of PBMT extension while other arch function will simply return true IIUC this is another extension that's not yet frozen or implemented in hardware? Is this one compatible with what's in the c906, or is it doing things its own way? if DIRECT_REMAP is enabled. This is required as arious different config (DIRECT_REMAP, GLOBAL_POOL) will be enabled in the defconfig so that a unified kernel image can boot on all RISC-V platforms. This patch series depends on Christoph's global pool support series[1]. Tested on Qemu, HiFive unleashed and beagleV. This series is also available at [2]. This series also solves the non-coherent DMA support on beagleV but requires additional beagleV specific patches[3] which will be upstreamed soon. [1] https://lists.linuxfoundation.org/pipermail/iommu/2021-July/057266.html [2] https://github.com/atishp04/linux/tree/riscv_nc_global_pool [3] https://github.com/atishp04/linux/tree/wip_beaglev_dma_nc_global Atish Patra (5): RISC-V: Implement arch_sync_dma* functions of: Move of_dma_get_range to of_address.h dma-mapping: Enable global non-coherent pool support for RISC-V dma-direct: Allocate dma pages directly if global pool allocation fails RISC-V: Support a new config option for non-coherent DMA arch/riscv/Kconfig | 8 +++ arch/riscv/include/asm/dma-noncoherent.h | 19 +++ arch/riscv/mm/Makefile | 1 + arch/riscv/mm/dma-noncoherent.c | 66 drivers/of/of_private.h | 10 include/linux/of_address.h | 12 + kernel/dma/coherent.c| 49 +++--- kernel/dma/direct.c | 7 ++- 8 files changed, 152 insertions(+), 20 deletions(-) create mode 100644 arch/riscv/include/asm/dma-noncoherent.h create mode 100644 arch/riscv/mm/dma-noncoherent.c Can you guys please make up your minds about how this is going to be supported at the ISA level? I get a different answer every day here: sometimes it's that these systems are not compliant, sometimes that Linux is the compliance suite, sometimes that we're doing an ISA extension, and sometimes that we're doing the SBI stuff. I don't really care all that much about what the decision is, but it's impossible to move forward without some semblance of a plan. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file
On Mon, 16 Sep 2019 11:18:00 PDT (-0700), w...@kernel.org wrote: On Sun, Sep 15, 2019 at 05:03:38AM +, Anup Patel wrote: > -Original Message- > From: linux-kernel-ow...@vger.kernel.org ow...@vger.kernel.org> On Behalf Of Palmer Dabbelt > Sent: Saturday, September 14, 2019 7:31 PM > To: w...@kernel.org > Cc: guo...@kernel.org; Will Deacon ; > julien.thie...@arm.com; a...@eecs.berkeley.edu; james.mo...@arm.com; > Arnd Bergmann ; suzuki.poul...@arm.com; > marc.zyng...@arm.com; catalin.mari...@arm.com; Anup Patel > ; linux-ker...@vger.kernel.org; > r...@linux.ibm.com; Christoph Hellwig ; Atish Patra > ; julien.gr...@arm.com; g...@garyguo.net; Paul > Walmsley ; christoffer.d...@arm.com; linux- > ri...@lists.infradead.org; kvm...@lists.cs.columbia.edu; linux-arm- > ker...@lists.infradead.org; iommu@lists.linux-foundation.org > Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a > separate file > > On Thu, 12 Sep 2019 07:02:56 PDT (-0700), w...@kernel.org wrote: > > On Sun, Sep 08, 2019 at 07:52:55AM +0800, Guo Ren wrote: > >> On Mon, Jun 24, 2019 at 6:40 PM Will Deacon wrote: > >> > > I'll keep my system use the same ASID for SMP + IOMMU :P > >> > > >> > You will want a separate allocator for that: > >> > > >> > https://lkml.kernel.org/r/20190610184714.6786-2-jean-philippe.bruck > >> > e...@arm.com > >> > >> Yes, it is hard to maintain ASID between IOMMU and CPUMMU or > >> different system, because it's difficult to synchronize the IO_ASID > >> when the CPU ASID is rollover. > >> But we could still use hardware broadcast TLB invalidation > >> instruction to uniformly manage the ASID and IO_ASID, or OTHER_ASID in > our IOMMU. > > > > That's probably a bad idea, because you'll likely stall execution on > > the CPU until the IOTLB has completed invalidation. In the case of > > ATS, I think an endpoint ATC is permitted to take over a minute to > > respond. In reality, I suspect the worst you'll ever see would be in > > the msec range, but that's still an unacceptable period of time to hold a > CPU. > > > >> Welcome to join our disscusion: > >> "Introduce an implementation of IOMMU in linux-riscv" > >> 9 Sep 2019, 10:45 Jade-room-I (Corinthia Hotel Lisbon) RISC-V MC > > > > I attended this session, but it unfortunately raised many more > > questions than it answered. > > Ya, we're a long way from figuring this out. For everyone's reference, here is our first attempt at RISC-V ASID allocator: http://archive.lwn.net:8080/linux-kernel/20190329045111.14040-1-anup.pa...@wdc.com/T/#u With a reply stating that the patch "absolutely does not work" ;) What exactly do you want people to do with that? It's an awful lot of effort to review this sort of stuff and given that Guo Ren is talking about sharing page tables between the CPU and an accelerator, maybe you're better off stabilising Linux for the platforms that you can actually test rather than getting so far ahead of yourselves that you end up with a bunch of wasted work on patches that probably won't get merged any time soon. Seriously, they say "walk before you can run", but this is more "crawl before you can fly". What's the rush? I agree, and I think I've been pretty clear here: we're not merging this ASID stuff until we have a platform we can test on, particularly as the platforms we have now already need some wacky hacks around TLB flushing that we haven't gotten to the bottom of. Will
Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file
On Thu, 12 Sep 2019 07:02:56 PDT (-0700), w...@kernel.org wrote: On Sun, Sep 08, 2019 at 07:52:55AM +0800, Guo Ren wrote: On Mon, Jun 24, 2019 at 6:40 PM Will Deacon wrote: > > I'll keep my system use the same ASID for SMP + IOMMU :P > > You will want a separate allocator for that: > > https://lkml.kernel.org/r/20190610184714.6786-2-jean-philippe.bruc...@arm.com Yes, it is hard to maintain ASID between IOMMU and CPUMMU or different system, because it's difficult to synchronize the IO_ASID when the CPU ASID is rollover. But we could still use hardware broadcast TLB invalidation instruction to uniformly manage the ASID and IO_ASID, or OTHER_ASID in our IOMMU. That's probably a bad idea, because you'll likely stall execution on the CPU until the IOTLB has completed invalidation. In the case of ATS, I think an endpoint ATC is permitted to take over a minute to respond. In reality, I suspect the worst you'll ever see would be in the msec range, but that's still an unacceptable period of time to hold a CPU. Welcome to join our disscusion: "Introduce an implementation of IOMMU in linux-riscv" 9 Sep 2019, 10:45 Jade-room-I (Corinthia Hotel Lisbon) RISC-V MC I attended this session, but it unfortunately raised many more questions than it answered. Ya, we're a long way from figuring this out.
Re: [PATCH 2/2] arch: switch the default on ARCH_HAS_SG_CHAIN
On Mon, 19 Nov 2018 05:58:52 PST (-0800), Christoph Hellwig wrote: On Fri, Nov 16, 2018 at 08:52:14AM -0800, Palmer Dabbelt wrote: As far as I can tell, it looks like m68k, mips, and powerpc mention an IOMMU in their ports, don't set ARCH_HAS_SG_CHAIN, and with this patch set won't set ARCH_NO_SG_CHAIN. m68k has no iommu, and not operations that operate on a scatterlist. mips has a trivial iommu driver (jazzdma), but I wrote the current instance of it, nad it is fine. powerpc has various iommu, but actually enables ARCH_HAS_SG_CHAIN unconditionally. The issue is that I'm not sure how to determine what constitutes a horrible legacy IOMMU, at least with respect to not being able to use scatterlist chaining. It basically means someone is iterating using manual pointer arithmetics over a multi-element scatterlist instead of using the sg_next and for_each_sg helpers. Ah, OK. In that case, feel free to drop a Reviewed-by: Palmer Dabbelt in there, if you think it helps any. Thanks for the help! ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 2/2] arch: switch the default on ARCH_HAS_SG_CHAIN
On Fri, 09 Nov 2018 01:00:08 PST (-0800), Christoph Hellwig wrote: These days architectures are mostly out of the business of dealing with struct scatterlist at all, unless they have architecture specific iommu drivers. Replace the ARCH_HAS_SG_CHAIN symbol with a ARCH_NO_SG_CHAIN one only enabled for architectures with horrible legacy iommu drivers like alpha and parisc, and conditionally for arm which wants to keep it disable for legacy platforms. Signed-off-by: Christoph Hellwig --- .../features/io/sg-chain/arch-support.txt | 33 --- arch/alpha/Kconfig| 1 + arch/arc/Kconfig | 1 - arch/arm/Kconfig | 2 +- arch/arm64/Kconfig| 1 - arch/ia64/Kconfig | 1 - arch/parisc/Kconfig | 1 + arch/powerpc/Kconfig | 1 - arch/s390/Kconfig | 1 - arch/sparc/Kconfig| 1 - arch/x86/Kconfig | 1 - arch/xtensa/Kconfig | 1 - include/linux/scatterlist.h | 6 ++-- lib/Kconfig | 2 +- lib/scatterlist.c | 2 +- 15 files changed, 8 insertions(+), 47 deletions(-) delete mode 100644 Documentation/features/io/sg-chain/arch-support.txt As far as I can tell, it looks like m68k, mips, and powerpc mention an IOMMU in their ports, don't set ARCH_HAS_SG_CHAIN, and with this patch set won't set ARCH_NO_SG_CHAIN. The issue is that I'm not sure how to determine what constitutes a horrible legacy IOMMU, at least with respect to not being able to use scatterlist chaining. I poked through the 68k stuff (sun3 and sun3x) and the only thing I see that's odd is that DVMA_PAGE_SHIFT is 13. That matches PAGE_SHIFT for the subarch, though, so I don't think it's an issue. It looks like the mention in MIPS is trivial, so that should be fine. There's a lot in the PC port so I just glanced at it. I assume it's OK, but I did see an explicit mention of SG in their generic IOMMU. There's a lot of flavors here so I'm not sure how well they all work. I don't really think that's enough for my reviewed-by tag, so feel free to give me some pointers if you're looking for one -- I'd be happy to lean about IOMMUs (particularly broken ones, as that might help prevent us from designing a broken one), I'm just not sure what I'm looking for. I'm going to assume you're going to merge this and I'll drop the RISC-V patch that turns on ARCH_HAS_SG_CHAIN. Thanks! diff --git a/Documentation/features/io/sg-chain/arch-support.txt b/Documentation/features/io/sg-chain/arch-support.txt deleted file mode 100644 index 6554f0372c3f.. --- a/Documentation/features/io/sg-chain/arch-support.txt +++ /dev/null @@ -1,33 +0,0 @@ -# -# Feature name: sg-chain -# Kconfig: ARCH_HAS_SG_CHAIN -# description: arch supports chained scatter-gather lists -# ---- -| arch |status| ---- -| alpha: | TODO | -| arc: | ok | -| arm: | ok | -| arm64: | ok | -| c6x: | TODO | -| h8300: | TODO | -| hexagon: | TODO | -|ia64: | ok | -|m68k: | TODO | -| microblaze: | TODO | -|mips: | TODO | -| nds32: | TODO | -| nios2: | TODO | -|openrisc: | TODO | -| parisc: | TODO | -| powerpc: | ok | -| riscv: | TODO | -|s390: | ok | -| sh: | TODO | -| sparc: | ok | -| um: | TODO | -| unicore32: | TODO | -| x86: | ok | -| xtensa: | TODO | ---- diff --git a/arch/alpha/Kconfig b/arch/alpha/Kconfig index 5b4f88363453..a7e748a46c18 100644 --- a/arch/alpha/Kconfig +++ b/arch/alpha/Kconfig @@ -5,6 +5,7 @@ config ALPHA select ARCH_MIGHT_HAVE_PC_PARPORT select ARCH_MIGHT_HAVE_PC_SERIO select ARCH_NO_PREEMPT + select ARCH_NO_SG_CHAIN select ARCH_USE_CMPXCHG_LOCKREF select HAVE_AOUT select HAVE_IDE diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig index c9e2a1323536..fd48d698da29 100644 --- a/arch/arc/Kconfig +++ b/arch/arc/Kconfig @@ -13,7 +13,6 @@ config ARC select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_SYNC_DMA_FOR_CPU select ARCH_HAS_SYNC_DMA_FOR_DEVICE - select ARCH_HAS_SG_CHAIN select ARCH_SUPPORTS_ATOMIC_RMW if ARC_HAS_LLSC select BUILDTIME_EXTABLE_SORT select CLONE_BACKWARDS diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 91be74d8df65..199eb62230b0 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -19,6 +19,7 @@ config ARM select ARCH_HAVE_CUSTOM_GPIO_H select
Re: [PATCH 1/2] csky, h8300, riscv: remove leftovers
On Fri, 09 Nov 2018 01:00:07 PST (-0800), Christoph Hellwig wrote: There has been no for a long time, which also means there is no point in using it from asm-generic. Signed-off-by: Christoph Hellwig --- arch/csky/include/asm/Kbuild | 1 - arch/h8300/include/asm/Kbuild | 1 - arch/riscv/include/asm/Kbuild | 1 - 3 files changed, 3 deletions(-) diff --git a/arch/csky/include/asm/Kbuild b/arch/csky/include/asm/Kbuild index 2a0abe8f2a35..7c48a123300d 100644 --- a/arch/csky/include/asm/Kbuild +++ b/arch/csky/include/asm/Kbuild @@ -34,7 +34,6 @@ generic-y += pci.h generic-y += percpu.h generic-y += preempt.h generic-y += qrwlock.h -generic-y += scatterlist.h generic-y += sections.h generic-y += serial.h generic-y += shm.h diff --git a/arch/h8300/include/asm/Kbuild b/arch/h8300/include/asm/Kbuild index a5d0b2991f47..32f0c8952147 100644 --- a/arch/h8300/include/asm/Kbuild +++ b/arch/h8300/include/asm/Kbuild @@ -36,7 +36,6 @@ generic-y += parport.h generic-y += percpu.h generic-y += pgalloc.h generic-y += preempt.h -generic-y += scatterlist.h generic-y += sections.h generic-y += serial.h generic-y += sizes.h diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 6a646d9ea780..011cc7c7b3ad 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -37,7 +37,6 @@ generic-y += poll.h generic-y += posix_types.h generic-y += preempt.h generic-y += resource.h -generic-y += scatterlist.h generic-y += sections.h generic-y += sembuf.h generic-y += serial.h Reviewed-by: Palmer Dabbelt ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: add swiotlb support for riscv
On Wed, 16 May 2018 07:42:59 PDT (-0700), Christoph Hellwig wrote: Current 64-bit RISC-V cores don't have any iommu support, so they need iommu support when used with more than 4GB DRAM (or even just address space depending on the SBI). This wires the support up based on earlier patches from Palmer. Yep: the FU540-C000 has a feature where DRAM is always mapped at 2GiB physical, so if we have more than 2GiB of DRAM (we have 8GiB on the only board it'll probably ever be attached to) then we need swiotlb to do 32-bit DMA. Note that the patches are on to of the dma mapping tree as they require the swiotlb Kconfig consolidation, and should probably merged through that tree as well. Git tree: git://git.infradead.org/users/hch/misc.git riscv-swiotlb Gitweb: http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/riscv-swiotlb Ah, great! This is much cleaner than our current implementation, which isn't based on the generic stuff :). I'm happy to have these through that tree if it makes the merging easier, feel free to add an Acked-by: Palmer Dabbelt <pal...@sifive.com> and/or a Reviewed-by: Palmer Dabbelt <pal...@sifive.com> on all 3 of these patches if that helps. I'm going to park them on my next-swiotlb branch so I can use them, but I won't submit them myself unless someone says something. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 5/5] PCI: remove PCI_DMA_BUS_IS_PHYS
On Tue, 24 Apr 2018 11:16:25 PDT (-0700), Christoph Hellwig wrote: This was used by the ide, scsi and networking code in the past to determine if they should bounce payloads. Now that the dma mapping always have to support dma to all physical memory (thanks to swiotlb for non-iommu systems) there is no need to this crude hack any more. Signed-off-by: Christoph Hellwig <h...@lst.de> --- [...] diff --git a/arch/riscv/include/asm/pci.h b/arch/riscv/include/asm/pci.h index 0f2fc9ef20fc..b3638c505728 100644 --- a/arch/riscv/include/asm/pci.h +++ b/arch/riscv/include/asm/pci.h @@ -26,9 +26,6 @@ /* RISC-V shim does not initialize PCI bus */ #define pcibios_assign_all_busses() 1 -/* We do not have an IOMMU */ -#define PCI_DMA_BUS_IS_PHYS 1 - extern int isa_dma_bridge_buggy; #ifdef CONFIG_PCI Thanks! Acked-by: Palmer Dabbelt <pal...@sifive.com> (For the RISC-V change) ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
