Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
Am 17.03.23 um 15:45 schrieb Alex Deucher: On Thu, Mar 16, 2023 at 7:09 PM Stefano Stabellini wrote: On Thu, 16 Mar 2023, Juergen Gross wrote: On 16.03.23 14:53, Alex Deucher wrote: On Thu, Mar 16, 2023 at 9:48 AM Juergen Gross wrote: On 16.03.23 14:45, Alex Deucher wrote: On Thu, Mar 16, 2023 at 3:50 AM Jan Beulich wrote: On 16.03.2023 00:25, Stefano Stabellini wrote: On Wed, 15 Mar 2023, Jan Beulich wrote: On 15.03.2023 01:52, Stefano Stabellini wrote: On Mon, 13 Mar 2023, Jan Beulich wrote: On 12.03.2023 13:01, Huang Rui wrote: Xen PVH is the paravirtualized mode and takes advantage of hardware virtualization support when possible. It will using the hardware IOMMU support instead of xen-swiotlb, so disable swiotlb if current domain is Xen PVH. But the kernel has no way (yet) to drive the IOMMU, so how can it get away without resorting to swiotlb in certain cases (like I/O to an address-restricted device)? I think Ray meant that, thanks to the IOMMU setup by Xen, there is no need for swiotlb-xen in Dom0. Address translations are done by the IOMMU so we can use guest physical addresses instead of machine addresses for DMA. This is a similar case to Dom0 on ARM when the IOMMU is available (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding case is XENFEAT_not_direct_mapped). But how does Xen using an IOMMU help with, as said, address-restricted devices? They may still need e.g. a 32-bit address to be programmed in, and if the kernel has memory beyond the 4G boundary not all I/O buffers may fulfill this requirement. In short, it is going to work as long as Linux has guest physical addresses (not machine addresses, those could be anything) lower than 4GB. If the address-restricted device does DMA via an IOMMU, then the device gets programmed by Linux using its guest physical addresses (not machine addresses). The 32-bit restriction would be applied by Linux to its choice of guest physical address to use to program the device, the same way it does on native. The device would be fine as it always uses Linux-provided <4GB addresses. After the IOMMU translation (pagetable setup by Xen), we could get any address, including >4GB addresses, and that is expected to work. I understand that's the "normal" way of working. But whatever the swiotlb is used for in baremetal Linux, that would similarly require its use in PVH (or HVM) aiui. So unconditionally disabling it in PVH would look to me like an incomplete attempt to disable its use altogether on x86. What difference of PVH vs baremetal am I missing here? swiotlb is not usable for GPUs even on bare metal. They often have hundreds or megs or even gigs of memory mapped on the device at any given time. Also, AMD GPUs support 44-48 bit DMA masks (depending on the chip family). But the swiotlb isn't per device, but system global. Sure, but if the swiotlb is in use, then you can't really use the GPU. So you get to pick one. The swiotlb is used only for buffers which are not within the DMA mask of a device (see dma_direct_map_page()). So an AMD GPU supporting a 44 bit DMA mask won't use the swiotlb unless you have a buffer above guest physical address of 16TB (so basically never). Disabling swiotlb in such a guest would OTOH mean, that a device with only 32 bit DMA mask passed through to this guest couldn't work with buffers above 4GB. I don't think this is acceptable. From the Xen subsystem in Linux point of view, the only thing we need to do is to make sure *not* to enable swiotlb_xen (yes "swiotlb_xen", not the global swiotlb) on PVH because it is not needed anyway. I think we should leave the global "swiotlb" setting alone. The global swiotlb is not relevant to Xen anyway, and surely baremetal Linux has to have a way to deal with swiotlb/GPU incompatibilities. We just have to avoid making things worse on Xen, and for that we just need to avoid unconditionally enabling swiotlb-xen. If the Xen subsystem doesn't enable swiotlb_xen/swiotlb, and no other subsystem enables swiotlb, then we have a good Linux configuration capable of handling the GPU properly. Alex, please correct me if I am wrong. How is x86_swiotlb_enable set to false on native (non-Xen) x86? In most cases we have an IOMMU enabled and IIRC, TTM has slightly different behavior for memory allocation depending on whether swiotlb would be needed or not. Well "slightly different" is an understatement. We need to disable quite a bunch of features to make swiotlb work with GPUs. Especially userptr and inter device sharing won't work any more. Regards, Christian. Alex
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On Thu, Mar 16, 2023 at 7:09 PM Stefano Stabellini wrote: > > On Thu, 16 Mar 2023, Juergen Gross wrote: > > On 16.03.23 14:53, Alex Deucher wrote: > > > On Thu, Mar 16, 2023 at 9:48 AM Juergen Gross wrote: > > > > > > > > On 16.03.23 14:45, Alex Deucher wrote: > > > > > On Thu, Mar 16, 2023 at 3:50 AM Jan Beulich wrote: > > > > > > > > > > > > On 16.03.2023 00:25, Stefano Stabellini wrote: > > > > > > > On Wed, 15 Mar 2023, Jan Beulich wrote: > > > > > > > > On 15.03.2023 01:52, Stefano Stabellini wrote: > > > > > > > > > On Mon, 13 Mar 2023, Jan Beulich wrote: > > > > > > > > > > On 12.03.2023 13:01, Huang Rui wrote: > > > > > > > > > > > Xen PVH is the paravirtualized mode and takes advantage of > > > > > > > > > > > hardware > > > > > > > > > > > virtualization support when possible. It will using the > > > > > > > > > > > hardware IOMMU > > > > > > > > > > > support instead of xen-swiotlb, so disable swiotlb if > > > > > > > > > > > current domain is > > > > > > > > > > > Xen PVH. > > > > > > > > > > > > > > > > > > > > But the kernel has no way (yet) to drive the IOMMU, so how > > > > > > > > > > can > > > > > > > > > > it get > > > > > > > > > > away without resorting to swiotlb in certain cases (like I/O > > > > > > > > > > to an > > > > > > > > > > address-restricted device)? > > > > > > > > > > > > > > > > > > I think Ray meant that, thanks to the IOMMU setup by Xen, > > > > > > > > > there > > > > > > > > > is no > > > > > > > > > need for swiotlb-xen in Dom0. Address translations are done by > > > > > > > > > the IOMMU > > > > > > > > > so we can use guest physical addresses instead of machine > > > > > > > > > addresses for > > > > > > > > > DMA. This is a similar case to Dom0 on ARM when the IOMMU is > > > > > > > > > available > > > > > > > > > (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the > > > > > > > > > corresponding > > > > > > > > > case is XENFEAT_not_direct_mapped). > > > > > > > > > > > > > > > > But how does Xen using an IOMMU help with, as said, > > > > > > > > address-restricted > > > > > > > > devices? They may still need e.g. a 32-bit address to be > > > > > > > > programmed in, > > > > > > > > and if the kernel has memory beyond the 4G boundary not all I/O > > > > > > > > buffers > > > > > > > > may fulfill this requirement. > > > > > > > > > > > > > > In short, it is going to work as long as Linux has guest physical > > > > > > > addresses (not machine addresses, those could be anything) lower > > > > > > > than > > > > > > > 4GB. > > > > > > > > > > > > > > If the address-restricted device does DMA via an IOMMU, then the > > > > > > > device > > > > > > > gets programmed by Linux using its guest physical addresses (not > > > > > > > machine > > > > > > > addresses). > > > > > > > > > > > > > > The 32-bit restriction would be applied by Linux to its choice of > > > > > > > guest > > > > > > > physical address to use to program the device, the same way it > > > > > > > does > > > > > > > on > > > > > > > native. The device would be fine as it always uses Linux-provided > > > > > > > <4GB > > > > > > > addresses. After the IOMMU translation (pagetable setup by Xen), > > > > > > > we > > > > > > > could get any address, including >4GB addresses, and that is > > > > > > > expected to > > > > > > > work. > > > > > > > > > > > > I understand that's the "normal" way of working. But whatever the > > > > > > swiotlb > > > > > > is used for in baremetal Linux, that would similarly require its use > > > > > > in > > > > > > PVH (or HVM) aiui. So unconditionally disabling it in PVH would look > > > > > > to > > > > > > me like an incomplete attempt to disable its use altogether on x86. > > > > > > What > > > > > > difference of PVH vs baremetal am I missing here? > > > > > > > > > > swiotlb is not usable for GPUs even on bare metal. They often have > > > > > hundreds or megs or even gigs of memory mapped on the device at any > > > > > given time. Also, AMD GPUs support 44-48 bit DMA masks (depending on > > > > > the chip family). > > > > > > > > But the swiotlb isn't per device, but system global. > > > > > > Sure, but if the swiotlb is in use, then you can't really use the GPU. > > > So you get to pick one. > > > > The swiotlb is used only for buffers which are not within the DMA mask of a > > device (see dma_direct_map_page()). So an AMD GPU supporting a 44 bit DMA > > mask > > won't use the swiotlb unless you have a buffer above guest physical address > > of > > 16TB (so basically never). > > > > Disabling swiotlb in such a guest would OTOH mean, that a device with only > > 32 bit DMA mask passed through to this guest couldn't work with buffers > > above 4GB. > > > > I don't think this is acceptable. > > From the Xen subsystem in Linux point of view, the only thing we need to > do is to make sure *not* to enable swiotlb_xen (yes "swiotlb_xen", not > the global swiotlb) on PVH because it is not needed anyway. > > I think we should leave the global "swiotlb"
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On Thu, Mar 16, 2023 at 04:09:44PM -0700, Stefano Stabellini wrote: > On Thu, 16 Mar 2023, Juergen Gross wrote: > > On 16.03.23 14:53, Alex Deucher wrote: > > > On Thu, Mar 16, 2023 at 9:48 AM Juergen Gross wrote: > > > > > > > > On 16.03.23 14:45, Alex Deucher wrote: > > > > > On Thu, Mar 16, 2023 at 3:50 AM Jan Beulich wrote: > > > > > > > > > > > > On 16.03.2023 00:25, Stefano Stabellini wrote: > > > > > > > On Wed, 15 Mar 2023, Jan Beulich wrote: > > > > > > > > On 15.03.2023 01:52, Stefano Stabellini wrote: > > > > > > > > > On Mon, 13 Mar 2023, Jan Beulich wrote: > > > > > > > > > > On 12.03.2023 13:01, Huang Rui wrote: > > > > > > > > > > > Xen PVH is the paravirtualized mode and takes advantage of > > > > > > > > > > > hardware > > > > > > > > > > > virtualization support when possible. It will using the > > > > > > > > > > > hardware IOMMU > > > > > > > > > > > support instead of xen-swiotlb, so disable swiotlb if > > > > > > > > > > > current domain is > > > > > > > > > > > Xen PVH. > > > > > > > > > > > > > > > > > > > > But the kernel has no way (yet) to drive the IOMMU, so how > > > > > > > > > > can > > > > > > > > > > it get > > > > > > > > > > away without resorting to swiotlb in certain cases (like I/O > > > > > > > > > > to an > > > > > > > > > > address-restricted device)? > > > > > > > > > > > > > > > > > > I think Ray meant that, thanks to the IOMMU setup by Xen, > > > > > > > > > there > > > > > > > > > is no > > > > > > > > > need for swiotlb-xen in Dom0. Address translations are done by > > > > > > > > > the IOMMU > > > > > > > > > so we can use guest physical addresses instead of machine > > > > > > > > > addresses for > > > > > > > > > DMA. This is a similar case to Dom0 on ARM when the IOMMU is > > > > > > > > > available > > > > > > > > > (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the > > > > > > > > > corresponding > > > > > > > > > case is XENFEAT_not_direct_mapped). > > > > > > > > > > > > > > > > But how does Xen using an IOMMU help with, as said, > > > > > > > > address-restricted > > > > > > > > devices? They may still need e.g. a 32-bit address to be > > > > > > > > programmed in, > > > > > > > > and if the kernel has memory beyond the 4G boundary not all I/O > > > > > > > > buffers > > > > > > > > may fulfill this requirement. > > > > > > > > > > > > > > In short, it is going to work as long as Linux has guest physical > > > > > > > addresses (not machine addresses, those could be anything) lower > > > > > > > than > > > > > > > 4GB. > > > > > > > > > > > > > > If the address-restricted device does DMA via an IOMMU, then the > > > > > > > device > > > > > > > gets programmed by Linux using its guest physical addresses (not > > > > > > > machine > > > > > > > addresses). > > > > > > > > > > > > > > The 32-bit restriction would be applied by Linux to its choice of > > > > > > > guest > > > > > > > physical address to use to program the device, the same way it > > > > > > > does > > > > > > > on > > > > > > > native. The device would be fine as it always uses Linux-provided > > > > > > > <4GB > > > > > > > addresses. After the IOMMU translation (pagetable setup by Xen), > > > > > > > we > > > > > > > could get any address, including >4GB addresses, and that is > > > > > > > expected to > > > > > > > work. > > > > > > > > > > > > I understand that's the "normal" way of working. But whatever the > > > > > > swiotlb > > > > > > is used for in baremetal Linux, that would similarly require its use > > > > > > in > > > > > > PVH (or HVM) aiui. So unconditionally disabling it in PVH would look > > > > > > to > > > > > > me like an incomplete attempt to disable its use altogether on x86. > > > > > > What > > > > > > difference of PVH vs baremetal am I missing here? > > > > > > > > > > swiotlb is not usable for GPUs even on bare metal. They often have > > > > > hundreds or megs or even gigs of memory mapped on the device at any > > > > > given time. Also, AMD GPUs support 44-48 bit DMA masks (depending on > > > > > the chip family). > > > > > > > > But the swiotlb isn't per device, but system global. > > > > > > Sure, but if the swiotlb is in use, then you can't really use the GPU. > > > So you get to pick one. > > > > The swiotlb is used only for buffers which are not within the DMA mask of a > > device (see dma_direct_map_page()). So an AMD GPU supporting a 44 bit DMA > > mask > > won't use the swiotlb unless you have a buffer above guest physical address > > of > > 16TB (so basically never). > > > > Disabling swiotlb in such a guest would OTOH mean, that a device with only > > 32 bit DMA mask passed through to this guest couldn't work with buffers > > above 4GB. > > > > I don't think this is acceptable. > > From the Xen subsystem in Linux point of view, the only thing we need to > do is to make sure *not* to enable swiotlb_xen (yes "swiotlb_xen", not > the global swiotlb) on PVH because it is not needed anyway. But this is already the case
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On Thu, 16 Mar 2023, Juergen Gross wrote: > On 16.03.23 14:53, Alex Deucher wrote: > > On Thu, Mar 16, 2023 at 9:48 AM Juergen Gross wrote: > > > > > > On 16.03.23 14:45, Alex Deucher wrote: > > > > On Thu, Mar 16, 2023 at 3:50 AM Jan Beulich wrote: > > > > > > > > > > On 16.03.2023 00:25, Stefano Stabellini wrote: > > > > > > On Wed, 15 Mar 2023, Jan Beulich wrote: > > > > > > > On 15.03.2023 01:52, Stefano Stabellini wrote: > > > > > > > > On Mon, 13 Mar 2023, Jan Beulich wrote: > > > > > > > > > On 12.03.2023 13:01, Huang Rui wrote: > > > > > > > > > > Xen PVH is the paravirtualized mode and takes advantage of > > > > > > > > > > hardware > > > > > > > > > > virtualization support when possible. It will using the > > > > > > > > > > hardware IOMMU > > > > > > > > > > support instead of xen-swiotlb, so disable swiotlb if > > > > > > > > > > current domain is > > > > > > > > > > Xen PVH. > > > > > > > > > > > > > > > > > > But the kernel has no way (yet) to drive the IOMMU, so how can > > > > > > > > > it get > > > > > > > > > away without resorting to swiotlb in certain cases (like I/O > > > > > > > > > to an > > > > > > > > > address-restricted device)? > > > > > > > > > > > > > > > > I think Ray meant that, thanks to the IOMMU setup by Xen, there > > > > > > > > is no > > > > > > > > need for swiotlb-xen in Dom0. Address translations are done by > > > > > > > > the IOMMU > > > > > > > > so we can use guest physical addresses instead of machine > > > > > > > > addresses for > > > > > > > > DMA. This is a similar case to Dom0 on ARM when the IOMMU is > > > > > > > > available > > > > > > > > (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the > > > > > > > > corresponding > > > > > > > > case is XENFEAT_not_direct_mapped). > > > > > > > > > > > > > > But how does Xen using an IOMMU help with, as said, > > > > > > > address-restricted > > > > > > > devices? They may still need e.g. a 32-bit address to be > > > > > > > programmed in, > > > > > > > and if the kernel has memory beyond the 4G boundary not all I/O > > > > > > > buffers > > > > > > > may fulfill this requirement. > > > > > > > > > > > > In short, it is going to work as long as Linux has guest physical > > > > > > addresses (not machine addresses, those could be anything) lower > > > > > > than > > > > > > 4GB. > > > > > > > > > > > > If the address-restricted device does DMA via an IOMMU, then the > > > > > > device > > > > > > gets programmed by Linux using its guest physical addresses (not > > > > > > machine > > > > > > addresses). > > > > > > > > > > > > The 32-bit restriction would be applied by Linux to its choice of > > > > > > guest > > > > > > physical address to use to program the device, the same way it does > > > > > > on > > > > > > native. The device would be fine as it always uses Linux-provided > > > > > > <4GB > > > > > > addresses. After the IOMMU translation (pagetable setup by Xen), we > > > > > > could get any address, including >4GB addresses, and that is > > > > > > expected to > > > > > > work. > > > > > > > > > > I understand that's the "normal" way of working. But whatever the > > > > > swiotlb > > > > > is used for in baremetal Linux, that would similarly require its use > > > > > in > > > > > PVH (or HVM) aiui. So unconditionally disabling it in PVH would look > > > > > to > > > > > me like an incomplete attempt to disable its use altogether on x86. > > > > > What > > > > > difference of PVH vs baremetal am I missing here? > > > > > > > > swiotlb is not usable for GPUs even on bare metal. They often have > > > > hundreds or megs or even gigs of memory mapped on the device at any > > > > given time. Also, AMD GPUs support 44-48 bit DMA masks (depending on > > > > the chip family). > > > > > > But the swiotlb isn't per device, but system global. > > > > Sure, but if the swiotlb is in use, then you can't really use the GPU. > > So you get to pick one. > > The swiotlb is used only for buffers which are not within the DMA mask of a > device (see dma_direct_map_page()). So an AMD GPU supporting a 44 bit DMA mask > won't use the swiotlb unless you have a buffer above guest physical address of > 16TB (so basically never). > > Disabling swiotlb in such a guest would OTOH mean, that a device with only > 32 bit DMA mask passed through to this guest couldn't work with buffers > above 4GB. > > I don't think this is acceptable. >From the Xen subsystem in Linux point of view, the only thing we need to do is to make sure *not* to enable swiotlb_xen (yes "swiotlb_xen", not the global swiotlb) on PVH because it is not needed anyway. I think we should leave the global "swiotlb" setting alone. The global swiotlb is not relevant to Xen anyway, and surely baremetal Linux has to have a way to deal with swiotlb/GPU incompatibilities. We just have to avoid making things worse on Xen, and for that we just need to avoid unconditionally enabling swiotlb-xen. If the Xen subsystem doesn't enable swiotlb_xen/swiotlb,
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On Sun, Mar 12, 2023 at 08:01:53PM +0800, Huang Rui wrote:
> Xen PVH is the paravirtualized mode and takes advantage of hardware
> virtualization support when possible. It will using the hardware IOMMU
> support instead of xen-swiotlb, so disable swiotlb if current domain is
> Xen PVH.
>
> Signed-off-by: Huang Rui
> ---
> arch/x86/kernel/pci-dma.c | 8 +++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
> index 30bbe4abb5d6..f5c73dd18f2a 100644
> --- a/arch/x86/kernel/pci-dma.c
> +++ b/arch/x86/kernel/pci-dma.c
> @@ -74,6 +74,12 @@ static inline void __init pci_swiotlb_detect(void)
> #ifdef CONFIG_SWIOTLB_XEN
> static void __init pci_xen_swiotlb_init(void)
> {
> + /* Xen PVH domain won't use swiotlb */
> + if (xen_pvh_domain()) {
> + x86_swiotlb_enable = false;
> + return;
> + }
I'm very confused by this: pci_xen_swiotlb_init() is only called for
PV domains, see the only caller in pci_iommu_alloc(). So this is just
dead code.
> +
> if (!xen_initial_domain() && !x86_swiotlb_enable)
> return;
> x86_swiotlb_enable = true;
> @@ -86,7 +92,7 @@ static void __init pci_xen_swiotlb_init(void)
>
> int pci_xen_swiotlb_init_late(void)
> {
> - if (dma_ops == _swiotlb_dma_ops)
> + if (xen_pvh_domain() || dma_ops == _swiotlb_dma_ops)
Same here, this function is only called by
pcifront_connect_and_init_dma() and pcifront should never attach on a
PVH domain, hence it's also dead code.
Thanks, Roger.
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On 16.03.23 14:53, Alex Deucher wrote: On Thu, Mar 16, 2023 at 9:48 AM Juergen Gross wrote: On 16.03.23 14:45, Alex Deucher wrote: On Thu, Mar 16, 2023 at 3:50 AM Jan Beulich wrote: On 16.03.2023 00:25, Stefano Stabellini wrote: On Wed, 15 Mar 2023, Jan Beulich wrote: On 15.03.2023 01:52, Stefano Stabellini wrote: On Mon, 13 Mar 2023, Jan Beulich wrote: On 12.03.2023 13:01, Huang Rui wrote: Xen PVH is the paravirtualized mode and takes advantage of hardware virtualization support when possible. It will using the hardware IOMMU support instead of xen-swiotlb, so disable swiotlb if current domain is Xen PVH. But the kernel has no way (yet) to drive the IOMMU, so how can it get away without resorting to swiotlb in certain cases (like I/O to an address-restricted device)? I think Ray meant that, thanks to the IOMMU setup by Xen, there is no need for swiotlb-xen in Dom0. Address translations are done by the IOMMU so we can use guest physical addresses instead of machine addresses for DMA. This is a similar case to Dom0 on ARM when the IOMMU is available (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding case is XENFEAT_not_direct_mapped). But how does Xen using an IOMMU help with, as said, address-restricted devices? They may still need e.g. a 32-bit address to be programmed in, and if the kernel has memory beyond the 4G boundary not all I/O buffers may fulfill this requirement. In short, it is going to work as long as Linux has guest physical addresses (not machine addresses, those could be anything) lower than 4GB. If the address-restricted device does DMA via an IOMMU, then the device gets programmed by Linux using its guest physical addresses (not machine addresses). The 32-bit restriction would be applied by Linux to its choice of guest physical address to use to program the device, the same way it does on native. The device would be fine as it always uses Linux-provided <4GB addresses. After the IOMMU translation (pagetable setup by Xen), we could get any address, including >4GB addresses, and that is expected to work. I understand that's the "normal" way of working. But whatever the swiotlb is used for in baremetal Linux, that would similarly require its use in PVH (or HVM) aiui. So unconditionally disabling it in PVH would look to me like an incomplete attempt to disable its use altogether on x86. What difference of PVH vs baremetal am I missing here? swiotlb is not usable for GPUs even on bare metal. They often have hundreds or megs or even gigs of memory mapped on the device at any given time. Also, AMD GPUs support 44-48 bit DMA masks (depending on the chip family). But the swiotlb isn't per device, but system global. Sure, but if the swiotlb is in use, then you can't really use the GPU. So you get to pick one. The swiotlb is used only for buffers which are not within the DMA mask of a device (see dma_direct_map_page()). So an AMD GPU supporting a 44 bit DMA mask won't use the swiotlb unless you have a buffer above guest physical address of 16TB (so basically never). Disabling swiotlb in such a guest would OTOH mean, that a device with only 32 bit DMA mask passed through to this guest couldn't work with buffers above 4GB. I don't think this is acceptable. Juergen OpenPGP_0xB0DE9DD628BF132F.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On 16.03.2023 14:53, Alex Deucher wrote: > On Thu, Mar 16, 2023 at 9:48 AM Juergen Gross wrote: >> >> On 16.03.23 14:45, Alex Deucher wrote: >>> On Thu, Mar 16, 2023 at 3:50 AM Jan Beulich wrote: On 16.03.2023 00:25, Stefano Stabellini wrote: > On Wed, 15 Mar 2023, Jan Beulich wrote: >> On 15.03.2023 01:52, Stefano Stabellini wrote: >>> On Mon, 13 Mar 2023, Jan Beulich wrote: On 12.03.2023 13:01, Huang Rui wrote: > Xen PVH is the paravirtualized mode and takes advantage of hardware > virtualization support when possible. It will using the hardware IOMMU > support instead of xen-swiotlb, so disable swiotlb if current domain > is > Xen PVH. But the kernel has no way (yet) to drive the IOMMU, so how can it get away without resorting to swiotlb in certain cases (like I/O to an address-restricted device)? >>> >>> I think Ray meant that, thanks to the IOMMU setup by Xen, there is no >>> need for swiotlb-xen in Dom0. Address translations are done by the IOMMU >>> so we can use guest physical addresses instead of machine addresses for >>> DMA. This is a similar case to Dom0 on ARM when the IOMMU is available >>> (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding >>> case is XENFEAT_not_direct_mapped). >> >> But how does Xen using an IOMMU help with, as said, address-restricted >> devices? They may still need e.g. a 32-bit address to be programmed in, >> and if the kernel has memory beyond the 4G boundary not all I/O buffers >> may fulfill this requirement. > > In short, it is going to work as long as Linux has guest physical > addresses (not machine addresses, those could be anything) lower than > 4GB. > > If the address-restricted device does DMA via an IOMMU, then the device > gets programmed by Linux using its guest physical addresses (not machine > addresses). > > The 32-bit restriction would be applied by Linux to its choice of guest > physical address to use to program the device, the same way it does on > native. The device would be fine as it always uses Linux-provided <4GB > addresses. After the IOMMU translation (pagetable setup by Xen), we > could get any address, including >4GB addresses, and that is expected to > work. I understand that's the "normal" way of working. But whatever the swiotlb is used for in baremetal Linux, that would similarly require its use in PVH (or HVM) aiui. So unconditionally disabling it in PVH would look to me like an incomplete attempt to disable its use altogether on x86. What difference of PVH vs baremetal am I missing here? >>> >>> swiotlb is not usable for GPUs even on bare metal. They often have >>> hundreds or megs or even gigs of memory mapped on the device at any >>> given time. Also, AMD GPUs support 44-48 bit DMA masks (depending on >>> the chip family). >> >> But the swiotlb isn't per device, but system global. > > Sure, but if the swiotlb is in use, then you can't really use the GPU. > So you get to pick one. Yet that "pick one" then can't be an unconditional disable in the source code. If there's no way to avoid swiotlb on a per-device basis, then users will need to be told to arrange for this via command line option when they want to use the GPU is certain ways. Jan
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On Thu, Mar 16, 2023 at 9:48 AM Juergen Gross wrote: > > On 16.03.23 14:45, Alex Deucher wrote: > > On Thu, Mar 16, 2023 at 3:50 AM Jan Beulich wrote: > >> > >> On 16.03.2023 00:25, Stefano Stabellini wrote: > >>> On Wed, 15 Mar 2023, Jan Beulich wrote: > On 15.03.2023 01:52, Stefano Stabellini wrote: > > On Mon, 13 Mar 2023, Jan Beulich wrote: > >> On 12.03.2023 13:01, Huang Rui wrote: > >>> Xen PVH is the paravirtualized mode and takes advantage of hardware > >>> virtualization support when possible. It will using the hardware IOMMU > >>> support instead of xen-swiotlb, so disable swiotlb if current domain > >>> is > >>> Xen PVH. > >> > >> But the kernel has no way (yet) to drive the IOMMU, so how can it get > >> away without resorting to swiotlb in certain cases (like I/O to an > >> address-restricted device)? > > > > I think Ray meant that, thanks to the IOMMU setup by Xen, there is no > > need for swiotlb-xen in Dom0. Address translations are done by the IOMMU > > so we can use guest physical addresses instead of machine addresses for > > DMA. This is a similar case to Dom0 on ARM when the IOMMU is available > > (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding > > case is XENFEAT_not_direct_mapped). > > But how does Xen using an IOMMU help with, as said, address-restricted > devices? They may still need e.g. a 32-bit address to be programmed in, > and if the kernel has memory beyond the 4G boundary not all I/O buffers > may fulfill this requirement. > >>> > >>> In short, it is going to work as long as Linux has guest physical > >>> addresses (not machine addresses, those could be anything) lower than > >>> 4GB. > >>> > >>> If the address-restricted device does DMA via an IOMMU, then the device > >>> gets programmed by Linux using its guest physical addresses (not machine > >>> addresses). > >>> > >>> The 32-bit restriction would be applied by Linux to its choice of guest > >>> physical address to use to program the device, the same way it does on > >>> native. The device would be fine as it always uses Linux-provided <4GB > >>> addresses. After the IOMMU translation (pagetable setup by Xen), we > >>> could get any address, including >4GB addresses, and that is expected to > >>> work. > >> > >> I understand that's the "normal" way of working. But whatever the swiotlb > >> is used for in baremetal Linux, that would similarly require its use in > >> PVH (or HVM) aiui. So unconditionally disabling it in PVH would look to > >> me like an incomplete attempt to disable its use altogether on x86. What > >> difference of PVH vs baremetal am I missing here? > > > > swiotlb is not usable for GPUs even on bare metal. They often have > > hundreds or megs or even gigs of memory mapped on the device at any > > given time. Also, AMD GPUs support 44-48 bit DMA masks (depending on > > the chip family). > > But the swiotlb isn't per device, but system global. Sure, but if the swiotlb is in use, then you can't really use the GPU. So you get to pick one. Alex
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On 16.03.23 14:45, Alex Deucher wrote: On Thu, Mar 16, 2023 at 3:50 AM Jan Beulich wrote: On 16.03.2023 00:25, Stefano Stabellini wrote: On Wed, 15 Mar 2023, Jan Beulich wrote: On 15.03.2023 01:52, Stefano Stabellini wrote: On Mon, 13 Mar 2023, Jan Beulich wrote: On 12.03.2023 13:01, Huang Rui wrote: Xen PVH is the paravirtualized mode and takes advantage of hardware virtualization support when possible. It will using the hardware IOMMU support instead of xen-swiotlb, so disable swiotlb if current domain is Xen PVH. But the kernel has no way (yet) to drive the IOMMU, so how can it get away without resorting to swiotlb in certain cases (like I/O to an address-restricted device)? I think Ray meant that, thanks to the IOMMU setup by Xen, there is no need for swiotlb-xen in Dom0. Address translations are done by the IOMMU so we can use guest physical addresses instead of machine addresses for DMA. This is a similar case to Dom0 on ARM when the IOMMU is available (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding case is XENFEAT_not_direct_mapped). But how does Xen using an IOMMU help with, as said, address-restricted devices? They may still need e.g. a 32-bit address to be programmed in, and if the kernel has memory beyond the 4G boundary not all I/O buffers may fulfill this requirement. In short, it is going to work as long as Linux has guest physical addresses (not machine addresses, those could be anything) lower than 4GB. If the address-restricted device does DMA via an IOMMU, then the device gets programmed by Linux using its guest physical addresses (not machine addresses). The 32-bit restriction would be applied by Linux to its choice of guest physical address to use to program the device, the same way it does on native. The device would be fine as it always uses Linux-provided <4GB addresses. After the IOMMU translation (pagetable setup by Xen), we could get any address, including >4GB addresses, and that is expected to work. I understand that's the "normal" way of working. But whatever the swiotlb is used for in baremetal Linux, that would similarly require its use in PVH (or HVM) aiui. So unconditionally disabling it in PVH would look to me like an incomplete attempt to disable its use altogether on x86. What difference of PVH vs baremetal am I missing here? swiotlb is not usable for GPUs even on bare metal. They often have hundreds or megs or even gigs of memory mapped on the device at any given time. Also, AMD GPUs support 44-48 bit DMA masks (depending on the chip family). But the swiotlb isn't per device, but system global. Juergen OpenPGP_0xB0DE9DD628BF132F.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On Thu, Mar 16, 2023 at 3:50 AM Jan Beulich wrote: > > On 16.03.2023 00:25, Stefano Stabellini wrote: > > On Wed, 15 Mar 2023, Jan Beulich wrote: > >> On 15.03.2023 01:52, Stefano Stabellini wrote: > >>> On Mon, 13 Mar 2023, Jan Beulich wrote: > On 12.03.2023 13:01, Huang Rui wrote: > > Xen PVH is the paravirtualized mode and takes advantage of hardware > > virtualization support when possible. It will using the hardware IOMMU > > support instead of xen-swiotlb, so disable swiotlb if current domain is > > Xen PVH. > > But the kernel has no way (yet) to drive the IOMMU, so how can it get > away without resorting to swiotlb in certain cases (like I/O to an > address-restricted device)? > >>> > >>> I think Ray meant that, thanks to the IOMMU setup by Xen, there is no > >>> need for swiotlb-xen in Dom0. Address translations are done by the IOMMU > >>> so we can use guest physical addresses instead of machine addresses for > >>> DMA. This is a similar case to Dom0 on ARM when the IOMMU is available > >>> (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding > >>> case is XENFEAT_not_direct_mapped). > >> > >> But how does Xen using an IOMMU help with, as said, address-restricted > >> devices? They may still need e.g. a 32-bit address to be programmed in, > >> and if the kernel has memory beyond the 4G boundary not all I/O buffers > >> may fulfill this requirement. > > > > In short, it is going to work as long as Linux has guest physical > > addresses (not machine addresses, those could be anything) lower than > > 4GB. > > > > If the address-restricted device does DMA via an IOMMU, then the device > > gets programmed by Linux using its guest physical addresses (not machine > > addresses). > > > > The 32-bit restriction would be applied by Linux to its choice of guest > > physical address to use to program the device, the same way it does on > > native. The device would be fine as it always uses Linux-provided <4GB > > addresses. After the IOMMU translation (pagetable setup by Xen), we > > could get any address, including >4GB addresses, and that is expected to > > work. > > I understand that's the "normal" way of working. But whatever the swiotlb > is used for in baremetal Linux, that would similarly require its use in > PVH (or HVM) aiui. So unconditionally disabling it in PVH would look to > me like an incomplete attempt to disable its use altogether on x86. What > difference of PVH vs baremetal am I missing here? swiotlb is not usable for GPUs even on bare metal. They often have hundreds or megs or even gigs of memory mapped on the device at any given time. Also, AMD GPUs support 44-48 bit DMA masks (depending on the chip family). Alex
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On 16.03.2023 00:25, Stefano Stabellini wrote: > On Wed, 15 Mar 2023, Jan Beulich wrote: >> On 15.03.2023 01:52, Stefano Stabellini wrote: >>> On Mon, 13 Mar 2023, Jan Beulich wrote: On 12.03.2023 13:01, Huang Rui wrote: > Xen PVH is the paravirtualized mode and takes advantage of hardware > virtualization support when possible. It will using the hardware IOMMU > support instead of xen-swiotlb, so disable swiotlb if current domain is > Xen PVH. But the kernel has no way (yet) to drive the IOMMU, so how can it get away without resorting to swiotlb in certain cases (like I/O to an address-restricted device)? >>> >>> I think Ray meant that, thanks to the IOMMU setup by Xen, there is no >>> need for swiotlb-xen in Dom0. Address translations are done by the IOMMU >>> so we can use guest physical addresses instead of machine addresses for >>> DMA. This is a similar case to Dom0 on ARM when the IOMMU is available >>> (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding >>> case is XENFEAT_not_direct_mapped). >> >> But how does Xen using an IOMMU help with, as said, address-restricted >> devices? They may still need e.g. a 32-bit address to be programmed in, >> and if the kernel has memory beyond the 4G boundary not all I/O buffers >> may fulfill this requirement. > > In short, it is going to work as long as Linux has guest physical > addresses (not machine addresses, those could be anything) lower than > 4GB. > > If the address-restricted device does DMA via an IOMMU, then the device > gets programmed by Linux using its guest physical addresses (not machine > addresses). > > The 32-bit restriction would be applied by Linux to its choice of guest > physical address to use to program the device, the same way it does on > native. The device would be fine as it always uses Linux-provided <4GB > addresses. After the IOMMU translation (pagetable setup by Xen), we > could get any address, including >4GB addresses, and that is expected to > work. I understand that's the "normal" way of working. But whatever the swiotlb is used for in baremetal Linux, that would similarly require its use in PVH (or HVM) aiui. So unconditionally disabling it in PVH would look to me like an incomplete attempt to disable its use altogether on x86. What difference of PVH vs baremetal am I missing here? Jan
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On Wed, 15 Mar 2023, Jan Beulich wrote: > On 15.03.2023 01:52, Stefano Stabellini wrote: > > On Mon, 13 Mar 2023, Jan Beulich wrote: > >> On 12.03.2023 13:01, Huang Rui wrote: > >>> Xen PVH is the paravirtualized mode and takes advantage of hardware > >>> virtualization support when possible. It will using the hardware IOMMU > >>> support instead of xen-swiotlb, so disable swiotlb if current domain is > >>> Xen PVH. > >> > >> But the kernel has no way (yet) to drive the IOMMU, so how can it get > >> away without resorting to swiotlb in certain cases (like I/O to an > >> address-restricted device)? > > > > I think Ray meant that, thanks to the IOMMU setup by Xen, there is no > > need for swiotlb-xen in Dom0. Address translations are done by the IOMMU > > so we can use guest physical addresses instead of machine addresses for > > DMA. This is a similar case to Dom0 on ARM when the IOMMU is available > > (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding > > case is XENFEAT_not_direct_mapped). > > But how does Xen using an IOMMU help with, as said, address-restricted > devices? They may still need e.g. a 32-bit address to be programmed in, > and if the kernel has memory beyond the 4G boundary not all I/O buffers > may fulfill this requirement. In short, it is going to work as long as Linux has guest physical addresses (not machine addresses, those could be anything) lower than 4GB. If the address-restricted device does DMA via an IOMMU, then the device gets programmed by Linux using its guest physical addresses (not machine addresses). The 32-bit restriction would be applied by Linux to its choice of guest physical address to use to program the device, the same way it does on native. The device would be fine as it always uses Linux-provided <4GB addresses. After the IOMMU translation (pagetable setup by Xen), we could get any address, including >4GB addresses, and that is expected to work.
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On 15.03.2023 01:52, Stefano Stabellini wrote: > On Mon, 13 Mar 2023, Jan Beulich wrote: >> On 12.03.2023 13:01, Huang Rui wrote: >>> Xen PVH is the paravirtualized mode and takes advantage of hardware >>> virtualization support when possible. It will using the hardware IOMMU >>> support instead of xen-swiotlb, so disable swiotlb if current domain is >>> Xen PVH. >> >> But the kernel has no way (yet) to drive the IOMMU, so how can it get >> away without resorting to swiotlb in certain cases (like I/O to an >> address-restricted device)? > > I think Ray meant that, thanks to the IOMMU setup by Xen, there is no > need for swiotlb-xen in Dom0. Address translations are done by the IOMMU > so we can use guest physical addresses instead of machine addresses for > DMA. This is a similar case to Dom0 on ARM when the IOMMU is available > (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding > case is XENFEAT_not_direct_mapped). But how does Xen using an IOMMU help with, as said, address-restricted devices? They may still need e.g. a 32-bit address to be programmed in, and if the kernel has memory beyond the 4G boundary not all I/O buffers may fulfill this requirement. Jan
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On 15.03.2023 05:14, Huang Rui wrote: > On Wed, Mar 15, 2023 at 08:52:30AM +0800, Stefano Stabellini wrote: >> On Mon, 13 Mar 2023, Jan Beulich wrote: >>> On 12.03.2023 13:01, Huang Rui wrote: Xen PVH is the paravirtualized mode and takes advantage of hardware virtualization support when possible. It will using the hardware IOMMU support instead of xen-swiotlb, so disable swiotlb if current domain is Xen PVH. >>> >>> But the kernel has no way (yet) to drive the IOMMU, so how can it get >>> away without resorting to swiotlb in certain cases (like I/O to an >>> address-restricted device)? >> >> I think Ray meant that, thanks to the IOMMU setup by Xen, there is no >> need for swiotlb-xen in Dom0. Address translations are done by the IOMMU >> so we can use guest physical addresses instead of machine addresses for >> DMA. This is a similar case to Dom0 on ARM when the IOMMU is available >> (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding >> case is XENFEAT_not_direct_mapped). > > Hi Jan, sorry to late reply. We are using the native kernel amdgpu and ttm > driver on Dom0, amdgpu/ttm would like to use IOMMU to allocate coherent > buffers for userptr that map the user space memory to gpu access, however, > swiotlb doesn't support this. In other words, with swiotlb, we only can > handle the buffer page by page. But how does outright disabling swiotlb help with this? There still wouldn't be an IOMMU that your kernel has control over. Looks like you want something like pvIOMMU, but that work was never completed. And even then the swiotlb may continue to be needed for other purposes. Jan
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On Wed, Mar 15, 2023 at 08:52:30AM +0800, Stefano Stabellini wrote: > On Mon, 13 Mar 2023, Jan Beulich wrote: > > On 12.03.2023 13:01, Huang Rui wrote: > > > Xen PVH is the paravirtualized mode and takes advantage of hardware > > > virtualization support when possible. It will using the hardware IOMMU > > > support instead of xen-swiotlb, so disable swiotlb if current domain is > > > Xen PVH. > > > > But the kernel has no way (yet) to drive the IOMMU, so how can it get > > away without resorting to swiotlb in certain cases (like I/O to an > > address-restricted device)? > > I think Ray meant that, thanks to the IOMMU setup by Xen, there is no > need for swiotlb-xen in Dom0. Address translations are done by the IOMMU > so we can use guest physical addresses instead of machine addresses for > DMA. This is a similar case to Dom0 on ARM when the IOMMU is available > (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding > case is XENFEAT_not_direct_mapped). Hi Jan, sorry to late reply. We are using the native kernel amdgpu and ttm driver on Dom0, amdgpu/ttm would like to use IOMMU to allocate coherent buffers for userptr that map the user space memory to gpu access, however, swiotlb doesn't support this. In other words, with swiotlb, we only can handle the buffer page by page. Thanks, Ray > > Jurgen, what do you think? Would you rather make xen_swiotlb_detect > common between ARM and x86?
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On Mon, 13 Mar 2023, Jan Beulich wrote: > On 12.03.2023 13:01, Huang Rui wrote: > > Xen PVH is the paravirtualized mode and takes advantage of hardware > > virtualization support when possible. It will using the hardware IOMMU > > support instead of xen-swiotlb, so disable swiotlb if current domain is > > Xen PVH. > > But the kernel has no way (yet) to drive the IOMMU, so how can it get > away without resorting to swiotlb in certain cases (like I/O to an > address-restricted device)? I think Ray meant that, thanks to the IOMMU setup by Xen, there is no need for swiotlb-xen in Dom0. Address translations are done by the IOMMU so we can use guest physical addresses instead of machine addresses for DMA. This is a similar case to Dom0 on ARM when the IOMMU is available (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding case is XENFEAT_not_direct_mapped). Jurgen, what do you think? Would you rather make xen_swiotlb_detect common between ARM and x86?
Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
On 12.03.2023 13:01, Huang Rui wrote: > Xen PVH is the paravirtualized mode and takes advantage of hardware > virtualization support when possible. It will using the hardware IOMMU > support instead of xen-swiotlb, so disable swiotlb if current domain is > Xen PVH. But the kernel has no way (yet) to drive the IOMMU, so how can it get away without resorting to swiotlb in certain cases (like I/O to an address-restricted device)? Jan
