On Sat, 30 Jan 2021, Julien Grall wrote:
> > > On 27/01/2021 11:47, Jukka Kaartinen wrote:
> > > >
> > > >
> > > > On Tue, Jan 26, 2021 at 10:22 PM Stefano Stabellini
> > > > <sstabell...@kernel.org <mailto:sstabell...@kernel.org>> wrote:
> > > >
> > > > On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
> > > > > On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini
> > > > <sstabell...@kernel.org <mailto:sstabell...@kernel.org>> wrote:
> > > > > On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
> > > > > > Thanks for the response!
> > > > > >
> > > > > > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini
> > > > <sstabell...@kernel.org <mailto:sstabell...@kernel.org>> wrote:
> > > > > > + xen-devel, Roman,
> > > > > >
> > > > > >
> > > > > > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > > > > > > Hi Stefano,
> > > > > > > I'm Jukka Kaartinen a SW developer working on
> > > > enabling hypervisors on mobile platforms. One of our HW that we use
> > > > on
> > > > > > development is
> > > > > > > Raspberry Pi 4B. I wonder if you could help me a
> > > > bit :).
> > > > > > >
> > > > > > > I'm trying to enable the GPU with Xen + Raspberry
> > > > Pi for
> > > > > > dom0.
> > > > https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> > > > <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605>
> > > >
> > > > > > >
> > > > > > > I got so far that GPU drivers are loaded (v3d &
> > > > vc4) without errors. But now Xen returns error when X is starting:
> > > > > > > (XEN) traps.c:1986:d0v1 HSR=0x93880045
> > > > pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> > > > > > > I tried to debug what causes this and looks
> > > > like find_mmio_handler cannot find handler.
> > > > > > > (See more here:
> > > > https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
> > > >
> > > > <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691>
> > > >
> > > > )
> > > > > > >
> > > > > > > Any ideas why the handler is not found?
> > > > > >
> > > > > >
> > > > > > Hi Jukka,
> > > > > >
> > > > > > I am glad to hear that you are interested in Xen on
> > > > RaspberryPi :-) I
> > > > > > haven't tried the GPU yet, I have been using the
> > > > serial only.
> > > > > > Roman, did you ever get the GPU working?
> > > > > >
> > > > > >
> > > > > > The error is a data abort error: Linux is trying to
> > > > access an address
> > > > > > which is not mapped to dom0. The address seems to
> > > > be 0x401315d000. It is
> > > > > > a pretty high address; I looked in device tree but
> > > > couldn't spot it.
> > > > > >
> > > > > > >From the HSR (the syndrom register) it looks like
> > > > it is a translation
> > > > > > fault at EL1 on stage1. As if the Linux address
> > > > mapping was wrong.
> > > > > > Anyone has any ideas how this could happen? Maybe a
> > > > reserved-memory
> > > > > > misconfiguration?
> > > > > >
> > > > > > I had issues with loading the driver in the first place.
> > > > Apparently swiotlb is used, maybe it can cause this. I also tried to
> > > > > enable CMA.
> > > > > > config.txt:
> > > > > > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
> > > > > > gpu_mem=128
> > > > >
> > > > > Also looking at your other reply and the implementation of
> > > > > vc4_bo_create, it looks like this is a CMA problem.
> > > > >
> > > > > It would be good to run a test with the swiotlb-xen
> > > > disabled:
> > > > >
> > > > > diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
> > > > > index 467fa225c3d0..2bdd12785d14 100644
> > > > > --- a/arch/arm/xen/mm.c
> > > > > +++ b/arch/arm/xen/mm.c
> > > > > @@ -138,8 +138,7 @@ void
> > > > xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int
> > > > order)
> > > > > static int __init xen_mm_init(void)
> > > > > {
> > > > > struct gnttab_cache_flush cflush;
> > > > > - if (!xen_initial_domain())
> > > > > - return 0;
> > > > > + return 0;
> > > > > xen_swiotlb_init(1, false);
> > > > >
> > > > > cflush.op = 0;
> > > > >
> > > > > With this change the kernel is not booting up. (btw. I'm using
> > > > USB SSD for my OS.)
> > > > > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> > > > > [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
> > > > > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
> > > > > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > > > > [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or
> > > > MSI/MSI-X might fail!
> > > > > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > > > > [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or
> > > > MSI/MSI-X might fail!
> > > > > [ 1.212820] usb 1-1: device descriptor read/64, error 18
> > > > > [ 1.452815] usb 1-1: device descriptor read/64, error 18
> > > > > [ 1.820813] usb 1-1: device descriptor read/64, error 18
> > > > > [ 2.060815] usb 1-1: device descriptor read/64, error 18
> > > > > [ 2.845548] usb 1-1: device descriptor read/8, error -61
> > > > > [ 2.977603] usb 1-1: device descriptor read/8, error -61
> > > > > [ 3.237530] usb 1-1: device descriptor read/8, error -61
> > > > > [ 3.369585] usb 1-1: device descriptor read/8, error -61
> > > > > [ 3.480765] usb usb1-port1: unable to enumerate USB device
> > > > >
> > > > > Traces stop here. I could try with a memory card. Maybe it makes
> > > > a difference.
> > > >
> > > > This is very surprising. Disabling swiotlb-xen should make things
> > > > better
> > > > not worse. The only reason I can think of why it could make things
> > > > worse
> > > > is if Linux runs out of low memory. Julien's patch
> > > > 437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have
> > > > addressed
> > > > that issue though. Julien, any ideas?
> > >
> > > I think, Stefano's small patch is not enough to disable the swiotlb as we
> > > will still override the DMA ops. You also likely want:
> > >
> > > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> > > index 8a8949174b1c..aa43e249ecdd 100644
> > > --- a/arch/arm/mm/dma-mapping.c
> > > +++ b/arch/arm/mm/dma-mapping.c
> > > @@ -2279,7 +2279,7 @@ void arch_setup_dma_ops(struct device *dev, u64
> > > dma_base, u64 size,
> > > set_dma_ops(dev, dma_ops);
> > >
> > > #ifdef CONFIG_XEN
> > > - if (xen_initial_domain())
> > > + if (0 || xen_initial_domain())
> > > dev->dma_ops = &xen_swiotlb_dma_ops;
> > > #endif
> > > dev->archdata.dma_ops_setup = true;
> > >
> > > Otherwise, you would still use the swiotlb DMA ops that would not be
> > > functional as we disabled the swiotlb.
> > >
> > > This would explain the following error because it will check whether the
> > > mask is valid using the callback dma_supported():
> > >
> > > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> > >
> > Good catch.
> > GPU works now and I can start X! Thanks! I was also able to create domU that
> > runs Raspian OS.
>
> Glad to hear it works! IIRC, the swiotlb may become necessary when running
> guest if the guest memory ends up to be used for DMA transaction.
It is necessary if you are using PV network or PV disk: memory shared by
another domU could end up being used in a DMA transaction. For that, you
need swiotlb-xen.
> > Now that swiotlb is disabled what does it mean?
>
> I can see two reasons:
> 1) You have limited memory below the 30 bits mark. So Swiotlb and CMA may
> try to fight for the low memory.
> 2) We found a few conversion bugs in the swiotlb on RPI4 last year (IIRC the
> DMA and physical address may be different). I looked at the Linux branch you
> are using and they seem to all be there. So there might be another bug.
>
> I am not sure how to figure out where is the problem. Stefano, do you have a
> suggestion where to start?
Both 1) and 2) are possible. It is also possible that another driver,
probably something related to CMA or DRM, has some special dma_ops
handling that doesn't work well together with swiotlb-xen.
Given that the original error seemed to be related to vc4_bo_create,
which calls dma_alloc_wc, I would add a couple of printks to
xen_swiotlb_alloc_coherent to help us figure it out:
diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 2b385c1b4a99..cac8b09af603 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -295,6 +295,7 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t
size,
/* Convert the size to actually allocated. */
size = 1UL << (order + XEN_PAGE_SHIFT);
+ printk("DEBUG %s %d size=%lu flags=%lx
attr=%lx\n",__func__,__LINE__,size,flags,attrs);
/* On ARM this function returns an ioremap'ped virtual address for
* which virt_to_phys doesn't return the corresponding physical
* address. In fact on ARM virt_to_phys only works for kernel direct
@@ -315,16 +316,20 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t
size,
phys = dma_to_phys(hwdev, *dma_handle);
dev_addr = xen_phys_to_dma(hwdev, phys);
if (((dev_addr + size - 1 <= dma_mask)) &&
- !range_straddles_page_boundary(phys, size))
+ !range_straddles_page_boundary(phys, size)) {
*dma_handle = dev_addr;
- else {
+ printk("DEBUG %s %d phys=%lx
dma=%lx\n",__func__,__LINE__,phys,dev_addr);
+ } else {
if (xen_create_contiguous_region(phys, order,
fls64(dma_mask), dma_handle)
!= 0) {
+ printk("DEBUG %s %d\n",__func__,__LINE__);
xen_free_coherent_pages(hwdev, size, ret,
(dma_addr_t)phys, attrs);
return NULL;
}
*dma_handle = phys_to_dma(hwdev, *dma_handle);
SetPageXenRemapped(virt_to_page(ret));
+ printk("DEBUG %s %d dma_mask=%d page_boundary=%d phys=%lx
dma=%lx\n",__func__,__LINE__,
+ ((dev_addr + size - 1 <=
dma_mask)),range_straddles_page_boundary(phys, size),phys,*dma_handle);
}
memset(ret, 0, size);
return ret;
> > And also can I pass the GPU to domU? Raspberry Pi 4 is limited HW and
> > doesn't have IOMMU. I'm trying to create similar OS like QubesOS where GPU,
> > Network, keyboard/mouse, ... are isolated to their own VMs.
>
> Without an IOMMU or any other HW mechamisns (e.g. MPU), it would not be safe
> to assign a DMA-capable device to a non-trusted VM.
>
> If you trust the VM where you assigned a device, then a possible approach
> would be to have the VM direct mapped (e.g. guest physical address == host
> physical address). Although, I can foreese some issues if you have multiple
> VMs requires memory below 30 bits (there seem to be limited amount)>
>
> If you don't trust the VM where you assigned a device, then your best option
> will be to expose a PV interface of the device and have your backend
> sanitizing the request and issuing it on behalf of the guest.
FYI you could do that with the existing PVFB drivers that only support 2D
graphics.
