Hello guys, Now I am trying to send an interrupt from the root-cell (Linux) to the bare-metal cell. The other way around, from the bare-metal to Linux, is working. Whenever I start the bare-metal cell, I can see that an interrupt has arrived by checking the /proc/interrupts (number 38 is the uio_ivshmem in my case).
The problem is that when I try to issue an interrupt from the root-cell, the interrupt counter is no incremented and the interrupt does not arrive in the bare-metal cell. I used the uio_send and I program that I implemented, which maps the pci region (0xfc100000) using mmap and write 1 to the doorbell. However, in both cases I can't see an interrupt in the /proc/interrupts. Looking at the ivshmem documentation (https://github.com/qemu/qemu/blob/master/docs/specs/ivshmem-spec.txt), the doorbell is divided in two spaces: "The written value's high 16 bits are the ID of the peer to interrupt, and its low 16 bits select an interrupt vector." How can I find the bare-metal peer's ID? What should be the interrupt vector value? Best, Giovani > I found the error. It was missing a config parameter in the non-root cell: > > .num_pci_devices = 2 (in the .cell struct) > > After inserting this, I got the correct output: > > Shared memory connection established: "gic-demo-ivshmem" <--> "ZynqMP-ZCU102" > > Thanks for the help. > > > Yes, this is strange and it is not happening. > > > > I added some prints in the pci_cell_init function (hypervisor/pci.c) and > > ivhsmem_init function (hypervisor/arm-common/ivhsmem.c). When I enable the > > root cell, I got the following: > > > > pci_cell_init - cell ZynqMP-ZCU102, id = 0, num_pci_devices 2 > > pci_cell_init - pci type = 3 > > Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102" Cell Id = 0 > > %%%%%% ivshmem_init() iv->bdf=0 > > pci_cell_init - pci type = 3 > > Adding virtual PCI device 00:00.1 to cell "ZynqMP-ZCU102" Cell Id = 0 > > iv->bdf = 0, dev_info->bdf = 1, id = 1 > > , shmem_region = 3, dev_info->shmem_region = 4 > > ,shmem_protocol = 0, dev protocol = 0 > > , peer_mem->phys_start = 34363932672, mem->phys_start = 34364981248 > > , peer_mem->size = 1048576, mem->size = 1048576 > > , cell->config->name = ZynqMP-ZCU102, peer config name = ZynqMP-ZCU102 > > %%%%%% ivshmem_init() iv->bdf=1 > > > > When I create the bare metal cell, I got the following output: > > > > # jailhouse cell create zynqmp-zcu102-gic-demo-ivshmem.cell > > Created cell "gic-demo-ivshmem" > > Page pool usage after cell creation: mem 56/995, remap 69/131072 > > > > It seems that pci_cell_init() is not being called for the non-root cell. > > > > > Hi, > > > at a first look the configuration looks ok for me. > > > > > > Il giorno giovedì 12 aprile 2018 19:49:09 UTC+2, Giovani Gracioli ha > > > scritto: > > > > Hello, > > > > > > > > Thank you for the answers. > > > > > > > > My comments are below. > > > > > > > > 1) The root config is defined as follows (copied only the relevant > > > > parts): > > > > > > > > .platform_info = { > > > > .pci_mmconfig_base = 0xfc000000, > > > > .pci_mmconfig_end_bus = 0, > > > > .pci_is_virtual = 1, > > > > .arm = { > > > > .gic_version = 2, > > > > .gicd_base = 0xf9010000, > > > > .gicc_base = 0xf902f000, > > > > .gich_base = 0xf9040000, > > > > .gicv_base = 0xf906f000, > > > > .maintenance_irq = 25, > > > > }, > > > > }, > > > > > > > > .mem_regions = { > > > > /* MMIO (permissive) */ { > > > > .phys_start = 0xfd000000, > > > > .virt_start = 0xfd000000, > > > > .size = 0x03000000, > > > > .flags = JAILHOUSE_MEM_READ | > > > > JAILHOUSE_MEM_WRITE | > > > > JAILHOUSE_MEM_IO, > > > > }, > > > > /* RAM */ { > > > > .phys_start = 0x0, > > > > .virt_start = 0x0, > > > > .size = 0x80000000, > > > > .flags = JAILHOUSE_MEM_READ | > > > > JAILHOUSE_MEM_WRITE | > > > > JAILHOUSE_MEM_EXECUTE, > > > > }, > > > > /* RAM */ { > > > > .phys_start = 0x800600000, > > > > .virt_start = 0x800600000, > > > > .size = 0x7fa00000, > > > > .flags = JAILHOUSE_MEM_READ | > > > > JAILHOUSE_MEM_WRITE | > > > > JAILHOUSE_MEM_EXECUTE, > > > > }, > > > > /* IVSHMEM shared memory region for 00:00.0 */ { > > > > .phys_start = 0x800400000, > > > > .virt_start = 0x800400000, > > > > .size = 0x100000, > > > > .flags = JAILHOUSE_MEM_READ | > > > > JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_ROOTSHARED, > > > > }, > > > > /* IVSHMEM shared memory region for 00:01.0 */ { > > > > .phys_start = 0x800500000, > > > > .virt_start = 0x800500000, > > > > .size = 0x100000, > > > > .flags = JAILHOUSE_MEM_READ | > > > > JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_ROOTSHARED, > > > > }, > > > > }, > > > > > > > > > > > > .pci_devices = { > > > > /* 00:00.0 */ { > > > > .type = JAILHOUSE_PCI_TYPE_IVSHMEM, > > > > .bdf = 0 << 3, > > > > .bar_mask = { > > > > 0xffffff00, 0xffffffff, 0x00000000, > > > > 0x00000000, 0x00000000, 0x00000000, > > > > }, > > > > .shmem_region = 3, > > > > .shmem_protocol = > > > > JAILHOUSE_SHMEM_PROTO_UNDEFINED, > > > > .num_msix_vectors = 0, > > > > .domain = 0x0000, > > > > }, > > > > /* 00:01.0 */ { > > > > .type = JAILHOUSE_PCI_TYPE_IVSHMEM, > > > > .bdf = 1 << 3, > > > > .bar_mask = { > > > > 0xffffff00, 0xffffffff, 0x00000000, > > > > 0x00000000, 0x00000000, 0x00000000, > > > > }, > > > > .shmem_region = 4, > > > > .shmem_protocol = > > > > JAILHOUSE_SHMEM_PROTO_UNDEFINED, > > > > .num_msix_vectors = 0, > > > > .domain = 0x0000, > > > > }, > > > > }, > > > > > > > > For the other inmate cell, the config is defined as follows: > > > > > > > > .mem_regions = { > > > > /* UART */ { > > > > .phys_start = 0xff010000, > > > > .virt_start = 0xff010000, > > > > .size = 0x1000, > > > > .flags = JAILHOUSE_MEM_READ | > > > > JAILHOUSE_MEM_WRITE | > > > > JAILHOUSE_MEM_IO | > > > > JAILHOUSE_MEM_ROOTSHARED, > > > > }, > > > > /* RAM */ { > > > > .phys_start = 0x800600000, > > > > .virt_start = 0, > > > > .size = 0x00010000, > > > > .flags = JAILHOUSE_MEM_READ | > > > > JAILHOUSE_MEM_WRITE | > > > > JAILHOUSE_MEM_EXECUTE | > > > > JAILHOUSE_MEM_LOADABLE, > > > > }, > > > > /* IVSHMEM shared memory region for 00:00.0 */ { > > > > .phys_start = 0x800400000, > > > > .virt_start = 0x800400000, > > > > .size = 0x100000, > > > > .flags = JAILHOUSE_MEM_READ | > > > > JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_ROOTSHARED, > > > > }, > > > > }, > > > > > > > > .pci_devices = { > > > > /* 00:00.0 */ { > > > > .type = JAILHOUSE_PCI_TYPE_IVSHMEM, > > > > .bdf = 0 << 3, > > > > .bar_mask = { > > > > 0xffffff00, 0xffffffff, 0x00000000, > > > > 0x00000000, 0x00000000, 0x00000000, > > > > }, > > > > .shmem_region = 2, > > > > .shmem_protocol = > > > > JAILHOUSE_SHMEM_PROTO_UNDEFINED, > > > > .num_msix_vectors = 0, > > > > .domain = 0x0000, > > > > }, > > > > }, > > > > > > > > When I load the uio_ivshmem driver after enabling the root cell, I got > > > > the output from the driver which says "using jailhouse mode" (note that > > > > I added some prints): > > > > > > > > OF: PCI: host bridge /vpci@0 ranges: > > > > OF: PCI: MEM 0xfc100000..0xfc103fff -> 0xfc100000 > > > > pci-host-generic fc000000.vpci: ECAM at [mem 0xfc000000-0xfc0f] > > > > pci-host-generic fc000000.vpci: PCI host bridge to bus 0000:00 > > > > pci_bus 0000:00: root bus resource [bus 00] > > > > pci_bus 0000:00: root bus resource [mem 0xfc100000-0xfc103fff] > > > > pci 0000:00:00.0: [1af4:1110] type 00 class 0xff0000 > > > > pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000000ff 64bit] > > > > pci 0000:00:01.0: [1af4:1110] type 00 class 0xff0000 > > > > pci 0000:00:01.0: reg 0x10: [mem 0x00000000-0x000000ff 64bit] > > > > pci 0000:00:00.0: BAR 0: assigned [mem 0xfc100000-0xfc1000ff 6] > > > > pci 0000:00:01.0: BAR 0: assigned [mem 0xfc100100-0xfc1001ff 6] > > > > **********ivshmem_pci_probe() > > > > uio_ivshmem 0000:00:00.0: enabling device (0000 -> 0002) > > > > uio_ivshmem 0000:00:00.0: using jailhouse mode > > > > **********ivshmem_pci_probe() mem[1] addr = 0000000800400000, 6 > > > > **********ivshmem_pci_probe() > > > > uio_ivshmem 0000:00:01.0: enabling device (0000 -> 0002) > > > > uio_ivshmem 0000:00:01.0: using jailhouse mode > > > > **********ivshmem_pci_probe() mem[1] addr = 0000000800500000, 6 > > > > > > > > The two vPCIs regions are mapped and appear in the lspci command: > > > > > > > > 00:00.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory > > > > Subsystem: Red Hat, Inc Inter-VM shared memory > > > > Flags: bus master, fast devsel, latency 0, IRQ 38 > > > > Memory at fc100000 (64-bit, non-prefetchable) [size=256] > > > > Kernel driver in use: uio_ivshmem > > > > > > > > 00:01.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory > > > > Subsystem: Red Hat, Inc Inter-VM shared memory > > > > Flags: bus master, fast devsel, latency 0, IRQ 39 > > > > Memory at fc100100 (64-bit, non-prefetchable) [size=256] > > > > Kernel driver in use: uio_ivshmem > > > > > > > > Also, I can see the addresses in the /proc/iomem: > > > > > > > > fc000000-fc0fffff : PCI ECAM > > > > fc100000-fc103fff : //vpci@0 > > > > fc100000-fc1000ff : 0000:00:00.0 > > > > fc100000-fc1000ff : ivshmem > > > > fc100100-fc1001ff : 0000:00:01.0 > > > > fc100100-fc1001ff : ivshmem > > > > > > > > However, I don't see any output (regarding the PCI) when I load the > > > > non-root inmate cell. > > > > > > > > > This is quite strange. Loading the inmate cell, you should see something > > > like this in the hypervisor's output: > > > > > > Adding virtual PCI device xx:xx.x to cell "your-inmate-cell-name" > > > Shared memory connection established: "inmate-cell-name" <--> > > > "your-root-cell-name" > > > > > > The check for binding is done on the start address and size of shared > > > regions and shmem protocol > > > > > > > > > > > 2) I use #define PCI_CFG_BASE 0xfc000000 in > > > > inmates/lib/arm-common/pci.c, which matches the .pci_mmconfig_base = > > > > 0xfc000000 in root cell config. > > > > > > > > 3) The output from the /sys/class/uio/uio0/maps[0,1] is: > > > > > > > > cat /sys/class/uio/uio0/maps/map0/addr > > > > 0x00000000fc100000 > > > > cat /sys/class/uio/uio0/maps/map0/offset > > > > 0x0 > > > > cat /sys/class/uio/uio0/maps/map0/size > > > > 0x0000000000001000 > > > > cat /sys/class/uio/uio0/maps/map0/name > > > > registers > > > > > > > > cat /sys/class/uio/uio0/maps/map1/addr > > > > > > > > 0x0000000800400000 > > > > cat /sys/class/uio/uio0/maps/map1/offset > > > > 0x0 > > > > cat /sys/class/uio/uio0/maps/map1/size > > > > 0x0000000000100000 > > > > cat /sys/class/uio/uio0/maps/map1/name > > > > shmem > > > > > > > > Sorry for the long email, but I guess with this detailed info it may > > > > help finding the error. > > > > > > > > > Il giorno giovedì 12 aprile 2018 08:51:39 UTC+2, Claudio Scordino ha > > > > > scritto: > > > > > > Hi Giovani, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2018-04-12 8:01 GMT+02:00 Jan Kiszka <[email protected]>: > > > > > > On 2018-04-11 19:40, Giovani Gracioli wrote: > > > > > > > > > > > > > Here is the output of the unhandled data read: > > > > > > > > > > > > > > > > > > > > > > > > > > Unhandled data read at 0xfc100000(2) > > > > > > > > > > > > > > > > > > > > > > > > > > FATAL: unhandled trap (exception class 0x24) > > > > > > > > > > > > > Cell state before exception: > > > > > > > > > > > > > pc: 0000000000001828 lr: 00000000000015f0 spsr: 60000005 > > > > > > >EL1 > > > > > > > > > > > > > sp: 0000000000003f30 esr: 24 1 1400006 > > > > > > > > > > > > > x0: 00000000fc100000 x1: 0000000000000000 x2: > > > > > > >0000000000000002 > > > > > > > > > > > > > x3: 00000000fc000000 x4: 0000000000000000 x5: > > > > > > >0000000000000000 > > > > > > > > > > > > > x6: 0000000000001000 x7: 0000000000000000 x8: > > > > > > >0000000000000000 > > > > > > > > > > > > > x9: 0000000000000000 x10: 0000000000000000 x11: > > > > > > >0000000000000000 > > > > > > > > > > > > > x12: 0000000000000000 x13: 0000000000000000 x14: > > > > > > > 0000000000000000 > > > > > > > > > > > > > x15: 0000000000000000 x16: 0000000000000000 x17: > > > > > > > 0000000000000000 > > > > > > > > > > > > > x18: 0000000000000000 x19: 0000000000001000 x20: > > > > > > > 000000000000ffff > > > > > > > > > > > > > x21: 0000000000001af4 x22: 0000000000001110 x23: > > > > > > > 0000000000001000 > > > > > > > > > > > > > x24: 0000000000002660 x25: 0000000000000000 x26: > > > > > > > 0000000000000000 > > > > > > > > > > > > > x27: 0000000000000000 x28: 0000000000000000 x29: > > > > > > > 0000000000000000 > > > > > > > > > > > > > > > > > > > > > > > > > > Parking CPU 3 (Cell: "gic-demo-ivshmem") > > > > > > > > > > > > > > > > > > > > > > > > > > Am I missing something in the configuration? > > > > > > > > > > > > > > > > > > > > > > > > Possibly. As this is code of Claudio and his colleagues, he may > > > > > > answer > > > > > > > > > > > > this better. > > > > > > > > > > > > > > > > > > > > > > > > A collegue of mine is looking at your issue. > > > > > > > > > > > > > > > > > > BTW, we plan to have a version of the PCI stuff rebased upstream in > > > > > > a couple of weeks. > > > > > > > > > > > > We'll post it on the ML once ready. > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > Claudio > > > > > > > > > > Hi all, > > > > > > > > > > i'm trying to understand what's wrong in the configuration. > > > > > 1. The PCI configuration of the root cell (and its relative shared > > > > > memory region ) needs a counterpart in the inmate configuration. The > > > > > two PCI must have the same bdf. In fact when the configuration is ok, > > > > > jailhouse prints on the screen che binding between the PCI devices. > > > > > > > > > > 2. The starting search address of the inmate library (#define > > > > > PCI_CFG_BASE in inmates/lib/arm-common/pci.c) must be the same of the > > > > > platform_info section in the root cell configuration ( > > > > > .platform_info = { > > > > > .pci_mmconfig_base = 0x48000000, > > > > > .pci_mmconfig_end_bus = 0x0, > > > > > .pci_is_virtual = 1, > > > > > ..... > > > > > ) > > > > > > > > > > 3. The best way to access the shared memory region from linux is > > > > > using the uio_ivshmem driver. It works correctly on ARM64. It just > > > > > needs the UIO kernel module as dependency. Once loaded, you can see > > > > > memory infos in /sys/class/uio/uio0/maps[0,1] which are shmem > > > > > informations and bar information. > > > > > > > > Regards, > > > --Luca -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
