Yes, this is strange and it is not happening.
I added some prints in the pci_cell_init function (hypervisor/pci.c) and
ivhsmem_init function (hypervisor/arm-common/ivhsmem.c). When I enable the root
cell, I got the following:
pci_cell_init - cell ZynqMP-ZCU102, id = 0, num_pci_devices 2
pci_cell_init - pci type = 3
Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102" Cell Id = 0
%%%%%% ivshmem_init() iv->bdf=0
pci_cell_init - pci type = 3
Adding virtual PCI device 00:00.1 to cell "ZynqMP-ZCU102" Cell Id = 0
iv->bdf = 0, dev_info->bdf = 1, id = 1
, shmem_region = 3, dev_info->shmem_region = 4
,shmem_protocol = 0, dev protocol = 0
, peer_mem->phys_start = 34363932672, mem->phys_start = 34364981248
, peer_mem->size = 1048576, mem->size = 1048576
, cell->config->name = ZynqMP-ZCU102, peer config name = ZynqMP-ZCU102
%%%%%% ivshmem_init() iv->bdf=1
When I create the bare metal cell, I got the following output:
# jailhouse cell create zynqmp-zcu102-gic-demo-ivshmem.cell
Created cell "gic-demo-ivshmem"
Page pool usage after cell creation: mem 56/995, remap 69/131072
It seems that pci_cell_init() is not being called for the non-root cell.
> Hi,
> at a first look the configuration looks ok for me.
>
> Il giorno giovedì 12 aprile 2018 19:49:09 UTC+2, Giovani Gracioli ha scritto:
> > Hello,
> >
> > Thank you for the answers.
> >
> > My comments are below.
> >
> > 1) The root config is defined as follows (copied only the relevant parts):
> >
> > .platform_info = {
> > .pci_mmconfig_base = 0xfc000000,
> > .pci_mmconfig_end_bus = 0,
> > .pci_is_virtual = 1,
> > .arm = {
> > .gic_version = 2,
> > .gicd_base = 0xf9010000,
> > .gicc_base = 0xf902f000,
> > .gich_base = 0xf9040000,
> > .gicv_base = 0xf906f000,
> > .maintenance_irq = 25,
> > },
> > },
> >
> > .mem_regions = {
> > /* MMIO (permissive) */ {
> > .phys_start = 0xfd000000,
> > .virt_start = 0xfd000000,
> > .size = 0x03000000,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > JAILHOUSE_MEM_IO,
> > },
> > /* RAM */ {
> > .phys_start = 0x0,
> > .virt_start = 0x0,
> > .size = 0x80000000,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > JAILHOUSE_MEM_EXECUTE,
> > },
> > /* RAM */ {
> > .phys_start = 0x800600000,
> > .virt_start = 0x800600000,
> > .size = 0x7fa00000,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > JAILHOUSE_MEM_EXECUTE,
> > },
> > /* IVSHMEM shared memory region for 00:00.0 */ {
> > .phys_start = 0x800400000,
> > .virt_start = 0x800400000,
> > .size = 0x100000,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > JAILHOUSE_MEM_ROOTSHARED,
> > },
> > /* IVSHMEM shared memory region for 00:01.0 */ {
> > .phys_start = 0x800500000,
> > .virt_start = 0x800500000,
> > .size = 0x100000,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > JAILHOUSE_MEM_ROOTSHARED,
> > },
> > },
> >
> >
> > .pci_devices = {
> > /* 00:00.0 */ {
> > .type = JAILHOUSE_PCI_TYPE_IVSHMEM,
> > .bdf = 0 << 3,
> > .bar_mask = {
> > 0xffffff00, 0xffffffff, 0x00000000,
> > 0x00000000, 0x00000000, 0x00000000,
> > },
> > .shmem_region = 3,
> > .shmem_protocol = JAILHOUSE_SHMEM_PROTO_UNDEFINED,
> > .num_msix_vectors = 0,
> > .domain = 0x0000,
> > },
> > /* 00:01.0 */ {
> > .type = JAILHOUSE_PCI_TYPE_IVSHMEM,
> > .bdf = 1 << 3,
> > .bar_mask = {
> > 0xffffff00, 0xffffffff, 0x00000000,
> > 0x00000000, 0x00000000, 0x00000000,
> > },
> > .shmem_region = 4,
> > .shmem_protocol = JAILHOUSE_SHMEM_PROTO_UNDEFINED,
> > .num_msix_vectors = 0,
> > .domain = 0x0000,
> > },
> > },
> >
> > For the other inmate cell, the config is defined as follows:
> >
> > .mem_regions = {
> > /* UART */ {
> > .phys_start = 0xff010000,
> > .virt_start = 0xff010000,
> > .size = 0x1000,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED,
> > },
> > /* RAM */ {
> > .phys_start = 0x800600000,
> > .virt_start = 0,
> > .size = 0x00010000,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > JAILHOUSE_MEM_EXECUTE |
> > JAILHOUSE_MEM_LOADABLE,
> > },
> > /* IVSHMEM shared memory region for 00:00.0 */ {
> > .phys_start = 0x800400000,
> > .virt_start = 0x800400000,
> > .size = 0x100000,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > JAILHOUSE_MEM_ROOTSHARED,
> > },
> > },
> >
> > .pci_devices = {
> > /* 00:00.0 */ {
> > .type = JAILHOUSE_PCI_TYPE_IVSHMEM,
> > .bdf = 0 << 3,
> > .bar_mask = {
> > 0xffffff00, 0xffffffff, 0x00000000,
> > 0x00000000, 0x00000000, 0x00000000,
> > },
> > .shmem_region = 2,
> > .shmem_protocol = JAILHOUSE_SHMEM_PROTO_UNDEFINED,
> > .num_msix_vectors = 0,
> > .domain = 0x0000,
> > },
> > },
> >
> > When I load the uio_ivshmem driver after enabling the root cell, I got the
> > output from the driver which says "using jailhouse mode" (note that I added
> > some prints):
> >
> > OF: PCI: host bridge /vpci@0 ranges:
> > OF: PCI: MEM 0xfc100000..0xfc103fff -> 0xfc100000
> > pci-host-generic fc000000.vpci: ECAM at [mem 0xfc000000-0xfc0f]
> > pci-host-generic fc000000.vpci: PCI host bridge to bus 0000:00
> > pci_bus 0000:00: root bus resource [bus 00]
> > pci_bus 0000:00: root bus resource [mem 0xfc100000-0xfc103fff]
> > pci 0000:00:00.0: [1af4:1110] type 00 class 0xff0000
> > pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000000ff 64bit]
> > pci 0000:00:01.0: [1af4:1110] type 00 class 0xff0000
> > pci 0000:00:01.0: reg 0x10: [mem 0x00000000-0x000000ff 64bit]
> > pci 0000:00:00.0: BAR 0: assigned [mem 0xfc100000-0xfc1000ff 6]
> > pci 0000:00:01.0: BAR 0: assigned [mem 0xfc100100-0xfc1001ff 6]
> > **********ivshmem_pci_probe()
> > uio_ivshmem 0000:00:00.0: enabling device (0000 -> 0002)
> > uio_ivshmem 0000:00:00.0: using jailhouse mode
> > **********ivshmem_pci_probe() mem[1] addr = 0000000800400000, 6
> > **********ivshmem_pci_probe()
> > uio_ivshmem 0000:00:01.0: enabling device (0000 -> 0002)
> > uio_ivshmem 0000:00:01.0: using jailhouse mode
> > **********ivshmem_pci_probe() mem[1] addr = 0000000800500000, 6
> >
> > The two vPCIs regions are mapped and appear in the lspci command:
> >
> > 00:00.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
> > Subsystem: Red Hat, Inc Inter-VM shared memory
> > Flags: bus master, fast devsel, latency 0, IRQ 38
> > Memory at fc100000 (64-bit, non-prefetchable) [size=256]
> > Kernel driver in use: uio_ivshmem
> >
> > 00:01.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
> > Subsystem: Red Hat, Inc Inter-VM shared memory
> > Flags: bus master, fast devsel, latency 0, IRQ 39
> > Memory at fc100100 (64-bit, non-prefetchable) [size=256]
> > Kernel driver in use: uio_ivshmem
> >
> > Also, I can see the addresses in the /proc/iomem:
> >
> > fc000000-fc0fffff : PCI ECAM
> > fc100000-fc103fff : //vpci@0
> > fc100000-fc1000ff : 0000:00:00.0
> > fc100000-fc1000ff : ivshmem
> > fc100100-fc1001ff : 0000:00:01.0
> > fc100100-fc1001ff : ivshmem
> >
> > However, I don't see any output (regarding the PCI) when I load the
> > non-root inmate cell.
>
>
> This is quite strange. Loading the inmate cell, you should see something like
> this in the hypervisor's output:
>
> Adding virtual PCI device xx:xx.x to cell "your-inmate-cell-name"
> Shared memory connection established: "inmate-cell-name" <-->
> "your-root-cell-name"
>
> The check for binding is done on the start address and size of shared regions
> and shmem protocol
>
> >
> > 2) I use #define PCI_CFG_BASE 0xfc000000 in
> > inmates/lib/arm-common/pci.c, which matches the .pci_mmconfig_base =
> > 0xfc000000 in root cell config.
> >
> > 3) The output from the /sys/class/uio/uio0/maps[0,1] is:
> >
> > cat /sys/class/uio/uio0/maps/map0/addr
> > 0x00000000fc100000
> > cat /sys/class/uio/uio0/maps/map0/offset
> > 0x0
> > cat /sys/class/uio/uio0/maps/map0/size
> > 0x0000000000001000
> > cat /sys/class/uio/uio0/maps/map0/name
> > registers
> >
> > cat /sys/class/uio/uio0/maps/map1/addr
> >
> > 0x0000000800400000
> > cat /sys/class/uio/uio0/maps/map1/offset
> > 0x0
> > cat /sys/class/uio/uio0/maps/map1/size
> > 0x0000000000100000
> > cat /sys/class/uio/uio0/maps/map1/name
> > shmem
> >
> > Sorry for the long email, but I guess with this detailed info it may help
> > finding the error.
> >
> > > Il giorno giovedì 12 aprile 2018 08:51:39 UTC+2, Claudio Scordino ha
> > > scritto:
> > > > Hi Giovani,
> > > >
> > > >
> > > >
> > > >
> > > > 2018-04-12 8:01 GMT+02:00 Jan Kiszka <[email protected]>:
> > > > On 2018-04-11 19:40, Giovani Gracioli wrote:
> > > >
> > > > > Here is the output of the unhandled data read:
> > > >
> > > > >
> > > >
> > > > > Unhandled data read at 0xfc100000(2)
> > > >
> > > > >
> > > >
> > > > > FATAL: unhandled trap (exception class 0x24)
> > > >
> > > > > Cell state before exception:
> > > >
> > > > > pc: 0000000000001828 lr: 00000000000015f0 spsr: 60000005 EL1
> > > >
> > > > > sp: 0000000000003f30 esr: 24 1 1400006
> > > >
> > > > > x0: 00000000fc100000 x1: 0000000000000000 x2: 0000000000000002
> > > >
> > > > > x3: 00000000fc000000 x4: 0000000000000000 x5: 0000000000000000
> > > >
> > > > > x6: 0000000000001000 x7: 0000000000000000 x8: 0000000000000000
> > > >
> > > > > x9: 0000000000000000 x10: 0000000000000000 x11: 0000000000000000
> > > >
> > > > > x12: 0000000000000000 x13: 0000000000000000 x14: 0000000000000000
> > > >
> > > > > x15: 0000000000000000 x16: 0000000000000000 x17: 0000000000000000
> > > >
> > > > > x18: 0000000000000000 x19: 0000000000001000 x20: 000000000000ffff
> > > >
> > > > > x21: 0000000000001af4 x22: 0000000000001110 x23: 0000000000001000
> > > >
> > > > > x24: 0000000000002660 x25: 0000000000000000 x26: 0000000000000000
> > > >
> > > > > x27: 0000000000000000 x28: 0000000000000000 x29: 0000000000000000
> > > >
> > > > >
> > > >
> > > > > Parking CPU 3 (Cell: "gic-demo-ivshmem")
> > > >
> > > > >
> > > >
> > > > > Am I missing something in the configuration?
> > > >
> > > >
> > > >
> > > > Possibly. As this is code of Claudio and his colleagues, he may answer
> > > >
> > > > this better.
> > > >
> > > >
> > > >
> > > > A collegue of mine is looking at your issue.
> > > >
> > > >
> > > > BTW, we plan to have a version of the PCI stuff rebased upstream in a
> > > > couple of weeks.
> > > >
> > > > We'll post it on the ML once ready.
> > > >
> > > >
> > > > Regards,
> > > >
> > > >
> > > > Claudio
> > >
> > > Hi all,
> > >
> > > i'm trying to understand what's wrong in the configuration.
> > > 1. The PCI configuration of the root cell (and its relative shared memory
> > > region ) needs a counterpart in the inmate configuration. The two PCI
> > > must have the same bdf. In fact when the configuration is ok, jailhouse
> > > prints on the screen che binding between the PCI devices.
> > >
> > > 2. The starting search address of the inmate library (#define
> > > PCI_CFG_BASE in inmates/lib/arm-common/pci.c) must be the same of the
> > > platform_info section in the root cell configuration (
> > > .platform_info = {
> > > .pci_mmconfig_base = 0x48000000,
> > > .pci_mmconfig_end_bus = 0x0,
> > > .pci_is_virtual = 1,
> > > .....
> > > )
> > >
> > > 3. The best way to access the shared memory region from linux is using
> > > the uio_ivshmem driver. It works correctly on ARM64. It just needs the
> > > UIO kernel module as dependency. Once loaded, you can see memory infos in
> > > /sys/class/uio/uio0/maps[0,1] which are shmem informations and bar
> > > information.
> > >
> Regards,
> --Luca
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
