Den fredag 22 december 2017 kl. 10:32:12 UTC+1 skrev Luca Cuomo: > Il giorno giovedì 21 dicembre 2017 23:47:39 UTC+1, jonas ha scritto: > > Den torsdag 21 december 2017 kl. 17:24:30 UTC+1 skrev Jan Kiszka: > > > On 2017-12-21 16:32, Luca Cuomo wrote: > > > > Il giorno giovedì 21 dicembre 2017 14:19:48 UTC+1, Jan Kiszka ha > > > > scritto: > > > >> On 2017-12-21 10:05, Luca Cuomo wrote: > > > >>> Il giorno domenica 10 dicembre 2017 17:34:24 UTC+1, jonas ha scritto: > > > >>>> Hi, > > > >>>> > > > >>>> I'll be making an effort to contribute my work to the master branch > > > >>>> of Jailhouse within the next couple of weeks. > > > >>>> > > > >>>> /Jonas > > > >>>> > > > >>>> Den fredag 8 december 2017 kl. 06:47:33 UTC+1 skrev Constantin Petra: > > > >>>>> Hi, > > > >>>>> > > > >>>>> > > > >>>>> I'm resending the patch(es) that were shared by Jonas a while ago. > > > >>>>> > > > >>>>> > > > >>>>> Best Regards, > > > >>>>> Constantin > > > >>>>> > > > >>>>> > > > >>>>> On Thu, Dec 7, 2017 at 10:08 PM, Henning Schild > > > >>>>> <[email protected]> wrote: > > > >>>>> Hi Claudio, > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> Am Thu, 7 Dec 2017 17:29:45 +0100 > > > >>>>> > > > >>>>> schrieb Claudio Scordino <[email protected]>: > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>>> Hi guys, > > > >>>>> > > > >>>>>> > > > >>>>> > > > >>>>>> 2017-08-09 15:23 GMT+02:00 Henning Schild > > > >>>>> > > > >>>>>> <[email protected]>: > > > >>>>> > > > >>>>>> > > > >>>>> > > > >>>>>>> Hey, > > > >>>>> > > > >>>>>>> > > > >>>>> > > > >>>>>>> unfortunately Jonas never published his overall changes, maybe now > > > >>>>> > > > >>>>>>> he understands why i kindly asked him to do so. > > > >>>>> > > > >>>>>>> I think Jonas maybe ran into every single problem one could > > > >>>>> > > > >>>>>>> encounter on the way, so if you read the thread you will probably > > > >>>>> > > > >>>>>>> be able to come up with a similar patch at some point. That would > > > >>>>> > > > >>>>>>> be the duplication of efforts, getting a first working patch into > > > >>>>>>> a > > > >>>>> > > > >>>>>>> mergeable form is another story. > > > >>>>> > > > >>>>>>> > > > >>>>> > > > >>>>>>> If there are legal reasons to not publish code on the list i > > > >>>>>>> suggest > > > >>>>> > > > >>>>>>> you exchange patches between each other. But of cause i would like > > > >>>>> > > > >>>>>>> to see contributions eventually ;). > > > >>>>> > > > >>>>>>> > > > >>>>> > > > >>>>>>> > > > >>>>> > > > >>>>>> We need to run IVSHMEM on the TX1. > > > >>>>> > > > >>>>>> Any chance of upstreaming those patches to not waste time > > > >>>>> > > > >>>>>> re-inventing the wheel ? > > > >>>>> > > > >>>>>> If that's not possible, please send me a copy privately. > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> Unfortunately i do not have those patches either. I am afraid > > > >>>>> someone > > > >>>>> > > > >>>>> will have to do that over again. > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> But the whole thread is basically about enabling the demo, which is > > > >>>>> > > > >>>>> interesting for people just getting started with ivshmem. And for > > > >>>>> > > > >>>>> people that want to implement their own protocol on top of it. > > > >>>>> > > > >>>>> If you are just looking at running ivshmem-net you are good to go, > > > >>>>> that > > > >>>>> > > > >>>>> code is in a working state. > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> Henning > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>>> Many thanks and best regards, > > > >>>>> > > > >>>>>> > > > >>>>> > > > >>>>>> Claudio > > > >>> > > > >>> Hi all, > > > >>> > > > >>> i've applied the provided patch and i'm trying to connect the linux > > > >>> root cell with the bare metal cell running the > > > >>> inmate/demo/arm/ivshmem-demo.c. I've attached the used configurations > > > >>> (jetson-tx1-ivshmem for the root cell and the other one for the bare > > > >>> metal). > > > >>> When i create the bare metal cell the connection between pci devices > > > >>> is correctly up. > > > >>> ----------------------------------------------------------------- > > > >>> Initializing Jailhouse hypervisor on CPU 2 > > > >>> Code location: 0x0000ffffc0200050 > > > >>> Page pool usage after early setup: mem 63/16358, remap 64/131072 > > > >>> Initializing processors: > > > >>> CPU 2... OK > > > >>> CPU 1... OK > > > >>> CPU 3... OK > > > >>> CPU 0... OK > > > >>> Adding virtual PCI device 00:0f.0 to cell "Jetson-TX1-ivshmem" > > > >>> Page pool usage after late setup: mem 74/16358, remap 69/131072 > > > >>> Activating hypervisor > > > >>> Adding virtual PCI device 00:0f.0 to cell "jetson-tx1-demo-shmem" > > > >>> Shared memory connection established: "jetson-tx1-demo-shmem" <--> > > > >>> "Jetson-TX1-ivshmem" > > > >>> ------------------------------------------------------------------- > > > >>> > > > >>> 1st problem: no PCI device appears in linux (lspci does not return > > > >>> anything) > > > >> > > > >> Are you using a Linux kernel with the Jailhouse-related patches? Did > > > >> you > > > >> enable CONFIG_PCI_HOST_GENERIC and CONFIG_PCI_DOMAINS? On most ARM > > > >> systems, Jailhouse exposes the ivshmem devices via a virtual host > > > >> bridge. > > > > > > > > Yes, i'm using a kernel for jailhouse on Jetson tx1. The kernel has the > > > > above CONFIGS. In root cell configuration if i put ".pci_is_virtual = > > > > 1," dmesg shows the message: > > > > [ 102.100405] jailhouse: CONFIG_OF_OVERLAY disabled > > > > [ 102.100417] jailhouse: failed to add virtual host controller > > > > [ 102.100422] The Jailhouse is opening. > > > > > > > > If i put it to 0 there is no error message but not pci device is set as > > > > before. > > > > > > > >> > > > >>> > > > >>> Then i launch the ivshmem-demo.bin. I've made some modification: > > > >>> * in inmate/lib/arm-common/pci.c the #define PCI_CFG_BASE > > > >>> (0x48000000) > > > >>> as defined in jetson-tx-ivshmem.c: > > > >>> config.header.platform_info.pci_mmconfig_base. > > > >>> In the same file i've enabled the print of pci_read/write_config > > > >>> * in inmate/demos/arm/ivshmem-demo.c i've removed the filter on > > > >>> class/revision in order to get a suitable pci device with > > > >>> proper > > > >>> deviceId:vendorId > > > >>> > > > >>> When the bare metal starts, it iterate on the memory with a lot of > > > >>> read > > > >>> > > > >>> pci_read_config(bdf:0x0, addr:0x0000000000000000, size:0x2), > > > >>> reg_addr0x48000000 > > > >>> pci_read_config(bdf:0x1, addr:0x0000000000000000, size:0x2), > > > >>> reg_addr0x48000100 > > > >>> pci_read_config(bdf:0x2, addr:0x0000000000000000, size:0x2), > > > >>> reg_addr0x48000200 > > > >>> pci_read_config(bdf:0x3, addr:0x0000000000000000, size:0x2), > > > >>> reg_addr0x48000300 > > > >>> > > > >>> .... after a while something happens (follow <---) > > > >>> > > > >>> IVSHMEM: Found 1af4:1110 at 07:10.0 <--- > > > >>> pci_read_config(bdf:0x780, addr:0x0000000000000008, size:0x4), > > > >>> reg_addr0x48078008 > > > >>> IVSHMEM: class/revision ff010000, not supported skipping device <--- > > > >>> //IGNORED > > > >>> pci_read_config(bdf:0x780, addr:0x0000000000000006, size:0x2), > > > >>> reg_addr0x48078004 > > > >>> pci_read_config(bdf:0x780, addr:0x0000000000000034, size:0x1), > > > >>> reg_addr0x48078034 > > > >>> IVSHMEM ERROR: device is not MSI-X capable <--- > > > >>> pci_read_config(bdf:0x780, addr:0x000000000000004c, size:0x4), > > > >>> reg_addr0x4807804c > > > >>> pci_read_config(bdf:0x780, addr:0x0000000000000048, size:0x4), > > > >>> reg_addr0x48078048 > > > >>> pci_read_config(bdf:0x780, addr:0x0000000000000044, size:0x4), > > > >>> reg_addr0x48078044 > > > >>> pci_read_config(bdf:0x780, addr:0x0000000000000040, size:0x4), > > > >>> reg_addr0x48078040 > > > >>> IVSHMEM: shmem is at 0x000000007bf00000 <--- > > > >>> pci_write_config(bdf:0x780, addr:0x0000000000000014, value:0x0, > > > >>> size:0x4), reg_addr0x48078014 > > > >>> pci_write_config(bdf:0x780, addr:0x0000000000000010, > > > >>> value:0x7c000000, size:0x4), reg_addr0x48078010 > > > >>> IVSHMEM: bar0 is at 0x000000007c000000 <--- > > > >>> .... > > > >>> IVSHMEM: mapped shmem and bars, got position 0x0000000000000001 <--- > > > >>> IVSHMEM: Enabled IRQ:0x9b > > > >>> IVSHMEM: Enabling IVSHMEM_IRQs > > > >>> ... > > > >>> > > > >>> 2nd problem: > > > >>> > > > >>> At the end of devices scanning here is the error: > > > >>> > > > >>> > > > >>> Unhandled data read at 0x10000(1) > > > >>> > > > >>> FATAL: unhandled trap (exception class 0x24) > > > >>> Cell state before exception: > > > >>> pc: 0000000000001a80 lr: 0000000000001ac0 spsr: 20000005 EL1 > > > >>> sp: 0000000000003da0 esr: 24 1 1130007 > > > >>> x0: 0000000070006000 x1: 00000000000000ff x2: 0000000000002238 > > > >>> x3: ffffffffffffffff x4: 0000000000003de0 x5: 0000000000000002 > > > >>> x6: 0000000000000000 x7: 0000000000000008 x8: 0000000000003de8 > > > >>> x9: 0000000000000000 x10: 0000000000000000 x11: 0000000000002690 > > > >>> x12: 0000000000000000 x13: 0000000000000000 x14: 0000000000000020 > > > >>> x15: 0000000000000000 x16: 0000000000000000 x17: 0000000000000000 > > > >>> x18: 0000000000000000 x19: 00000000000000ff x20: 0000000000010000 > > > >>> x21: 0000000000001000 x22: 00000000000011b8 x23: 00000000ffffffd0 > > > >>> x24: 0000000000003f70 x25: 000000000000288b x26: 000000000000272c > > > >>> x27: 0000000000000001 x28: 0000000000000780 x29: 0000000000000000 > > > >>> > > > >>> Parking CPU 3 (Cell: "jetson-tx1-demo-shmem") > > > >>> > > > >> > > > >> That indicates a mismatch between the cell configuration and the > > > >> hardware information that is either hard-coded into the inmate code or > > > >> otherwise provided to it. I didn't check the ivshmem-demo code but I > > > >> suspect the location of the virtual PCI host controller is hard-coded. > > > >> A > > > >> Linux inmate would get it from a device tree provided to its boot. > > > >> > > > >> Jan > > > > I've checked and the .pci_mmconfig_base is equal to > > > > inmate/lib/arm-common/pci.c:#define PCI_CFG_BASE. > > > > The previous error was on a read on a not terminated string. I've fixed > > > > the problem and the new output looks like this > > > > > > > > IVSHMEM: Found 1af4:1110 at 07:10.0 > > > > IVSHMEM: class/revision ff010000, not supported skipping device > > > > //forced to be true > > > > > > This device was configured in the cell config to be used as virtual > > > Ethernet (over ivshmem). You should set shmem_protocol on both ends to 0 > > > in order to define the right type. > > > > > > > That's right. I added some printouts in the hypervisor for this case in the > > patch indicating protocol mismatch between root-cell and bare-metal cell > > configs. > > Ok, JAILHOUSE_SHMEM_PROTO_UNDEFINED is the right value. But i can't still see > any pci device in linux. > > > > > > > IVSHMEM ERROR: device is not MSI-X capable > > > > > > Not sure is this is now only a warning, but if the demo code still does > > > not support INTx, event signaling via interrupts will not work on this > > > host platform. > > > > > > Jan > > > > > > > Yes, the printout originates from the ivshmem-demo for x86, which requires > > support for MSI-X (which is available and supported/used/required by > > Jailhouse on x86, but not on ARM, if I've understood it correctly). The > > printout should have been removed in the patch... > > > > /Jonas > > In the ivshmem-demo i see that the interrupt sending is still done writing to > the doorbell register. > Now that i've no access via uio driver (no pci device in linux) i'm trying to > send interrupt to the bare metal by mmapping /dev/mem with the proper offset > and then writing to the doorbell register. Is it the right way? > > Thanks, > --Luca >
Yes, that's how I did it as well. Take a look at: ivshmem-guest-code/uio/tests/Interrupts/VM/uio_send.c for an idea of how it is supposed to work. I did a user-space inter-cell communication driver based on this, which basically first mmaps 256 B from /dev/uio0, then mmaps the area corresponding to the shmem area provided by the Jailhouse IVSHMEM virtual PCI device. I then separated the shmem area into one tx- and one rx-part and used the (MMIO) LSTATE register provided in BAR0 between two cells inter-connected via IVSHMEM. Writing to the LSTATE register in one cell is intercepted by Jailhouse, which updates the RSTATE register of the peer cell and triggers an IVSHMEM interrupt in the peer. Once the interrupt is registered by the peer, it reads its RSTATE register to determine at what offset in the rx-part to look for the message from the transmitting cell. The tx-part of the shmem in the transmitting cell is set up to be the rx-part in the receiving cell, and vice versa. /Jonas > > > > > > IVSHMEM: shmem is at 0x000000007bf00000 > > > > IVSHMEM: bar0 is at 0x000000007c000000 > > > > IVSHMEM: mapped shmem and bars, got position 0x0000000000000001 > > > > IVSHMEM: Enabled IRQ:0x9b > > > > IVSHMEM: Enabling IVSHMEM_IRQs > > > > IVSHMEM: Done setting up... > > > > IVSHMEM: 07:10.0 sending IRQ (by writing 1 to 0x7c00000c) > > > > IVSHMEM: waiting for interrupt. > > > > > > > > It looks like a misconfigured area (bdf, class/revision are not as > > > > expected). > > > > > > > > --Luca > > > > > > > > > > > > -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
