Il giorno venerdì 22 dicembre 2017 12:54:52 UTC+1, jonas ha scritto: > Den fredag 22 december 2017 kl. 10:32:12 UTC+1 skrev Luca Cuomo: > > Il giorno giovedì 21 dicembre 2017 23:47:39 UTC+1, jonas ha scritto: > > > Den torsdag 21 december 2017 kl. 17:24:30 UTC+1 skrev Jan Kiszka: > > > > On 2017-12-21 16:32, Luca Cuomo wrote: > > > > > Il giorno giovedì 21 dicembre 2017 14:19:48 UTC+1, Jan Kiszka ha > > > > > scritto: > > > > >> On 2017-12-21 10:05, Luca Cuomo wrote: > > > > >>> Il giorno domenica 10 dicembre 2017 17:34:24 UTC+1, jonas ha > > > > >>> scritto: > > > > >>>> Hi, > > > > >>>> > > > > >>>> I'll be making an effort to contribute my work to the master > > > > >>>> branch of Jailhouse within the next couple of weeks. > > > > >>>> > > > > >>>> /Jonas > > > > >>>> > > > > >>>> Den fredag 8 december 2017 kl. 06:47:33 UTC+1 skrev Constantin > > > > >>>> Petra: > > > > >>>>> Hi, > > > > >>>>> > > > > >>>>> > > > > >>>>> I'm resending the patch(es) that were shared by Jonas a while ago. > > > > >>>>> > > > > >>>>> > > > > >>>>> Best Regards, > > > > >>>>> Constantin > > > > >>>>> > > > > >>>>> > > > > >>>>> On Thu, Dec 7, 2017 at 10:08 PM, Henning Schild > > > > >>>>> <henning...@siemens.com> wrote: > > > > >>>>> Hi Claudio, > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> Am Thu, 7 Dec 2017 17:29:45 +0100 > > > > >>>>> > > > > >>>>> schrieb Claudio Scordino <cla...@evidence.eu.com>: > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>>> Hi guys, > > > > >>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>>>> 2017-08-09 15:23 GMT+02:00 Henning Schild > > > > >>>>> > > > > >>>>>> <henning...@siemens.com>: > > > > >>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>>>>> Hey, > > > > >>>>> > > > > >>>>>>> > > > > >>>>> > > > > >>>>>>> unfortunately Jonas never published his overall changes, maybe > > > > >>>>>>> now > > > > >>>>> > > > > >>>>>>> he understands why i kindly asked him to do so. > > > > >>>>> > > > > >>>>>>> I think Jonas maybe ran into every single problem one could > > > > >>>>> > > > > >>>>>>> encounter on the way, so if you read the thread you will > > > > >>>>>>> probably > > > > >>>>> > > > > >>>>>>> be able to come up with a similar patch at some point. That > > > > >>>>>>> would > > > > >>>>> > > > > >>>>>>> be the duplication of efforts, getting a first working patch > > > > >>>>>>> into a > > > > >>>>> > > > > >>>>>>> mergeable form is another story. > > > > >>>>> > > > > >>>>>>> > > > > >>>>> > > > > >>>>>>> If there are legal reasons to not publish code on the list i > > > > >>>>>>> suggest > > > > >>>>> > > > > >>>>>>> you exchange patches between each other. But of cause i would > > > > >>>>>>> like > > > > >>>>> > > > > >>>>>>> to see contributions eventually ;). > > > > >>>>> > > > > >>>>>>> > > > > >>>>> > > > > >>>>>>> > > > > >>>>> > > > > >>>>>> We need to run IVSHMEM on the TX1. > > > > >>>>> > > > > >>>>>> Any chance of upstreaming those patches to not waste time > > > > >>>>> > > > > >>>>>> re-inventing the wheel ? > > > > >>>>> > > > > >>>>>> If that's not possible, please send me a copy privately. > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> Unfortunately i do not have those patches either. I am afraid > > > > >>>>> someone > > > > >>>>> > > > > >>>>> will have to do that over again. > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> But the whole thread is basically about enabling the demo, which > > > > >>>>> is > > > > >>>>> > > > > >>>>> interesting for people just getting started with ivshmem. And for > > > > >>>>> > > > > >>>>> people that want to implement their own protocol on top of it. > > > > >>>>> > > > > >>>>> If you are just looking at running ivshmem-net you are good to > > > > >>>>> go, that > > > > >>>>> > > > > >>>>> code is in a working state. > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> Henning > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>>> Many thanks and best regards, > > > > >>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>>>> Claudio > > > > >>> > > > > >>> Hi all, > > > > >>> > > > > >>> i've applied the provided patch and i'm trying to connect the linux > > > > >>> root cell with the bare metal cell running the > > > > >>> inmate/demo/arm/ivshmem-demo.c. I've attached the used > > > > >>> configurations (jetson-tx1-ivshmem for the root cell and the other > > > > >>> one for the bare metal). > > > > >>> When i create the bare metal cell the connection between pci > > > > >>> devices is correctly up. > > > > >>> ----------------------------------------------------------------- > > > > >>> Initializing Jailhouse hypervisor on CPU 2 > > > > >>> Code location: 0x0000ffffc0200050 > > > > >>> Page pool usage after early setup: mem 63/16358, remap 64/131072 > > > > >>> Initializing processors: > > > > >>> CPU 2... OK > > > > >>> CPU 1... OK > > > > >>> CPU 3... OK > > > > >>> CPU 0... OK > > > > >>> Adding virtual PCI device 00:0f.0 to cell "Jetson-TX1-ivshmem" > > > > >>> Page pool usage after late setup: mem 74/16358, remap 69/131072 > > > > >>> Activating hypervisor > > > > >>> Adding virtual PCI device 00:0f.0 to cell "jetson-tx1-demo-shmem" > > > > >>> Shared memory connection established: "jetson-tx1-demo-shmem" <--> > > > > >>> "Jetson-TX1-ivshmem" > > > > >>> ------------------------------------------------------------------- > > > > >>> > > > > >>> 1st problem: no PCI device appears in linux (lspci does not return > > > > >>> anything) > > > > >> > > > > >> Are you using a Linux kernel with the Jailhouse-related patches? Did > > > > >> you > > > > >> enable CONFIG_PCI_HOST_GENERIC and CONFIG_PCI_DOMAINS? On most ARM > > > > >> systems, Jailhouse exposes the ivshmem devices via a virtual host > > > > >> bridge. > > > > > > > > > > Yes, i'm using a kernel for jailhouse on Jetson tx1. The kernel has > > > > > the above CONFIGS. In root cell configuration if i put > > > > > ".pci_is_virtual = 1," dmesg shows the message: > > > > > [ 102.100405] jailhouse: CONFIG_OF_OVERLAY disabled > > > > > [ 102.100417] jailhouse: failed to add virtual host controller > > > > > [ 102.100422] The Jailhouse is opening. > > > > > > > > > > If i put it to 0 there is no error message but not pci device is set > > > > > as before. > > > > > > > > > >> > > > > >>> > > > > >>> Then i launch the ivshmem-demo.bin. I've made some modification: > > > > >>> * in inmate/lib/arm-common/pci.c the #define PCI_CFG_BASE > > > > >>> (0x48000000) > > > > >>> as defined in jetson-tx-ivshmem.c: > > > > >>> config.header.platform_info.pci_mmconfig_base. > > > > >>> In the same file i've enabled the print of > > > > >>> pci_read/write_config > > > > >>> * in inmate/demos/arm/ivshmem-demo.c i've removed the filter on > > > > >>> class/revision in order to get a suitable pci device with > > > > >>> proper > > > > >>> deviceId:vendorId > > > > >>> > > > > >>> When the bare metal starts, it iterate on the memory with a lot of > > > > >>> read > > > > >>> > > > > >>> pci_read_config(bdf:0x0, addr:0x0000000000000000, size:0x2), > > > > >>> reg_addr0x48000000 > > > > >>> pci_read_config(bdf:0x1, addr:0x0000000000000000, size:0x2), > > > > >>> reg_addr0x48000100 > > > > >>> pci_read_config(bdf:0x2, addr:0x0000000000000000, size:0x2), > > > > >>> reg_addr0x48000200 > > > > >>> pci_read_config(bdf:0x3, addr:0x0000000000000000, size:0x2), > > > > >>> reg_addr0x48000300 > > > > >>> > > > > >>> .... after a while something happens (follow <---) > > > > >>> > > > > >>> IVSHMEM: Found 1af4:1110 at 07:10.0 <--- > > > > >>> pci_read_config(bdf:0x780, addr:0x0000000000000008, size:0x4), > > > > >>> reg_addr0x48078008 > > > > >>> IVSHMEM: class/revision ff010000, not supported skipping device > > > > >>> <--- //IGNORED > > > > >>> pci_read_config(bdf:0x780, addr:0x0000000000000006, size:0x2), > > > > >>> reg_addr0x48078004 > > > > >>> pci_read_config(bdf:0x780, addr:0x0000000000000034, size:0x1), > > > > >>> reg_addr0x48078034 > > > > >>> IVSHMEM ERROR: device is not MSI-X capable <--- > > > > >>> pci_read_config(bdf:0x780, addr:0x000000000000004c, size:0x4), > > > > >>> reg_addr0x4807804c > > > > >>> pci_read_config(bdf:0x780, addr:0x0000000000000048, size:0x4), > > > > >>> reg_addr0x48078048 > > > > >>> pci_read_config(bdf:0x780, addr:0x0000000000000044, size:0x4), > > > > >>> reg_addr0x48078044 > > > > >>> pci_read_config(bdf:0x780, addr:0x0000000000000040, size:0x4), > > > > >>> reg_addr0x48078040 > > > > >>> IVSHMEM: shmem is at 0x000000007bf00000 <--- > > > > >>> pci_write_config(bdf:0x780, addr:0x0000000000000014, value:0x0, > > > > >>> size:0x4), reg_addr0x48078014 > > > > >>> pci_write_config(bdf:0x780, addr:0x0000000000000010, > > > > >>> value:0x7c000000, size:0x4), reg_addr0x48078010 > > > > >>> IVSHMEM: bar0 is at 0x000000007c000000 <--- > > > > >>> .... > > > > >>> IVSHMEM: mapped shmem and bars, got position 0x0000000000000001 > > > > >>> <--- > > > > >>> IVSHMEM: Enabled IRQ:0x9b > > > > >>> IVSHMEM: Enabling IVSHMEM_IRQs > > > > >>> ... > > > > >>> > > > > >>> 2nd problem: > > > > >>> > > > > >>> At the end of devices scanning here is the error: > > > > >>> > > > > >>> > > > > >>> Unhandled data read at 0x10000(1) > > > > >>> > > > > >>> FATAL: unhandled trap (exception class 0x24) > > > > >>> Cell state before exception: > > > > >>> pc: 0000000000001a80 lr: 0000000000001ac0 spsr: 20000005 EL1 > > > > >>> sp: 0000000000003da0 esr: 24 1 1130007 > > > > >>> x0: 0000000070006000 x1: 00000000000000ff x2: 0000000000002238 > > > > >>> x3: ffffffffffffffff x4: 0000000000003de0 x5: 0000000000000002 > > > > >>> x6: 0000000000000000 x7: 0000000000000008 x8: 0000000000003de8 > > > > >>> x9: 0000000000000000 x10: 0000000000000000 x11: 0000000000002690 > > > > >>> x12: 0000000000000000 x13: 0000000000000000 x14: 0000000000000020 > > > > >>> x15: 0000000000000000 x16: 0000000000000000 x17: 0000000000000000 > > > > >>> x18: 0000000000000000 x19: 00000000000000ff x20: 0000000000010000 > > > > >>> x21: 0000000000001000 x22: 00000000000011b8 x23: 00000000ffffffd0 > > > > >>> x24: 0000000000003f70 x25: 000000000000288b x26: 000000000000272c > > > > >>> x27: 0000000000000001 x28: 0000000000000780 x29: 0000000000000000 > > > > >>> > > > > >>> Parking CPU 3 (Cell: "jetson-tx1-demo-shmem") > > > > >>> > > > > >> > > > > >> That indicates a mismatch between the cell configuration and the > > > > >> hardware information that is either hard-coded into the inmate code > > > > >> or > > > > >> otherwise provided to it. I didn't check the ivshmem-demo code but I > > > > >> suspect the location of the virtual PCI host controller is > > > > >> hard-coded. A > > > > >> Linux inmate would get it from a device tree provided to its boot. > > > > >> > > > > >> Jan > > > > > I've checked and the .pci_mmconfig_base is equal to > > > > > inmate/lib/arm-common/pci.c:#define PCI_CFG_BASE. > > > > > The previous error was on a read on a not terminated string. I've > > > > > fixed the problem and the new output looks like this > > > > > > > > > > IVSHMEM: Found 1af4:1110 at 07:10.0 > > > > > IVSHMEM: class/revision ff010000, not supported skipping device > > > > > //forced to be true > > > > > > > > This device was configured in the cell config to be used as virtual > > > > Ethernet (over ivshmem). You should set shmem_protocol on both ends to 0 > > > > in order to define the right type. > > > > > > > > > > That's right. I added some printouts in the hypervisor for this case in > > > the patch indicating protocol mismatch between root-cell and bare-metal > > > cell configs. > > > > Ok, JAILHOUSE_SHMEM_PROTO_UNDEFINED is the right value. But i can't still > > see any pci device in linux. > > > > > > > > > > IVSHMEM ERROR: device is not MSI-X capable > > > > > > > > Not sure is this is now only a warning, but if the demo code still does > > > > not support INTx, event signaling via interrupts will not work on this > > > > host platform. > > > > > > > > Jan > > > > > > > > > > Yes, the printout originates from the ivshmem-demo for x86, which > > > requires support for MSI-X (which is available and > > > supported/used/required by Jailhouse on x86, but not on ARM, if I've > > > understood it correctly). The printout should have been removed in the > > > patch... > > > > > > /Jonas > > > > In the ivshmem-demo i see that the interrupt sending is still done writing > > to the doorbell register. > > Now that i've no access via uio driver (no pci device in linux) i'm trying > > to send interrupt to the bare metal by mmapping /dev/mem with the proper > > offset and then writing to the doorbell register. Is it the right way? > > > > Thanks, > > --Luca > > > > Yes, that's how I did it as well. > > Take a look at: > ivshmem-guest-code/uio/tests/Interrupts/VM/uio_send.c > for an idea of how it is supposed to work. > > I did a user-space inter-cell communication driver based on this, which > basically first mmaps 256 B from /dev/uio0, then mmaps the area corresponding > to the shmem area provided by the Jailhouse IVSHMEM virtual PCI device.
Yes, i've succesfully used it on x86 platform, mmpapping /dev/uio maps[0] as bar0 and maps[1] as shmem. But unfortunately the uio_ivshmem cannot detect any pci device. Furthermore i've added the section you suggestion of irqchips for the bare metal inmate configuration. The address is different (for jetson tx1 gicd_base = 0x50041000). But when i run the bare metal ivshmem-demo, immediately after the call gic_enable_irq, then interrupts arrive (continously) and serving them blocks the inmate execution. IVSHMEM: handle_irq(irqn:155) - interrupt #0 IVSHMEM: handle_irq(irqn:155) - interrupt #1 IVSHMEM: handle_irq(irqn:155) - interrupt #2 IVSHMEM: handle_irq(irqn:155) - interrupt #3 IVSHMEM: handle_irq(irqn:155) - interrupt #4 IVSHMEM: handle_irq(irqn:155) - interrupt #5 IVSHMEM: handle_irq(irqn:155) - interrupt #6 IVSHMEM: handle_irq(irqn:155) - interrupt #7 ... and so on And later the machine reboot by itself. > I then separated the shmem area into one tx- and one rx-part and used the > (MMIO) LSTATE register provided in BAR0 between two cells inter-connected via > IVSHMEM. Writing to the LSTATE register in one cell is intercepted by > Jailhouse, which updates the RSTATE register of the peer cell and triggers an > IVSHMEM interrupt in the peer. Once the interrupt is registered by the peer, > it reads its RSTATE register to determine at what offset in the rx-part to > look for the message from the transmitting cell. The tx-part of the shmem in > the transmitting cell is set up to be the rx-part in the receiving cell, and > vice versa. > > /Jonas > > > > > > > > > IVSHMEM: shmem is at 0x000000007bf00000 > > > > > IVSHMEM: bar0 is at 0x000000007c000000 > > > > > IVSHMEM: mapped shmem and bars, got position 0x0000000000000001 > > > > > IVSHMEM: Enabled IRQ:0x9b > > > > > IVSHMEM: Enabling IVSHMEM_IRQs > > > > > IVSHMEM: Done setting up... > > > > > IVSHMEM: 07:10.0 sending IRQ (by writing 1 to 0x7c00000c) > > > > > IVSHMEM: waiting for interrupt. > > > > > > > > > > It looks like a misconfigured area (bdf, class/revision are not as > > > > > expected). > > > > > > > > > > --Luca > > > > > > > > > > > > > > > -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
