On 26.07.21 19:14, Huang Shihua wrote: > > > On Wednesday, 21 July 2021 at 17:50:53 UTC+2 [email protected] wrote: > > On 13.07.21 18:09, Huang Shihua wrote: > > HI, > > > > Currently, I'm trying to run the ivshmem-demo to establish > communication > > between Linux root cell and one non-root cell. Configuration files > are > > attached. > > > > Two cases were tested: > > > > 1. Let the non-root cell load the ivshmem-demo and then target at > > itself (target=1). _All interrupts can be sent and received > correctly_. > > 2. Let the root cell and the non-root cell send interrupts to each > > other. I.e., root cell runs /./tools/demos/ivshmem-demo -t 1, /while > > the non-root cell load /inmates/demos/x86/ivshmem-demo.bin -s > > "target=0" -a 0x1000 /and then run. The result turned out to be, > > * the non-root cell got the interrupts from the root cell, > > * _while the root cell did not receive any interrupt._ > > > > As Jan mentioned > > > in https://groups.google.com/g/jailhouse-dev/c/GRCWFzNaHX8/m/ht8z51BOCgAJ > <https://groups.google.com/g/jailhouse-dev/c/GRCWFzNaHX8/m/ht8z51BOCgAJ>, > > > tuning the iommu index should do the trick. > > However, unfortunately, it did not work for me :c > > > > There are 8 iommu units on the hardware, I tuned the iommu index > in the > > Wow, 8 units... > > > root cell configuration from 0 to 7. The same behavior, no interrupts > > were received by the root cell, remains when tuning the index from > 0 to > > 6. When the iommu is set to 7, the kernel crashed immediately when > the > > demo was started on the non-root cell. > > > > Any idea regarding why the root cell always failed to receive > interrupts? > > This may require in-detail debugging. For that, you would have to > instrument the hypervisor along its virtual IRQ injection path. That > starts in ivshmem_trigger_interrupt() (hypervisor/ivshmem.c). The > sending side will call it on writing the doorbell registers. Check > along > this call path if conditions to actually send the IRQ are not met. > > If all are met, the hypervisor sends an IPI to a target cell CPU (will > be directly delivered to the guest) that should cause the normal IRQ > processing there. But usually, we do not get so far in such cases. > > Another function of interest here is arch_ivshmem_update_msix() when > called for the root cell while it defines where ivshmem IRQs should go > to. Possibly, Jailhouse decides that the programming Linux issued is > not > valid and therefore leaves the irq_cache that > arch_ivshmem_trigger_interrupt() uses invalid. You can also check that > via instrumentations (printk). > > > Indeed, when .iommu is assigned as 0,1,..6, irq_cache is invalid. I suspect > the reason is that their correpsonding VT-d interrupt remappting table > entries > are not for ivshmem devices, i.e., unmatched device ID. > When .iommu is tuned to 7, irq_cache becomes valid. >
OK, then we know what needs to be set. I will have to check eventually if we can read out that information also from sysfs so that this guessing can end. > (BTW, as I mentioned before, the kernel crashed immediately when the > demo was started on the non-root cell. _One missing detail here is_, on the > root-cell side, ./tools/demos/ivshmem-demo is running/has run, i.e., > init_control has been set to 1. If ./tools/demos/ivshmem-demo has not been > run on the root cell yet, then starting the demo on the non-root cell > will not > kill the kernel.) Now we need to understand the crash. The root cell kernel oopses, right? Any logs from that? And what do yo mean with init_control? > > To avoid the kernel crashing situation, I only ran the demo on the > non-root cell. With .iommu being set validly, I will expect at least > seeing the > interrupt count increases, when grep ivshmem /proc/interrupts. > But nope, _still no interrupts received on the root cell_. > If there is no driver registered on the root side or not opened (by the demo app), then the interrupt reception is disabled. We need to debug the "hot" case. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/2d2c72b6-cae0-e210-8db2-630b33180335%40siemens.com.
