On Wednesday, 21 July 2021 at 17:50:53 UTC+2 [email protected] wrote:
> On 13.07.21 18:09, Huang Shihua wrote: > > HI, > > > > Currently, I'm trying to run the ivshmem-demo to establish communication > > between Linux root cell and one non-root cell. Configuration files are > > attached. > > > > Two cases were tested: > > > > 1. Let the non-root cell load the ivshmem-demo and then target at > > itself (target=1). _All interrupts can be sent and received correctly_. > > 2. Let the root cell and the non-root cell send interrupts to each > > other. I.e., root cell runs /./tools/demos/ivshmem-demo -t 1, /while > > the non-root cell load /inmates/demos/x86/ivshmem-demo.bin -s > > "target=0" -a 0x1000 /and then run. The result turned out to be, > > * the non-root cell got the interrupts from the root cell, > > * _while the root cell did not receive any interrupt._ > > > > As Jan mentioned > > in > https://groups.google.com/g/jailhouse-dev/c/GRCWFzNaHX8/m/ht8z51BOCgAJ, > > tuning the iommu index should do the trick. > > However, unfortunately, it did not work for me :c > > > > There are 8 iommu units on the hardware, I tuned the iommu index in the > > Wow, 8 units... > > > root cell configuration from 0 to 7. The same behavior, no interrupts > > were received by the root cell, remains when tuning the index from 0 to > > 6. When the iommu is set to 7, the kernel crashed immediately when the > > demo was started on the non-root cell. > > > > Any idea regarding why the root cell always failed to receive > interrupts? > > This may require in-detail debugging. For that, you would have to > instrument the hypervisor along its virtual IRQ injection path. That > starts in ivshmem_trigger_interrupt() (hypervisor/ivshmem.c). The > sending side will call it on writing the doorbell registers. Check along > this call path if conditions to actually send the IRQ are not met. > > If all are met, the hypervisor sends an IPI to a target cell CPU (will > be directly delivered to the guest) that should cause the normal IRQ > processing there. But usually, we do not get so far in such cases. > > Another function of interest here is arch_ivshmem_update_msix() when > called for the root cell while it defines where ivshmem IRQs should go > to. Possibly, Jailhouse decides that the programming Linux issued is not > valid and therefore leaves the irq_cache that > arch_ivshmem_trigger_interrupt() uses invalid. You can also check that > via instrumentations (printk). Indeed, when .iommu is assigned as 0,1,..6, irq_cache is invalid. I suspect the reason is that their correpsonding VT-d interrupt remappting table entries are not for ivshmem devices, i.e., unmatched device ID. When .iommu is tuned to 7, irq_cache becomes valid. (BTW, as I mentioned before, the kernel crashed immediately when the demo was started on the non-root cell. *One missing detail here is*, on the root-cell side, ./tools/demos/ivshmem-demo is running/has run, i.e., init_control has been set to 1. If ./tools/demos/ivshmem-demo has not been run on the root cell yet, then starting the demo on the non-root cell will not kill the kernel.) To avoid the kernel crashing situation, I only ran the demo on the non-root cell. With .iommu being set validly, I will expect at least seeing the interrupt count increases, when grep ivshmem /proc/interrupts. But nope, *still no interrupts received on the root cell*. > We likely need better tooling (tracing) for these hairy cases... > > Jan > > -- > Siemens AG, T RDA IOT > Corporate Competence Center Embedded Linux Any more hints? Kind regards, Shihua -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/6f3f0b24-cfee-4c08-86c4-8a0cc9183a2fn%40googlegroups.com.
