> Subject: Re: jailhouse jitter? > > On 03.07.20 20:08, Angelo Ruocco wrote: > > Adding José Martins in cc (direct gicv2 injection was his idea) as he > > might be interested in this. > > > > On 02/07/2020, Jan Kiszka <[email protected]> wrote: > >> On 02.07.20 18:45, Jan Kiszka wrote: > >>> On 02.07.20 15:31, Angelo Ruocco wrote: > >>>> On 02/07/2020, 'Nikhil Devshatwar' via Jailhouse > >>>> <[email protected]> wrote: > >>>>> On 02/07/20 5:14 pm, Jan Kiszka wrote: > >>>>>> BTW, regarding direct interrupt delivery on ARM: In > >>>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2 > >>>>>> > Flwn.net%2FArticles%2F820830&data=02%7C01%7Cpeng.fan%40nxp.co > >>>>>> > m%7C6ecf1e2eb2714099a5ae08d82cf63a2c%7C686ea1d3bc2b4c6fa92cd99 > c5c > >>>>>> > 301635%7C0%7C0%7C637308783336041620&sdata=PviVkkKPB8Klql5U > z%2 > >>>>>> B9rhZvAZxG00siR2yx%2F48P6kg4%3D&reserved=0, it is > reported > >>>>>> that Bao has "found a way to map interrupts directly into > >>>>>> guests". I didn't find the time yet to check if that is actually > >>>>>> exit-free delivery, and that as a smart trick or rather a > >>>>>> problematic hack. Or if that sentence is rather a misunderstanding. > There is also the sentence: "Interrupts [...] have to be mediated through the > hypervisor, which is unfortunate since that increases latency." > >>>>>> > >>>>> > >>>>> I found this interesting and tried to read more about this. > >>>>> I found some commits at [1] which does the direct injection. > >>>>> Here is my rough understanding: > >>>>> > >>>>> * He is not setting the HCR_EL.FMO bit and that way all virtual > >>>>> interrupts are turned off > >>>>> * There is a new handler for handling "lower_el_aarch64_fiq" > >>>>> which ends up being handled by the Hypervisor > >>>>> * GICD is still being emulated so guests won't be able to route > >>>>> interrupts to wrong CPUs. Isolation is maintained > >>>>> * For triggering interrupts from Hypervisor (SGIs, etc) he is > >>>>> using SMC calls and has a new service implemented in the ATF [2] > >>>>> * I could not understand how the lower_el_aarch64_fiq exception is > >>>>> fired so that the Hypervisor is invoked > >>>>> > >>>>> My guess is that most of the code change ihere s to enable > >>>>> interrupts in the Hypervisor. Resetting HCR_EL2.FMO would send the > >>>>> interrupts at EL1 directly. > >>>> > >>>> Yup, that's more or less it. > >>>> > >>>> Just one clarifications, Bao *does* set the HCR_EL2.FMO, it doesn't > >>>> set the HCR_EL2.HMO. The HMO bit lets the interrupts through, and > >>>> they > >>> > >>> HCR_EL2.HMO? On which revision of the ARMv8 spec are we looking? > >>> Cannot find that in mine. > > > > Ugh, it's a typo, I meant HCR_EL2.IMO, my bad. > > > >>>> go directly to EL1/0. The virtual machines have complete access to > >>>> the gicc and receive IRQs without overhead. > >>>> As you have already pointed out, gicd is still emulated, so all the > >>>> operation of "enable/disable/set_tarrget" are slow as usual, and > >>>> still under control of the hypervisor. > >>> > >>> And so is inner-guest SGI (IPI) submission, I suppose. Just like on > >>> Intel (so far). > > > > Yes, everything that's gicd related is still handled by the > > hypervisor. The logic though is extremely simple, so the cost becomes > > almost the same as an empty hypercall. > > > >>>> > >>>> The problem is that IRQs are completely invisible to EL2, so Bao > >>>> needs to use FIQs for its internal functions. And to do that the > >>>> only way is to go through the secure monitor in EL3 for each > >>>> operation, having custom services running in the arm trusted firmware. > >>> > >>> Ah, interesting pattern: FIQ becomes the NMI that enables such a > >>> trick on Intel CPUs. If FIQ triggering architecturally limited to > >>> EL3, or is that SoC-specific? > > > > It's arm specific, FIQs are meant to be only used in the secure world, > > so EL3 and secure EL1. > > > >>> I'm not fully familiar with the ATF/TEE world yet, but your approach > >>> to model that service as SPD seems to block other use cases, like > >>> having a full TEE (OP-TEE). Wouldn't it rather make sense to model > >>> that as TEE app? > > > > Yes, it could be done as TEE app, but done like this shouldn't affect > > other use-cases, that we could of think of anyway... > > > >>>> > >>>> I've even thought of trying something similar in jailhouse, using > >>>> the linux driver that, running in EL1, could let jailhouse access > >>>> the IRQs even without the HMO bit set, and therefore allowing > >>>> direct IRQs access to inmates without the need to put a custom > >>>> service in the arm trusted firmware, but I found it a bit too intricate. > >>>> > >>> > >>> You can't use the guest to model management interrupts that are > >>> there to interrupt the guest and inform the hypervisor about a > >>> high-prio event (like "kill that guest"). > > > > True, and you have a lot of other problems having to rely on a guest > > for hypervisor tasks. > > > >> I started to read and think about this more - and then I suddenly > >> found SDEI [1]. Isn't our problem of having a non-IRQ event already > >> solved by registering on / sending some event 0? And SDEI is implemented > by ATF.
I am a bit lost (: How SDEI here would help interrupt inject? > >> > >> That would nicely overcome all the problems of the baod patch to ATF > >> (integration, security, missing gicv3 support, blocking of other > >> SPDs...). And even if I miss some catch in SDEI, the principle of > >> synthetic events defined there is what we want for hypervisor > >> management IPIs. > > > > Oh I didn't know about SDEI, it looks promising, but in the current > > state of the arm developer website I can't find any useful > > information, it doesn't even seem to be mentioned in the arm > > architecture reference manual. > > > >> The only downside: On a safety critical system, you may rather want > >> to get the firmware out of the picture during runtime, to reduce the > >> safety scope to a real minimum. But, IIRC, GICv4 will solve direct > >> interrupt injection in HW anyway. > > > > Yeah, in the GICv4 specifications there's the ability to directly > > inject the interrupts, but depending on the field it might be too long > > down the road for someone to wait. > > > > > > Forgot to share: > > - > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub. > com%2Fsiemens%2Fjailhouse%2Fcommits%2Fwip%2Farm64-zero-exits& > ;data=02%7C01%7Cpeng.fan%40nxp.com%7C6ecf1e2eb2714099a5ae08d82c > f63a2c%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637308783 > 336041620&sdata=4WqAL3pvGByyvJjQH3%2BfiLUHCOLEXa569nSUwFT > QiTQ%3D&reserved=0 > - > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgit.kiszk > a.org%2F%3Fp%3Darm-trusted-firmware.git%3Ba%3Dshortlog%3Bh%3Drefs > %2Fheads%2Fqueue&data=02%7C01%7Cpeng.fan%40nxp.com%7C6ecf > 1e2eb2714099a5ae08d82cf63a2c%7C686ea1d3bc2b4c6fa92cd99c5c301635 > %7C0%7C0%7C637308783336041620&sdata=BlleBLfrVIlHQ6Gn8MHN% > 2BWv2v4C2GxBiTDjXzPLaDq8%3D&reserved=0 > > All a bit hacky still but apparently no longer exploding. And without runtime > exists of the gic-demo. > > SDEI turned out to be more effort than expected because it is enabled only in > very few ATF targets so far. The queue above adds it to QEMU > (arm64) and ZynqMP (tested on Ultra96). TI-K3, RPi4, ESPRESSO/ > MACCHIATObin should be similarly extensible, and all that also in upstream - > at least as configurable feature of those platforms. I'll check i.MX platforms. Thanks, Peng. > > Jan > > -- > Siemens AG, Corporate Technology, CT RDA IOT SES-DE > Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/DB6PR0402MB2760B4463A03D7FC4EE3E4E988780%40DB6PR0402MB2760.eurprd04.prod.outlook.com.
