Hi Jan > Subject: RE: jailhouse jitter? > > > Subject: Re: jailhouse jitter? > > > > On 03.07.20 20:08, Angelo Ruocco wrote: > > > Adding José Martins in cc (direct gicv2 injection was his idea) as > > > he might be interested in this. > > > > > > On 02/07/2020, Jan Kiszka <[email protected]> wrote: > > >> On 02.07.20 18:45, Jan Kiszka wrote: > > >>> On 02.07.20 15:31, Angelo Ruocco wrote: > > >>>> On 02/07/2020, 'Nikhil Devshatwar' via Jailhouse > > >>>> <[email protected]> wrote: > > >>>>> On 02/07/20 5:14 pm, Jan Kiszka wrote: > > >>>>>> BTW, regarding direct interrupt delivery on ARM: In > > >>>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F > > >>>>>> %252 > > >>>>>> > > > Flwn.net%2FArticles%2F820830&data=02%7C01%7Cpeng.fan%40nxp.co > > >>>>>> > > > m%7C6ecf1e2eb2714099a5ae08d82cf63a2c%7C686ea1d3bc2b4c6fa92cd99 > > c5c > > >>>>>> > > > 301635%7C0%7C0%7C637308783336041620&sdata=PviVkkKPB8Klql5U > > z%2 > > >>>>>> B9rhZvAZxG00siR2yx%2F48P6kg4%3D&reserved=0, it is > > reported > > >>>>>> that Bao has "found a way to map interrupts directly into > > >>>>>> guests". I didn't find the time yet to check if that is > > >>>>>> actually exit-free delivery, and that as a smart trick or > > >>>>>> rather a problematic hack. Or if that sentence is rather a > misunderstanding. > > There is also the sentence: "Interrupts [...] have to be mediated > > through the hypervisor, which is unfortunate since that increases latency." > > >>>>>> > > >>>>> > > >>>>> I found this interesting and tried to read more about this. > > >>>>> I found some commits at [1] which does the direct injection. > > >>>>> Here is my rough understanding: > > >>>>> > > >>>>> * He is not setting the HCR_EL.FMO bit and that way all virtual > > >>>>> interrupts are turned off > > >>>>> * There is a new handler for handling "lower_el_aarch64_fiq" > > >>>>> which ends up being handled by the Hypervisor > > >>>>> * GICD is still being emulated so guests won't be able to route > > >>>>> interrupts to wrong CPUs. Isolation is maintained > > >>>>> * For triggering interrupts from Hypervisor (SGIs, etc) he is > > >>>>> using SMC calls and has a new service implemented in the ATF [2] > > >>>>> * I could not understand how the lower_el_aarch64_fiq exception > > >>>>> is fired so that the Hypervisor is invoked > > >>>>> > > >>>>> My guess is that most of the code change ihere s to enable > > >>>>> interrupts in the Hypervisor. Resetting HCR_EL2.FMO would send > > >>>>> the interrupts at EL1 directly. > > >>>> > > >>>> Yup, that's more or less it. > > >>>> > > >>>> Just one clarifications, Bao *does* set the HCR_EL2.FMO, it > > >>>> doesn't set the HCR_EL2.HMO. The HMO bit lets the interrupts > > >>>> through, and they > > >>> > > >>> HCR_EL2.HMO? On which revision of the ARMv8 spec are we looking? > > >>> Cannot find that in mine. > > > > > > Ugh, it's a typo, I meant HCR_EL2.IMO, my bad. > > > > > >>>> go directly to EL1/0. The virtual machines have complete access > > >>>> to the gicc and receive IRQs without overhead. > > >>>> As you have already pointed out, gicd is still emulated, so all > > >>>> the operation of "enable/disable/set_tarrget" are slow as usual, > > >>>> and still under control of the hypervisor. > > >>> > > >>> And so is inner-guest SGI (IPI) submission, I suppose. Just like > > >>> on Intel (so far). > > > > > > Yes, everything that's gicd related is still handled by the > > > hypervisor. The logic though is extremely simple, so the cost > > > becomes almost the same as an empty hypercall. > > > > > >>>> > > >>>> The problem is that IRQs are completely invisible to EL2, so Bao > > >>>> needs to use FIQs for its internal functions. And to do that the > > >>>> only way is to go through the secure monitor in EL3 for each > > >>>> operation, having custom services running in the arm trusted > firmware. > > >>> > > >>> Ah, interesting pattern: FIQ becomes the NMI that enables such a > > >>> trick on Intel CPUs. If FIQ triggering architecturally limited to > > >>> EL3, or is that SoC-specific? > > > > > > It's arm specific, FIQs are meant to be only used in the secure > > > world, so EL3 and secure EL1. > > > > > >>> I'm not fully familiar with the ATF/TEE world yet, but your > > >>> approach to model that service as SPD seems to block other use > > >>> cases, like having a full TEE (OP-TEE). Wouldn't it rather make > > >>> sense to model that as TEE app? > > > > > > Yes, it could be done as TEE app, but done like this shouldn't > > > affect other use-cases, that we could of think of anyway... > > > > > >>>> > > >>>> I've even thought of trying something similar in jailhouse, using > > >>>> the linux driver that, running in EL1, could let jailhouse access > > >>>> the IRQs even without the HMO bit set, and therefore allowing > > >>>> direct IRQs access to inmates without the need to put a custom > > >>>> service in the arm trusted firmware, but I found it a bit too > > >>>> intricate. > > >>>> > > >>> > > >>> You can't use the guest to model management interrupts that are > > >>> there to interrupt the guest and inform the hypervisor about a > > >>> high-prio event (like "kill that guest"). > > > > > > True, and you have a lot of other problems having to rely on a guest > > > for hypervisor tasks. > > > > > >> I started to read and think about this more - and then I suddenly > > >> found SDEI [1]. Isn't our problem of having a non-IRQ event already > > >> solved by registering on / sending some event 0? And SDEI is > > >> implemented > > by ATF. > > I am a bit lost (: How SDEI here would help interrupt inject? > > > >> > > >> That would nicely overcome all the problems of the baod patch to > > >> ATF (integration, security, missing gicv3 support, blocking of > > >> other SPDs...). And even if I miss some catch in SDEI, the > > >> principle of synthetic events defined there is what we want for > > >> hypervisor management IPIs. > > > > > > Oh I didn't know about SDEI, it looks promising, but in the current > > > state of the arm developer website I can't find any useful > > > information, it doesn't even seem to be mentioned in the arm > > > architecture reference manual. > > > > > >> The only downside: On a safety critical system, you may rather want > > >> to get the firmware out of the picture during runtime, to reduce > > >> the safety scope to a real minimum. But, IIRC, GICv4 will solve > > >> direct interrupt injection in HW anyway. > > > > > > Yeah, in the GICv4 specifications there's the ability to directly > > > inject the interrupts, but depending on the field it might be too > > > long down the road for someone to wait. > > > > > > > > > > Forgot to share: > > > > - > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub. > > > com%2Fsiemens%2Fjailhouse%2Fcommits%2Fwip%2Farm64-zero-exits& > > ;data=02%7C01%7Cpeng.fan%40nxp.com%7C6ecf1e2eb2714099a5ae08d8 > 2c > > > f63a2c%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637308783 > > > 336041620&sdata=4WqAL3pvGByyvJjQH3%2BfiLUHCOLEXa569nSUwFT > > QiTQ%3D&reserved=0 > > - > > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgit.k > > iszk > a.org%2F%3Fp%3Darm-trusted-firmware.git%3Ba%3Dshortlog%3Bh%3Drefs > > %2Fheads%2Fqueue&data=02%7C01%7Cpeng.fan%40nxp.com%7C6e > cf > > > 1e2eb2714099a5ae08d82cf63a2c%7C686ea1d3bc2b4c6fa92cd99c5c301635 > > %7C0%7C0%7C637308783336041620&sdata=BlleBLfrVIlHQ6Gn8MHN > % > > 2BWv2v4C2GxBiTDjXzPLaDq8%3D&reserved=0 > > > > All a bit hacky still but apparently no longer exploding. And without > > runtime exists of the gic-demo. > > > > SDEI turned out to be more effort than expected because it is enabled > > only in very few ATF targets so far. The queue above adds it to QEMU > > (arm64) and ZynqMP (tested on Ultra96). TI-K3, RPi4, ESPRESSO/ > > MACCHIATObin should be similarly extensible, and all that also in > > upstream - at least as configurable feature of those platforms. > > I'll check i.MX platforms.
I tested the SDEI on i.MX8MM, it shows the jitter became smaller. Without SDEI, the gic-demo jitter is 999ns+ With SDEI, the gic-demo jitter is 124ns~246ns. Indeed no more vmexits. But the max jitter, some times SDEI bigger only when program start up, mostly because of CACHE WARM UP I think. Will you move SDEI support to jailhouse mainline? Thanks, Peng. > > Thanks, > Peng. > > > > > Jan > > > > -- > > Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate > > Competence Center Embedded Linux > > -- > You received this message because you are subscribed to the Google Groups > "Jailhouse" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups > .google.com%2Fd%2Fmsgid%2Fjailhouse-dev%2FDB6PR0402MB2760B4463A > 03D7FC4EE3E4E988780%2540DB6PR0402MB2760.eurprd04.prod.outlook.co > m&data=02%7C01%7Cpeng.fan%40nxp.com%7C3a26236d791743a6f86 > 508d82d40f145%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63 > 7309104236572621&sdata=R2N6gPwNR%2FBnGLi1M6l1Yg%2F8U4mki > uRsWrxPW9PmE8Y%3D&reserved=0. -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/DB6PR0402MB2760EA5C808D0A49208E7A9788720%40DB6PR0402MB2760.eurprd04.prod.outlook.com.
