> Subject: Re: [PATCH 06/10] Add libbaremetal Since we come to discuss in a bigger picture, how do you suggest to proceed about the bootloader part?
Thanks, Peng. > > On 8/14/20 7:20 PM, Jan Kiszka wrote: > > On 14.08.20 19:06, Ralf Ramsauer wrote: > >> > >> > >> On 8/14/20 5:47 PM, Jan Kiszka wrote: > >>> On 14.08.20 17:06, Ralf Ramsauer wrote: > >>>> (+Cc Wolfgang, we recently had discussion on the boot process of > >>>> Jailhouse) > >>>> > >>>> On 8/14/20 11:43 AM, Jan Kiszka wrote: > >>>>> On 14.08.20 11:08, Peng Fan wrote: > >>>>>>> Subject: Re: [PATCH 06/10] Add libbaremetal > >>>>>>> > >>>>>>> On 07.08.20 05:06, [email protected] wrote: > >>>>>>>> From: Peng Fan <[email protected]> > >>>>>>>> > >>>>>>>> Add libbaremetal for sharing code between inmates and jailhouse > >>>>>>>> baremetal loader. > >>>>>>>> > >>>>>>>> The Makefile code is copied from inmates, currently only > >>>>>>>> string.c is moved from inmates to libbaremetal. In future, we > >>>>>>>> might need to share uart/mmu and etc. > >>>>>>> > >>>>>>> Might quickly become confusing to have two libs. What prevents > >>>>>>> renaming inmates/lib completely into libbaremetal? Sure, there > >>>>>>> are some jailhouse specifics in inmates/lib, but those could > >>>>>>> likely be put in some corner. > >>>>>> > >>>>>> How about rename inmates to baremetal? > >>>>>> And add bootloader stuff under baremetal? > >>>>> > >>>>> We could do > >>>>> > >>>>> baremetal > >>>> > >>>> What we provide is a minimalist runtime environment, which is not > >>>> just limited to baremetal applications (depends on the definition > >>>> of baremetal, though). But… > >>>> > >>>>> ├── demo-inmates > >>>>> ├── lib > >>>>> ├── tests > >>>>> └── tools > >>>>> > >>>>> and put the jailhouse loader under tools. Inmates is a "brand" in > >>>>> Jailhouse context, so it should remain in some form. But it's > >>>>> true, some > >>>> > >>>> … yep, the branding is the point why we should keep calling them > >>>> 'inmates'. > >>>> > >>>>> of the existing demo inmates can already run without jailhouse. > >>>> > >>>> And that's the second point - we already have some kind of support > >>>> for this kind of use case. However, I agree that I wouldn't expect > >>>> the preloader inside inmates/. > >>>> > >>>>> > >>>>> Ralf, what do you think? > >>>> > >>>> Peng, I just read the code of the loader and I try to understand > >>>> what how it's exactly working. Nice work! Do I see that correctly: > >>>> You basically "imitate" Linux in a way that you manually do the > >>>> early setup (some register setup, GIC init, kick off all CPUs, hv stubs, > >>>> ...). > >>>> > >>>> IOW, you bring the system to the same state, where the Linux driver > >>>> would hand over Jailhouse. You activate Jailhouse, we then return > >>>> to the loader which got lifted to the first cell, the root cell. > >>>> Now you use the loader to partition the system, i.e., to create > >>>> cells. Therefore, you use the hypercall interface, just like Linux > >>>> would typically do it for us. Finally, there's the hand over to > >>>> Linux. > >>>> > >>>> But do we really need a separate loader? Can't we just place the > >>>> loader-related stuff at, say, the end of jailhouse.bin? > >>>> > >>>> To be more concrete, i think it should be possible to have the full > >>>> preloader-thing located under hypervisor/arch/arm64/. It should > >>>> also be possible to link everything that is required into > >>>> jailhouse.bin - so we would get one universal binary instead of a > >>>> chain of dependent binaries. [1] > >>> > >>> Interesting idea. > >>> > >>>> > >>>> Pro: You can make early use of the UART in your preloader without > >>>> having the need to, for example, duplicate drivers. That's quite > >>>> useful for debugging if something goes wrong very early. Secondly, > >>>> you can also piggyback on existing gic routines, no need to > >>>> reimplement phys_processor_id, … > >>>> > >>>> The next thing after early boot would be the handover to jailhouse. > >>>> Before the handover, we set a flag that indicates that we do an > >>>> early boot of Jailhouse. We can use this flag to do the full > >>>> partitioning of the system before we return to the guests in one > >>>> single step. IOW, we never return to the preloader once the > >>>> handover happened. We can do that, as u-boot already placed our > >>>> guests and we just need to start them (as we would start regular > >>>> cells). > >>>> > >>>> The amount of additional logic in the actual hypervisor core should > >>>> be reasonably low. > >>>> > >>>> Did you already consider that strategy? Jan, what do you think? > >>> > >>> As we are already in visionary phase... There is another use case to > >>> consider: > >>> > >>> I'd like to have a Jailhouse variant that comes without a root cell. > >>> That means, we will need a booting procedure that creates all cells, > >>> loads and starts them - and then drops all the logic that could > >>> configure or destroy them (or never includes it in the first place - > >>> build-time setting). This should reduce the Jailhouse runtime code > >>> to much less than 10K LoC. > >> > >> Ok, that's indeed an awesome idea. > >> > >>> > >>> Peng's approach is useful in case you do want to keep the > >>> flexibility to reconfigure the system from a root cell, just like > >>> you can do when > >> > >> To be honest, I consider the "dynamic" reconfigurability of Jailhouse > >> as a development and testing utility rather than something that would > >> ever be useful for operative usage. This includes, of course, > >> jailhouse disabling. > > > > Reconfiguration, yes, but reloading can be a topic in certain scenarios. > > The current config locking mechanisms allows to keep this door for the > > root cell open, even while there is a critical cell running. > > > >> > >> What would be great would be a single-shot 'jailhouse freeze' (or, > >> ha, 'jailhouse detention') call, that throws away the whole hypercall > >> interface. Having such a thing, we an drop everything that is somehow > >> related to cell management: create, start, stop, destroy, stats, > >> disable, enable, ... > >> > >> And that would in fact be no big deal: Annotate all affected > >> routines, replace hyp vectors with stubs, drop the section, done. > >> Every code that has no other users than from a hypercall can be > >> dropped. Or do I miss something? > > > > I haven't thought this through in details yet, but I would be > > surprised if there weren't at least some smaller challenges in > > implementing that cleanly. Still, it's mostly about disabling code. > > > >> > >>> booting via Linux. The idea of adding a preloader is minimal > >>> invasive to the existing setup. My root-less mode will be maximal > >>> invasive, > though. > >> > >> Why do you think that it would be maximal invasive? > > > > As it changes the current hypervisor code for sure. Just loading > > Jailhouse via a different thing than Linux does not bring many > > changes, at least to the core. See this series. > > > >> > >> Oh btw: who would receive the freed memory? Would it remain in the > >> hypervisor or be assigned back to a cell? Though that should probably > >> only be a few pages. > > > > What freed memory? When things are only created, nothing will be freed > > anymore. There will also be no jailhouse disable. > > There will, e.g., be a cell_create routine, that will be called once per > cell. After > the creation of those cells, drop it, we won't need it again. > > If we have a universal binary, there will be a jailhouse disable will be > present > in the beginning. Once we decide freeze the configuration, drop it. If we make > those things compile-time configurable, then it has potential to become a > variant hell. > > > > >> > >>> If root-less would be sufficient for cases you do not want > >>> Linux-based boot, we can skip this approach and head for root-less full > steam. > >> > >> root-less means that the detention call (I like that) comes after > >> cell creation, but before cell start. .oO(we can even drop the > >> cell_start code, if we find a smart strategy to drop code right > >> before vmreturn) > >> > > > > I would rather envision some alternative configuration that contains > > all cell configs, not just the root cell (which would become a > > non-root cell as well). Loading would have to happen into the target > > memory regions before jailhouse enable, and the enabling would also > > imply starting all cells. > > > >>> > >>> If there are use cases for all three variant, we can try to look for > >>> common pieces in the two Linux-free options. My root-less mode would > >>> also need a single-step "create and start all cells", e.g. And your > >>> idea to include the bootstrap logic into an init section of > >>> jailhouse.bin is also attractive for both. > >> > >> Yes, I see the overlaps. By the way... Why don't we actually do: > >> > >> $ cat jh.bin sys.cell c1.cell c2.cell > jh.image > > Even better: > > cat preloader.bin jh.bin sys.cell c1.cell c2.cell > jh.image > > Again, the preloader.bin is still a binary image that only contains the > preloader > section, but u-boot can directly jump into it. It is still developed under > hypervisor/arch/foo and hence has full visibility of all symbols that are > available in regular jailhouse code and can potentially use them. We just > exploit the linker to export that section to a separate binary. > > If someone wants bare-metal boot mode, we just glue things together. > > >> > >> for the bare metal case? > > > > That comes close to what I have in mind, config-wise. sys.cell would > > still need a counter of cells that follow (simple to add to struct > > jailhouse_system). > > Not necessarily. We can also define a zero-termination. Once the first byte of > the JHCELL magic isn't present, we're done. But that's details. > > Ralf > > > > > Jan > > -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/DB6PR0402MB2760128F2AF78973F466B6B3885F0%40DB6PR0402MB2760.eurprd04.prod.outlook.com.
