> > Dear Jan, dear Jailhouse community, > > > > We are running Jailhouse on an Intel X86 server-grade processor and use > the Jailhouse CAT module to isolate cache usage between cells. This patch > series contains several fixes to the CAT module, to ensure the root cell > remains isolated from non-root cells. > > I have tried to explain the grounds for change in each patch, with counter > examples where needed. > > > > Jan, I hope to extend the CAT module in the future to allow each cell to > change its CBM during run-time, e.g. to distinguish critical and non-critical > sections. Now, the module dynamically allocates each cell a COS, and > considers any write to MSR_IA32_L3_MASK an illegal VM exit. > > Right, the simpler model. > > > Disregarding the last patch in this series, each cell had its own unique > > COS, > which might justify a cell changing its own CBM, for example as long as it is > a > subset of the CBM in the config file. However, in the last patch of the > series I > drop this behavior due to issues on larger CPUs -- the number of CPUs might > outrange the number of COSes available. > > One COS -> one cell is so far the model. If you have one cell -> one CPU, you > may hit that limit, true. May I ask how large your systems are?
We are running the Xeon Scalable series. These packages come with 8-28 cores. Our system has access to 16 COSes, which should be sufficient for 11 bits in the CBM. I didn't take the opportunity to check to what extend the amount of COSes varies among the different processors in the series, but I this to be invariant. > > > > > Allowing a cell to cleanly edit its own CBM, without affecting other cells, > > is > difficult if the COSes are dynamically assigned at cell creation. That is why > I > am considering to extend the cache-regions in the config file to allow for a > statically allocated COS -- and make this a statically partitioned resource > just > like e.g. memory. > > What are your thoughts on this? Any reason why you omitted this in the > first place, and opted for dynamically allocated COSes? > > The "why" is generally best answered by "it's simpler to implement in the > hypervisor". Any moderation of hardware features add code to the > hypervisor. And an interface that could break or be misused if not done > properly > > So, you want multiple COS per cell in order to hand out sub-control over the > cell's CBM to it and to intra-cell partitioning? Sounds complex from the > hypervisor POV, though I understand that there can be use cases. I understand that disallowing modifications to the CBM is the simplest solution. Modifying CBMs properly while COSes are allocated dynamically as is the case now, will get complex very quickly. For example, should one cell COS be modified if another cell relying on the same COS wants to modify the corresponding CBM? If I understand correctly, I would expect it to be outside of Jailhouse's philosophy to start modifying COSes at runtime to accommodate cells changing their CBMs. Additionally, I find this undesirable as this would require hypervisor exits which incur additional latency. Ideally, all these resources are allocated statically, at cell creation. As an alternative, I would consider allocating COSes to a cell statically in the cell's config file. Then, the cell would be able to modify the CBMs corresponding to its own COSes, perhaps controlled by some additional flags. Anyway, I will look into the possibilities offered by VMCS. If I can find a workable solution which preserves the simplicity on the hypervisor side while allowing the cell some more control, I will propose this as a patch. You can then consider whether this solution would suit Jailhouse's simplicity and whether you'd like it included in the mainline. This might take a while though. > > > > > Thanks, > > > > Best regards, Bram Hooimeijer > > > > Bram Hooimeijer (6): > > x86/cat.c: Fix type freed_mask > > x86/cat.c: Fix CBM for non-root cell w/ root COS. > > x86/cat.c: Fix returning bits upon cell exit > > x86/cat.c: Fix off-by-one error > > x86/cat.c: Fix overlap on moving the root COS CBM > > x86/cat.c: Add COS re-use in cells with same mask. > > > > hypervisor/arch/x86/cat.c | 140 > > ++++++++++++++++++++++++++------------ > > 1 file changed, 95 insertions(+), 45 deletions(-) > > > > Thanks for these, need to dig into the details. > > I have locally an unfinished CAT rework pending that addresses L2 > partitioning as found on Apollo Lake and newer - or conceptually also multi- > socket L3 partitioning. For that, it resolves the wrong initial assumption > that > CAT is system-global. Baseline is a new CPU description, see [1]. All that > will > clash with these, but it's my job to resolve that. Thanks. If there is anything I can do to help on this, let me know. I will keep the new config files in mind with future patches. I only took a quick look, but I understand that Apollo Lake offers L2 partitioning only because it does not have an L3 cache. It might be worthwhile to abstract this to LLC (last level cache), if this relieves some of the complexity. As far as I know, there are no CPUs currently that offer both LLC and MLC (mid level cache) partitioning, though there are some Intel rumors suggesting this might be introduced in the future. This might introduce new complexity. For some thoughts on this, see this LWN patch [2]. Thanks, Bram [2] https://lwn.net/ml/linux-kernel/[email protected]/ > > Jan > > [1] > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu > b.com%2Fsiemens%2Fjailhouse%2Fcommits%2Fwip%2Fnew-cpu- > config&data=04%7C01%7Cbram.hooimeijer%40prodrive- > technologies.com%7C6559d5c239d143be055e08d8c79c15bf%7C612607c95af7 > 4e7f8976faf1ae77be60%7C0%7C0%7C637478820492030539%7CUnknown%7C > TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL > CJXVCI6Mn0%3D%7C1000&sdata=35z3GNSCx%2BeBq0sJvTt4tADM3IIW > UG1Evtf04JsiFIs%3D&reserved=0 > > -- > Siemens AG, T RDA IOT > Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/AS8PR02MB6663E803EF369EF2764E94A2B6B49%40AS8PR02MB6663.eurprd02.prod.outlook.com.
