Re: [PATCH 0/8] Add cache coloring support for Arm
On 22.04.20 10:51, Jan Kiszka wrote: On 22.04.20 09:22, Marco Solieri wrote: On Wed, Apr 22, 2020 at 08:42:32AM +0200, Jan Kiszka wrote: On 27.03.19 13:18, Marco Solieri wrote: Predictability of memory access latency is severely menaced by the multi-core architectures where the last level of cache (LLC) is shared, jeopardizing applicability of many Arm platform in real-time critical and mixed-criticality scenarios. Support for cache coloring is introduced, a transparent software technique allowing partitioning the LLC to avoid mutual interference between inmates. [...] Thanks for updating this! I will refresh my caches on the topic and provide feedback soon (I already have some questions and remarks but I'd like to double-check them). Looking forward to hear from you. Done with the deeper review. Overall, the series looks fairly good. I see just two bigger open issues: - inmate loading interface - more architectural independence But I think those should be solvable. As you likely read, there are better chances in sight to also address the root cell issue by booting Jailhouse from a loader. I share the same view. On the other hand, it ties the cache colouring with the Linux-independent boot. This is not ideal from an quality perspective, because it introduces a dependency between otherwise unrelated features, including one definitely optional (as long as Jailhouse will stay a "Linux-based hypervisor"). Also, from a process perspective, it forces the colouring-related activities and deliveries to be postponed after reaching a somewhat stable architecture for the independent loader (colouring pages is a loader matter). The other option is the hot-remapping of the root-cell memory, which we already wrote and tested on an older version of Jailhouse extended with a SMMU support. From a quality perspective, it looks comparable, and it does not introduces constraints on the development process. As pointed out back then, there are still open questions regarding the reliability of such a hot-remapping approach, besides the complexity. Anyway, we now do have SMMU support in Jailhouse (first issue to report against your series, patch 9 ;) ), we could look into that systematically. That would then leave us only with the question how to handle the hypervisor itself /wrt coloring. Correct. Provided that can buy us worthwhile improvements. We already have experimentally proven on two other hypervisors (Xen and Bao) that the interrupt response time hugely depends on the cache performances of the hypervisor's routines for guest injection. Cache partitioning is therefore mandatory for predictability. What measures did you apply on the hypervisors? Replicate the code into memory that has the "right" local color? Ensure that core- and guest-local data is in color memory? How did you handle naturally shared r/w data structures? These questions are still open. I don't see hypervisor coloring as a must-have for starting to merge things at all. But I'd like to have some idea of the structures and interfaces we merge will not cause massive breakages when adding that later. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/c25b626d-2aab-2ccd-f129-40e8b525a232%40siemens.com.
Re: [PATCH v2 9/9] Documentation: add description and usage of cache coloring support
On 21.04.20 12:03, 'Marco Solieri' via Jailhouse wrote: From: Luca Miccio Signed-off-by: Luca Miccio Signed-off-by: Marco Solieri --- Documentation/cache-coloring.md | 278 1 file changed, 278 insertions(+) create mode 100644 Documentation/cache-coloring.md diff --git a/Documentation/cache-coloring.md b/Documentation/cache-coloring.md new file mode 100644 index ..09aa2e17 --- /dev/null +++ b/Documentation/cache-coloring.md @@ -0,0 +1,278 @@ +Cache Coloring Support +== + +Introduction + + +### Cache partitioning and coloring + + Motivation + +Cache hierarchies of modern multi-core CPUs typically have first levels +dedicated +to each core (hence using multiple cache units), while the last level cache +(LLC) is shared among all of them. Such configuration implies that memory +operations on one core, e.g., running one Jailhouse inmate, are able to generate +timing *interference* on another core, e.g., hosting another inmate. More +specifically, data cached by the latter core can be evicted by cache store +operations performed by the former. In practice, this means that the memory +latency experienced by one core depends on the other cores (in-)activity. + +The obvious solution is to provide hardware mechanisms allowing either: a +fine-grained control with cache lock-down, as offered on the previous v7 +generation of Arm architectures; or a coarse-grained control with LLC +partitioning among different cores, as featured on the "Cache Allocation +Technology" of the high-end segment of recent Intel architecture and supported +by the Jailhouse hypervisor. + + Cache coloring + +Cache coloring is a *software technique* that permits LLC partitioning, +therefore eliminating mutual core interference, and thus guaranteeing higher and +more predictable performances for memory accesses. A given memory space in +central memory is partioned into subsets called colors, so that addresses in +different colors are necessarily cached in different LLC lines. On Arm +architectures, colors are easily defined by the following circular striding. + Is this striding architecturally defined? Across ARMv7 and v8? Or could implementation deviate as well? Do you happen to know how AMD or Intel do this, de facto or even in a documented way? At worst, the mapping algorithm would have to be factored out and implemented specifically, but not the logic to exploit this for cache partitioning. +``` + _ _ ___ _ _ _ _ + | | | | | | | + | c_0 | c_1 | | c_n | c_0 | c_1 | + _ _ _|_|_|_ _ _|_|_|_|_ _ _ + : : + '.. ' +. color 0 . +. ... . + : : +. ...: :. . +``` + +Cache coloring suffices to define separate domains that are guaranteed to be +*free from interference* with respect to the mutual evictions, but it does not +protect from minor interference effects still present on LLC shared +subcomponents (almost negligible), nor from the major source of contention +present in central memory. + +It is also worth remarking that cache coloring also partitions the central +memory availability accordingly to the color allocation--assigning, for +instance, half of the LLC size is possible if and only if half of the DRAM space +is assigned, too. + + +### Cache coloring in Jailhouse + +The *cache coloring support in Jailhouse* allows partitioning the cache by +simply partitioning the colors available on the specific platform, whose number +may vary depending on the specific cache implementation. More detail about color +availability and selection is provided in [Usage](#usage). + + Supported architectures + +Cache coloring is available on Arm64 architectures. In particular, extensive +testing has been performed on v8 CPUs, namely on the A53 and A57 processors +equipping Xilinx ZCU102 and ZCU104. + + Limitations + +Since Jailhouse is currently lacking SMMU support, and since the colored memory No longer true. We already have SMMUv3, and I will review the pending v2 patches soon. +mapping must be provided to DMA devices to allow them a coherent memory view, +coloring for this kind of devices is not available. +This also explain why also coloring support for the Linux root cell is not +provided, although possible and tested with a simple hot remapping procedure. I think this should be reworded so that it is made clear that dynamic reconfiguration of DMA mappings (or actually their activation) is not supported, and therefore also root cell coloring. + +### Further readings + +Relevance, applicability, and evaluation results of the Jailhouse cache coloring +support are reported in several recent works. A
Re: [PATCH v2 8/9] configs: add colored cell configuations for ZCU102
On 21.04.20 12:03, 'Marco Solieri' via Jailhouse wrote:
From: Luca Miccio
Signed-off-by: Luca Miccio
Signed-off-by: Marco Solieri
---
configs/arm64/zynqmp-zcu102-inmate-demo-col.c | 79 +++
configs/arm64/zynqmp-zcu102-linux-demo-col.c | 128 ++
2 files changed, 207 insertions(+)
create mode 100644 configs/arm64/zynqmp-zcu102-inmate-demo-col.c
create mode 100644 configs/arm64/zynqmp-zcu102-linux-demo-col.c
diff --git a/configs/arm64/zynqmp-zcu102-inmate-demo-col.c
b/configs/arm64/zynqmp-zcu102-inmate-demo-col.c
new file mode 100644
index ..83188b27
--- /dev/null
+++ b/configs/arm64/zynqmp-zcu102-inmate-demo-col.c
@@ -0,0 +1,79 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Configuration for demo inmate on Xilinx ZynqMP ZCU102 eval board:
+ * 1 CPU, 64K RAM, 1 serial port, color range [0-7]
+ *
+ * Copyright (c) Universita' degli Studi di Modena e Reggio Emilia 2020
+ *
+ * Authors:
+ * Luca Miccio
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include
+#include
+
+struct {
+ struct jailhouse_cell_desc cell;
+ __u64 cpus[1];
+ struct jailhouse_memory mem_regions[3];
+ struct jailhouse_cache cache_regions[1];
+} __attribute__((packed)) config = {
+ .cell = {
+ .signature = JAILHOUSE_CELL_DESC_SIGNATURE,
+ .revision = JAILHOUSE_CONFIG_REVISION,
+ .name = "inmate-demo-col",
+ .flags = JAILHOUSE_CELL_PASSIVE_COMMREG,
+
+ .cpu_set_size = sizeof(config.cpus),
+ .num_memory_regions = ARRAY_SIZE(config.mem_regions),
+ .num_irqchips = 0,
+ .num_pci_devices = 0,
+ .num_cache_regions = ARRAY_SIZE(config.cache_regions),
+
+ .console = {
+ .address = 0xff01,
+ .type = JAILHOUSE_CON_TYPE_XUARTPS,
+ .flags = JAILHOUSE_CON_ACCESS_MMIO |
+JAILHOUSE_CON_REGDIST_4,
+ },
+ },
+
+ .cpus = {
+ 0x2,
+ },
+
+ .mem_regions = {
+ /* UART */ {
+ .phys_start = 0xff01,
+ .virt_start = 0xff01,
+ .size = 0x1000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED,
+ },
+ /* RAM */ {
+ .phys_start = 0x80060,
+ .virt_start = 0,
+ .size = 0x0001,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE |
+ JAILHOUSE_MEM_COLORED,
+ },
+ /* communication region */ {
+ .virt_start = 0x8000,
+ .size = 0x1000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_COMM_REGION,
+ },
+ },
+
+ .cache_regions = {
+ {
+ .start = 0,
+ .size = 8,
+ },
+ },
+};
diff --git a/configs/arm64/zynqmp-zcu102-linux-demo-col.c
b/configs/arm64/zynqmp-zcu102-linux-demo-col.c
new file mode 100644
index ..66f12a66
--- /dev/null
+++ b/configs/arm64/zynqmp-zcu102-linux-demo-col.c
@@ -0,0 +1,128 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Configuration for linux-demo inmate on ZynqMP ZCU102:
+ * 2 CPUs, 128M RAM, serial port 2, color range [0-7]
+ *
+ * Copyright (c) Universita' degli Studi di Modena e Reggio Emilia 2020
+ *
+ * Authors:
+ * Luca Miccio
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include
+#include
+
+struct {
+ struct jailhouse_cell_desc cell;
+ __u64 cpus[1];
+ struct jailhouse_memory mem_regions[12];
+ struct jailhouse_cache cache_regions[1];
+ struct jailhouse_irqchip irqchips[1];
+ struct jailhouse_pci_device pci_devices[2];
+} __attribute__((packed)) config = {
+ .cell = {
+ .signature = JAILHOUSE_CELL_DESC_SIGNATURE,
+ .revision = JAILHOUSE_CONFIG_REVISION,
+ .name = "ZynqMP-linux-demo-col",
+ .flags = JAILHOUSE_CELL_PASSIVE_COMMREG,
+
+ .cpu_set_size = sizeof(config.cpus),
+ .num_memory_regions = ARRAY_SIZE(config.mem_regions),
+ .num_irqchips = ARRAY_SIZE(config.irqchips),
+ .num_pci_devices = ARRAY_SIZE(config.pci_devices),
+ .num_cache_regions = ARRAY_SIZE(config.cache_regions),
+ .vpci_irq_base =
Re: [PATCH v2 7/9] hypervisor, arm64: add cache coloring support
On 21.04.20 12:03, 'Marco Solieri' via Jailhouse wrote:
From: Luca Miccio
Implement all the functions needed by the coloring interface for the
arm64 architecture.
Coloring selection is retrieved by the jailhouse_cache structure(s) in
cell's configuration. Each structure defines a color range that will be
mapped to the corresponding color bitmask. The configuration is
cell-wide and will be used with all the memory regions flagged with
JAILHOUSE_MEM_COLORED.
If no color selection is provided by the user and coloring is enabled,
use all the available colors on the platform.
Signed-off-by: Luca Miccio
Signed-off-by: Marco Solieri
Acked-by: Angelo Ruocco
---
hypervisor/arch/arm64/Kbuild | 1 +
hypervisor/arch/arm64/coloring.c | 232 +++
2 files changed, 233 insertions(+)
create mode 100644 hypervisor/arch/arm64/coloring.c
diff --git a/hypervisor/arch/arm64/Kbuild b/hypervisor/arch/arm64/Kbuild
index c34b0f32..6c566e4d 100644
--- a/hypervisor/arch/arm64/Kbuild
+++ b/hypervisor/arch/arm64/Kbuild
@@ -22,3 +22,4 @@ always := lib.a
lib-y := $(common-objs-y)
lib-y += entry.o setup.o control.o mmio.o paging.o caches.o traps.o
lib-y += iommu.o smmu-v3.o ti-pvu.o
+lib-$(CONFIG_COLORING) += coloring.o
diff --git a/hypervisor/arch/arm64/coloring.c b/hypervisor/arch/arm64/coloring.c
new file mode 100644
index ..cb2d80e9
--- /dev/null
+++ b/hypervisor/arch/arm64/coloring.c
@@ -0,0 +1,232 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) Universita' di Modena e Reggio Emilia, 2020
+ *
+ * Authors:
+ * Luca Miccio
+ * Marco Solieri
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+#include
+#include
+#include
+#include
+#include
+
+#define for_each_cache_region(cache, config, counter) \
+ for ((cache) = jailhouse_cell_cache_regions(config), (counter) = 0;\
+(counter) < (config)->num_cache_regions; \
+(cache)++, (counter)++)
Should be factored out and made generally available. I'm carrying
something similar for Intel CAT (not published yet, so you couldn't know).
+
+/** Default color bitmask uses all available colors */
+unsigned long color_bitmask_default[COLOR_BITMASK_SIZE];
+
+/** Do care bits for coloring */
+unsigned long addr_col_mask;
+
+/** Max number of colors available on the platform */
+#define COLORING_MAX_NUM ((addr_col_mask >> PAGE_SHIFT) + 1)
+
+#define MSB_LONG_IDX(word) (word ? (BITS_PER_LONG - clz(word) - 1) : 0)
Add blank line.
+static inline unsigned long msb_color_bitmask(unsigned long *color_bitmask)
No need for inline.
+{
+ unsigned long ret = 0;u
+ unsigned int layer = COLOR_BITMASK_SIZE - 1;
+
+ if (!color_bitmask)
+ return 0;
+
+ while (!ret) {
+ ret = MSB_LONG_IDX(color_bitmask[layer]);
+ layer--;
+ }
+
+ return ret;
+}
+
+#define CTR_LINESIZE_MASK 0x7
+#define CTR_SIZE_SHIFT 13
+#define CTR_SIZE_MASK 0x3FFF
+#define CTR_SELECT_L2 (1 << 1)
+#define CTR_SELECT_L3 (1 << 2)
+#define CTR_CTYPEn_MASK0x7
+#define CTR_CTYPE2_SHIFT 3
+#define CTR_LLC_ON (1 << 2)
+#define CTR_LOC_SHIFT 24
+#define CTR_LOC_MASK 0x7
+#define CTR_LOC_NOT_IMPLEMENTED(1 << 0)
+
+unsigned long get_llc_way_size(void)
+{
+ unsigned int cache_sel;
+ unsigned int cache_global_info;
+ unsigned int cache_info;
+ unsigned int cache_line_size;
+ unsigned int cache_set_num;
+ unsigned int cache_sel_tmp;
+
+ arm_read_sysreg(CLIDR_EL1, cache_global_info);
+
+ /* Check if at least L2 is implemented */
+ if (((cache_global_info >> CTR_LOC_SHIFT) & CTR_LOC_MASK)
+ == CTR_LOC_NOT_IMPLEMENTED) {
+ printk("ERROR: L2 Cache not implemented\n");
+ return trace_error(-ENODEV);
+ }
+
+ /* Save old value of CSSELR_EL1 */
+ arm_read_sysreg(CSSELR_EL1, cache_sel_tmp);
+
+ /* Get LLC index */
+ if (((cache_global_info >> CTR_CTYPE2_SHIFT) & CTR_CTYPEn_MASK)
+ == CTR_LLC_ON)
+ cache_sel = CTR_SELECT_L2;
+ else
+ cache_sel = CTR_SELECT_L3;
+
+ /* Select the correct LLC in CSSELR_EL1 */
+ arm_write_sysreg(CSSELR_EL1, cache_sel);
+
+ /* Ensure write */
+ isb();
+
+ /* Get info about the LLC */
+ arm_read_sysreg(CCSIDR_EL1, cache_info);
+
+ /* ARM TRM: (Log2(Number of bytes in cache line)) - 4. */
+ cache_line_size = 1 << ((cache_info & CTR_LINESIZE_MASK) + 4);
+ /* ARM TRM: (Number of sets in cache) - 1 */
+ cache_set_num = ((cache_info >> CTR_SIZE_SHIFT) & CTR_SIZE_MASK) + 1;
+
+ /* Restore value in CSSELR_EL1 */
+ arm_write_sysreg(CSSELR_EL1, cache_sel_tmp);
+
+ /* Ensure
Re: [PATCH v2 5/9] hypervisor: introduce color_root_cell_management
On 21.04.20 12:03, 'Marco Solieri' via Jailhouse wrote: From: Luca Miccio The usual life cycle of an inmate is divided in 4 phases: creation, loading, starting and destruction. The most crucial phase for coloring is the loading one because Jailhouse relies on Linux and its driver module for loading binaries. In order to avoid exposing the coloring logic outside the hypervisor, use a separate memory for loading binaries to colored regions. During creation and destruction phases there is no functional change: the memory is mapped and unmapped using coloring functions as they should since we are handling colored regions. During loading phase a special colored mapping is created. The latter has the same physical start and coloring logic as the cell's memory but virtually starts at col_load_address. This will expose a virtually contiguous memory region that will be used by the driver to load binaries. During starting phase this special memory is destroyed. Add the missing CELL_CREATE state and introduce a management function that handles the colored scenario for all the above phases. Signed-off-by: Luca Miccio Signed-off-by: Marco Solieri --- hypervisor/control.c | 128 --- 1 file changed, 119 insertions(+), 9 deletions(-) Well, I already state my aversion, specifically due to these stats. ;) Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/b032e557-c190-3ef1-6ff6-3c93b2c97c6e%40siemens.com.
Re: [PATCH v2 3/9] driver: introduce col_load_address for colored binaries loading
On 21.04.20 12:03, 'Marco Solieri' via Jailhouse wrote: From: Luca Miccio Currently Jailhouse loads inmate's binaries by mapping its memory region to the root cell. When coloring is enabled this operation becomes not trivial. To avoid logic duplication in the driver module, use a special memory space when loading to colored regions. This convenient memory space starts from a fixed address defined by `col_loads_address` and will be mapped by the hypervisor using the same size and coloring configuration as the inmate. Since there could be platforms with different memory space layouts, the choice of `col_loads_address` has to be done accordingly. Allow the user to set this value in the root-cell configuration and set the default to 16 GiB. The latter has been empirically choosen as default value. This is better than the hypercall in v1, but I still dislike the approach for the complexity growth in the hypervisor. I see the point that some logic - namely the strip width calculation - would have to be duplicated into the Linux driver, but it feels to me that this would be a better alternative overall. Did you think through or even try such an approach at all? Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/c0710d88-a53b-c17e-8c1d-6615b9b8bf09%40siemens.com.
Re: ivhsmem_uio driver update
On 30/04/20 8:13 pm, Jan Kiszka wrote: On 30.04.20 15:32, 'Nikhil Devshatwar' via Jailhouse wrote: On 30/04/20 7:01 pm, 'Nikhil Devshatwar' via Jailhouse wrote: On 09/03/20 4:52 pm, 'Nikhil Devshatwar' via Jailhouse wrote: On 09/03/20 4:39 pm, Jan Kiszka wrote: On 09.03.20 11:12, Nikhil Devshatwar wrote: Hi Henning, As part of the TI SDK, we were integrating the ivhsmem_uio as external module from https://github.com/henning-schild-work/ivshmem-guest-code Since the ivshmem changes in the Jailhouse, that driver no more works (because of the vendor ID change) Do you plan to host the updated version of the driver in this repo? I know that the driver is already part of the siemens repo but it is much simpler to integrate an external module in yocto. Actually, that new driver is a complete rewrite, not just an ID update. It allows to securely pass access to unprivileged users and provides all features of the new ivshmem interface. With this driver, I can get the uio device registered but when I run the uio_send and uio_read apps, Using the uio_send and uio_read apps from https://github.com/henning-schild-work/ivshmem-guest-code/tree/master/uio/tests/Interrupts/VM Yeah, Henning already suggested to shut that repo down as people are still finding it... Hi Jan, Please follow the *updated* description in Documentation/inter-cell-communication.md regarding demos. I am now running the tools/ivshmem-demo.c application. I am getting failure to mmap the registers region because the default sizes for most of the ivshmem regions are multiple of 4096 When runnning a 64 bit kernel, mapping a 4k memory chunk in userspace fails with EINVAL error [1] How can this be fixed? [1] https://elixir.bootlin.com/linux/v5.4.38/source/drivers/uio/uio.c#L737 Regards, Nikhil D Jan -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/3c98f0a4-6067-8f00-0b8e-0ca15ede7ce3%40ti.com.
