We identified a cache invalidation problem on an ARM64 (Xilinx Zynq Ultrascale+ MPSoc, ARM v8) SoC. We have a two CPUs setup where one CPU runs Linux and the other runs a real time workload on the other CPU. The real time workload needs on average 60us, sometimes it gets up to 80us but it must not exceed 100us.
The Linux cell can extend the execution time of the real time workload in the other cell by starting stress-ng with --tlb-shootdown 2. The execution time rises to over 280us with a few spikes at around 350us. The problem is the cache flush/ invalidation which is performed locally and signal to the other CPUs in the same cacheable domain. This is required on SMP systems where the memory is shared between CPUs but can be omitted in setup where each CPU has exclusive memory. The here suggested solution is to boot Linux with HCR_EL2.FB set which forces a broadcast of each "local" flush/ invalidation instruction to the CPUs within the inner shareable. So __tlbi(vmalle1) is promoted to __tlbi(vmalle1is). Then the opcodes can be replaced to the local version while it still behaves as it before and notifies the CPU within the inner shareable. The second step is to clear HCR_EL2.FB once jailhouse ensures that only one CPU cells are started. Since each cell has exclusive memory access, the invalidation can be limited to the local CPU. As a result of the change, the stress-ng load in the Linux cell does no longer interfere with real time cell. This change has been tested since last December. The patches 1 - 3 are against jailhouse and basically set/ clear HCR_EL2.FB. Patch 4 is against Linux. Should #1 - #3 be accepted I don't see how #4 could be integrated upstream. The closest thing I could imagine is to hide it behind an alternative macro similar to X86's X86_FEATURE_UP for the lock prefix. Then patch it once the system goes to UP mode. Sebastian -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/20230602074808.1383333-1-bigeasy%40linutronix.de.
