From: Mykola Kvach <mykola_kv...@epam.com> This is part 2 of version 5 of the ARM Xen system suspend/resume patch series, based on earlier work by Mirela Simonovic and Mykyta Poturai.
The first part is here: https://marc.info/?l=xen-devel&m=175659181415965&w=2 This version is ported to Xen master (4.21-unstable) and includes extensive improvements based on reviewer feedback. The patch series restructures code to improve robustness, maintainability, and implements system Suspend-to-RAM support on ARM64 hardware domains. At a high-level, this patch series provides: - Support for Host system suspend/resume via PSCI SYSTEM_SUSPEND (ARM64) - Suspend/resume infrastructure for CPU context, timers, GICv2/GICv3 and IPMMU-VMSA - Proper error propagation and recovery throughout the suspend/resume flow Key updates in this series: - Introduced architecture-specific suspend/resume infrastructure (new `suspend.c`, `suspend.h`, low-level context save/restore in `head.S`) - Integrated GICv2/GICv3 suspend and resume, including memory-backed context save/restore with error handling - Added time and IRQ suspend/resume hooks, ensuring correct timer/interrupt state across suspend cycles - Implemented proper PSCI SYSTEM_SUSPEND invocation and version checks - Improved state management and recovery in error cases during suspend/resume - Added support for IPMMU-VMSA context save/restore - Added support for GICv3 eSPI registers context save/restore --- TODOs: - Test system suspend with llc_coloring_enabled set and verify functionality - Implement SMMUv3 suspend/resume handlers - Enable "xl suspend" support on ARM - Properly disable Xen timer watchdog from relevant services (only init.d left) - Add suspend/resume CI test for ARM (QEMU if feasible) - Investigate feasibility and need for implementing system suspend on ARM32 --- Changelog for v6: - Add suspend/resume support for GICv3 eSPI registers (to be applied after the main eSPI series). - Drop redundant iommu_enabled check from host system suspend. - Switch from continue_hypercall_on_cpu to a dedicated tasklet for system suspend, avoiding user register modification and decoupling guest/system suspend status. - Refactor IOMMU register context code. - Improve IRQ handling: call handler->disable(), move system state checks, and skip IRQ release during suspend inside release_irq(). - Remove redundant GICv3 save/restore state logic now handled during vCPU context switch. - Clarify and unify error/warning messages, comments, and documentation. - Correct loops for saving/restoring priorities and merge loops where possible. - Add explicit error for unimplemented ITS suspend support. - Add missing GICD_CTLR_DS bit definition and clarify GICR_WAKER comments. - Cleanup active and enable registers before restoring. - Minor comment improvements and code cleanups. Changes introduced in V5: - Add support for IPMMU-VMSA context save/restore - Add support for GICv3 context save/restore - Select HAS_SYSTEM_SUSPEND in ARM_64 instead of ARM - Check llc_coloring_enabled instead of LLC_COLORING during the selection of HAS_SYSTEM_SUSPEND config - Call host_system_suspend from guest PSCI system suspend instead of arch_domain_shutdown, reducing the complexity of the new code Changes introduced in V4: - Remove the prior tasklet-based workaround in favor of a more straightforward and safer solution. - Rework the approach by adding explicit system state checks around request_irq and release_irq calls; skip these calls during suspend and resume states to avoid unsafe memory operations when IRQs are disabled. - Prevent reinitialization of local IRQ descriptors on system resume. - Restore the state of local IRQs during system resume for secondary CPUs. - Drop code for saving and restoring VCPU context (see part 1 of the patch series for details). - Remove IOMMU suspend and resume calls until these features are implemented. - Move system suspend logic to arch_domain_shutdown, invoked from domain_shutdown. - Add console_end_sync to the resume path after system suspend. - Drop unnecessary DAIF masking; interrupts are already masked on resume. - Remove leftover TLB flush instructions; flushing is handled in enable_mmu. - Avoid setting x19 in hyp_resume as it is not required. - Replace prepare_secondary_mm with set_init_ttbr, and call it from system_suspend. - Produce a build-time error for ARM32 when CONFIG_SYSTEM_SUSPEND is enabled. - Use register_t instead of uint64_t in the cpu_context structure. - Apply minor fixes such as renaming functions, updating comments, and modifying commit messages to accurately reflect the changes introduced by this patch series. For earlier changelogs, please refer to the previous cover letters. Previous versions: V1: https://marc.info/?l=xen-devel&m=154202231501850&w=2 V2: https://marc.info/?l=xen-devel&m=166514782207736&w=2 V3: https://lists.xen.org/archives/html/xen-devel/2025-03/msg00168.html Mirela Simonovic (6): xen/arm: Add suspend and resume timer helpers xen/arm: gic-v2: Implement GIC suspend/resume functions xen/arm: Implement PSCI SYSTEM_SUSPEND call (host interface) xen/arm: Resume memory management on Xen resume xen/arm: Save/restore context on suspend/resume xen/arm: Add support for system suspend triggered by hardware domain Mykola Kvach (5): xen/arm: gic-v3: Implement GICv3 suspend/resume functions xen/arm: Don't release IRQs on suspend xen/arm: irq: avoid local IRQ descriptors reinit on system resume xen/arm: irq: Restore state of local IRQs during system resume xen/arm: gic-v3: Add suspend/resume support for eSPI registers Oleksandr Tyshchenko (2): iommu/ipmmu-vmsa: Implement suspend/resume callbacks xen/arm: Suspend/resume IOMMU on Xen suspend/resume xen/arch/arm/Kconfig | 1 + xen/arch/arm/Makefile | 1 + xen/arch/arm/arm64/head.S | 112 +++++++++ xen/arch/arm/gic-v2.c | 143 +++++++++++ xen/arch/arm/gic-v3-lpi.c | 3 + xen/arch/arm/gic-v3.c | 288 +++++++++++++++++++++++ xen/arch/arm/gic.c | 32 +++ xen/arch/arm/include/asm/gic.h | 12 + xen/arch/arm/include/asm/gic_v3_defs.h | 1 + xen/arch/arm/include/asm/mm.h | 2 + xen/arch/arm/include/asm/psci.h | 1 + xen/arch/arm/include/asm/suspend.h | 46 ++++ xen/arch/arm/include/asm/time.h | 5 + xen/arch/arm/irq.c | 46 ++++ xen/arch/arm/mmu/smpboot.c | 2 +- xen/arch/arm/psci.c | 23 +- xen/arch/arm/suspend.c | 175 ++++++++++++++ xen/arch/arm/tee/ffa_notif.c | 2 +- xen/arch/arm/time.c | 49 +++- xen/arch/arm/vpsci.c | 9 +- xen/common/domain.c | 4 + xen/drivers/passthrough/arm/ipmmu-vmsa.c | 257 ++++++++++++++++++++ xen/drivers/passthrough/arm/smmu-v3.c | 10 + xen/drivers/passthrough/arm/smmu.c | 10 + 24 files changed, 1220 insertions(+), 14 deletions(-) create mode 100644 xen/arch/arm/include/asm/suspend.h create mode 100644 xen/arch/arm/suspend.c -- 2.48.1
