On 20.09.26 21:36, Pasha Tatashin wrote:
> This RFC series is a PoC intended to showcase e2e working OrphanVMs
> uninterrupted continuing execution on preserved physical CPUs across a
> host kernel live update, prepared for the LPC'26 presentation at the
> KVM Microconf [1].
>
> The original design proposal discussion can be found at [2].
>
> This series is applied on top of v7.3-rc1 + the "[PATCH v4 00/11]
> liveupdate: kvm: Guest_memfd preservation" series [3]. The whole branch
> can be accessed at [4].
>
> It was tested on Intel Xeon Granite Rapids-AP, AMD EPYC Turin, and ARM
> Neoverse V2, as well as in emulation on QEMU and Intel Simics. However,
> it is still very early WIP, and is not anywhere near being production
> ready.
>
> Overview & Submission Strategy
> ==============================
> While this RFC bundles the e2e stack, it is NOT intended to be merged as
> a single series, the goal of this series is to have a discussion about
> the technical layers, and the challenges that need to be solved in order
> to have full Caretaker support within the kernel.
>
> Some of the arch/* changes duplicate small helpers (such as GICv3 list
> register accessors or low-level VMCS/VMCB/sysreg helpers) that could be
> shared with existing KVM/arch code if those routines and their callees
> are moved to static inlines or annotated with __cpu_preserved_text (and
> built without ftrace/KASAN/per-CPU references, as enforced by objtool
> and modpost). To avoid sprinkling __cpu_preserved_text annotations
> across core arch/* files in this PoC, those helpers are kept local to
> caretaker.c for now and can be consolidated when individual platform
> support is upstreamed.
>
> The series is structured into at least 8 workstreams that will be
> discussed and worked on separately:
>
> 1. Workstream 1: Preparation Patches (Patches 01-06)
> Architecture-neutral and arch-specific preparatory cleanups (in
> future these could be bundled with the workstreams that require these
> changes):
> 01. KVM: luo: Delegate VM creation type to kvm_arch_vm_luo_preserve
> 02. KVM: arm64: Split demux_c15_{get,set}_val from userspace
> accessors
> 03. KVM: arm64: Split kvm_sys_reg_{get,set}_user from kernel
> accessors
> 04. x86/mm/ident_map: Add force_pte to support 4K PTE identity
> mappings
> 05. arm64: mm: Add trans_pgd_map_range() support
> 06. x86/smp: Skip offline CPUs for REBOOT_VECTOR in
> native_stop_other_cpus()
>
> 2. Workstream 2: In-Kernel In-RAM vCPU State Preservation via LUO
> (Patches 07-10)
> Allows performing suspend/resume-type preservation across kexec
> without carrying the vCPU internal state in the VMM in userspace,
> keeping all vCPU state in-kernel:
> 07. KVM: luo: Support vCPU file preservation across live updates
> 08. KVM: x86: Add x86 vCPU LUO preservation ABI and register
> helpers
> 09. KVM: x86: Implement architectural vCPU state preservation via
> LUO
> 10. KVM: arm64: Implement architectural vCPU state preservation via
> LUO
>
> 3. Workstream 3: Physical CPU Preservation Core & x86 Support
> (Patches 11-24)
> Introduces the physical CPU preservation subsystem (cpu_preserve),
> linker sections (.text.cpu_preserved, .data.cpu_preserved) with
> build-time section isolation checks in modpost and objtool, isolated
> transition page table management (struct cpu_preserved_as), and sysfs
> interfaces. Allows booting the machine with some CPUs removed from
> the OS scheduler, and provides infrastructure for executing orphaned
> jobs on such CPUs:
> 11. liveupdate: Define CPU preservation linker sections
> 12. liveupdate: Add liveupdate_session_name() helper
> 13. cpu_preserve: Add physical CPU preservation ABI and core API
> headers
> 14. cpu_preserve: Add core physical CPU preservation state and park
> loop
> 15. cpu_preserve: Add physical CPU preservation lifecycle and build
> rules
> 16. liveupdate: cpu_preserve: Add sysfs interface
> 17. liveupdate: cpu_preserve: Add isolated address space management
> API
> 18. liveupdate: cpu_preserve: Add LUO file handler for preserved
> physical CPUs
> 19. x86: liveupdate: Add low-level physical CPU preservation
> assembly
> 20. x86: liveupdate: Add physical CPU preservation context and page
> table support
> 21. selftests: liveupdate: Add physical CPU preservation unit tests
> 22. selftests: liveupdate: Add physical CPU preservation live
> update tests
> 23. Documentation: liveupdate: Add physical CPU preservation
> documentation
> 24. MAINTAINERS: Add entry for KVM Caretaker
>
> 4. Workstream 4: ARM64 Support for Physical CPU Preservation (Patch 25)
> Enables physical CPU preservation on ARM64 using isolated transition
> page tables (trans_pgd_map_range()), cache maintenance to PoC, and
> GICv3 redistributor SGI wake helpers:
> 25. arm64: liveupdate: Add support for physical CPU preservation
>
> 5. Workstream 5: On-Core Scheduling Framework (oncore) (Patches 26-27)
> (Note: This layer sits between cpu_preserve and the KVM Caretaker and
> Kernel Caretaker (Future Work for Orphaned Processes))
> Provides a runqueue and time-slice scheduling framework
> (kernel/liveupdate/oncore.c) that multiplexes preserved workloads
> onto preserved physical CPUs during the kexec blackout window:
> 26. oncore: Add on-core KHO ABI and public framework headers
> 27. oncore: Implement on-core session lifecycle and scheduling loop
>
> 6. Workstream 6: KVM Caretaker Core & Intel VMX Support
> (Patches 28-39)
> Introduces the KVM Caretaker execution engine (virt/kvm/caretaker.c),
> vCPU detach/attach/cancel lifecycle, optional cross-kexec execution
> telemetry and debugfs reporting, x86 TDP MMU KHO page table
> preservation, x86 common Caretaker runtime (LAPIC timer injection,
> VM-exit dispatch, instruction decoding), and the Intel VMX Caretaker
> backend:
> 28. KVM: caretaker: Add Caretaker control block and architecture
> ops headers
> 29. KVM: caretaker: Implement Caretaker session memory mapping
> helpers
> 30. KVM: caretaker: Integrate Caretaker vCPU detach, attach, and
> cancel with KVM
> 31. KVM: caretaker: Add generic KHO ABI telemetry and debugfs
> reporting
> 32. KVM: x86: Add TDP MMU KHO preservation helpers
> 33. KVM: x86: Add Caretaker x86 KHO ABI and runtime context headers
> 34. KVM: x86: Implement Caretaker LAPIC timer and interrupt
> injection
> 35. KVM: x86: Implement Caretaker VM-exit dispatch and instruction
> decoders
> 36. KVM: x86: Implement Caretaker run loop and LUO detach/attach
> lifecycle
> 37. KVM: VMX: Add Caretaker VMX assembly guest entry/exit routine
> and helpers
> 38. KVM: VMX: Implement Caretaker VMX VMCS lifecycle and exit
> dispatch
> 39. KVM: VMX: Integrate Caretaker VMX detach serialization and KVM
> registration
>
> 7. Workstream 7: KVM Caretaker for AMD SVM (Patches 40-41)
> Implements the AMD SVM Caretaker backend (caretaker_vmenter.S, VMCB
> lifecycle, exit decoding, and next-RIP sync):
> 40. KVM: SVM: Add Caretaker SVM assembly guest entry/exit routine
> 41. KVM: SVM: Implement Caretaker SVM VMCB lifecycle and exit
> dispatch
>
> 8. Workstream 8: KVM Caretaker for ARM64 (Patches 42-46)
> Implements the ARM64 VHE Caretaker backend (caretaker_vmenter.S,
> EL2 exception vectors, GICv3 CPU interface & arch timer emulation,
> system register trap handling, and Stage-2 fault handling):
> 42. KVM: arm64: Add Caretaker arm64 KHO ABI and runtime context
> headers
> 43. KVM: arm64: Add Caretaker EL2 exception vectors and guest
> entry/exit assembly
> 44. KVM: arm64: Implement Caretaker GICv3 CPU interface and arch
> timer emulation
> 45. KVM: arm64: Implement Caretaker system register trap and
> exception handlers
> 46. KVM: arm64: Implement Caretaker vCPU run loop and LUO
> detach/attach lifecycle
>
> Architectural Design Layers
> ===========================
> The OrphanVM architecture is layered so that each subsystem has zero
> upward symbol dependencies:
>
> +--------------------------------------------------------------------+
> | Layer 4: KVM Caretaker Engine (virt/kvm/caretaker.c + arch/kvm/*) |
> | - VMENTER/VMRUN/ERET guest execution loop |
> | - Handling of timer preemption, HLT/WFI, MSR/sysreg traps |
> | - Yield on unhandled I/O or Stage-2 MMIO faults |
> +--------------------------------------------------------------------+
I had a superficial look at the code only, but it looked like we're not
reusing any of the existing KVM code for entry or exit handling. I was
hoping we could refactor some bits of KVM code into functionality that
is safe stand-alone and functionality that does require the larger
kernel ecosystem, then annotate the stand-alone ones as caretaker (btw,
abbreviating as kvmct would shorten your function names significantly
:)) and then consume them from both contexts.
Do you have any timing measurements for the approach as is?
Also, your patch set is cut off at 39/46. I was very curious to see how
you deal with vGIC and context switching in the caretaker environment :).
Alex
