On 02/03/21 21:47, Dov Murik wrote:
In order to allow OVMF to run the migration helper in parallel to the
guest OS, we introduce the notion of auxiliary vcpus, which are usable
for OVMF but are hidden from the guest OS. These might have other
future uses for in-guest operations/agents.
Hi Dov,
I think the helper approach to migration in general is great, but I'm
not sure I agree with the concept of auxiliary vCPUs. I would rather
have a completely separate VM file descriptor that does not even go
through the regular KVM run loop. Patches were posted recently to the
KVM mailing list to create secondary VMs sharing the encryption context
(ASID) with a primary VM.
When starting the VM, the firmware would proceed with attestation as
usual, detect it was running as a migration helper VM during the SEC
phase, and divert execution to the migration helper instead of
continuing with PEI.
The main advantage would be that the migration VM would not have to
share the address space with the primary VM. This would allow migrating
encrypted RAM areas that are not visible to the primary VM, for example
PCI BARs (those areas would be a problem for the kernel migration bitmap
though; I'll remark on that separately on Ashish's KVM series).
The VM would not even have an interrupt controller, so that HLT exits to
the host when it's done processing the mailbox. This would make it much
simpler to audit both the QEMU and the firmware sides.
Paolo
In the target VM we need the migration handler running to receive
incoming RAM pages; to achieve that, we boot the VM into OVMF with a
special fw_cfg value that causes OVMF to not boot the guest OS; we then
allow QEMU to receive an incoming migration by issuing a new
start-migrate-incoming QMP command.
The confidential RAM migration requires checking whether a given guest
RAM page is encrypted or not. This is currently achieved using AMD's
patches which track the encryption status of guest pages in KVM, using
hypercalls from OVMF and guest Linux to report changes of such status.
The QEMU side of these patches is included as the first two patches in
this series. The concrete implementation of this page encryption tracking
is currently in flux in the KVM mailing list, but the underlying
implementation doesn't affect our confidential RAM migration as long as
it can check whether a given guest address is encrypted.
List of patches in this series:
1-2: reposting AMD encrypted page bitmap support.
3-11: introduce the notion of auxiliary vcpus.
12-21: introduce the migration specifics.
22-23: fix devices issues when loading state into a live VM
24: introduce the start-migrate-incoming QMP command to switch the
target into accepting the incoming migration.
25: remove SEV migration blocker
26: add documentation
Brijesh Singh (1):
kvm: add support to sync the page encryption state bitmap
Dov Murik (21):
linux-headers: Add definitions of KVM page encryption bitmap ioctls
machine: Add auxcpus=N suboption to -smp
hw/boards: Add aux flag to CPUArchId
hw/i386: Mark auxiliary vcpus in possible_cpus
cpu: Add boolean aux field to CPUState
hw/i386: Set CPUState.aux=true for auxiliary vcpus
softmmu: Don't sync aux vcpus in pre_loadvm
softmmu: Add cpu_synchronize_without_aux_post_init
migration: Add helpers to save confidential RAM
migration: Add helpers to load confidential RAM
migration: Introduce gpa_inside_migration_helper_shared_area
migration: Save confidential guest RAM using migration helper
migration: Load confidential guest RAM using migration helper
migration: Stop VM after loading confidential RAM
migration: Don't sync vcpus when migrating confidential guests
migration: When starting target, don't sync auxiliary vcpus
hw/isa/lpc_ich9: Allow updating an already-running VM
target/i386: Re-sync kvm-clock after confidential guest migration
migration: Add start-migrate-incoming QMP command
target/i386: SEV: Allow migration unless there are no aux vcpus
docs: Add confidential guest live migration documentation
Tobin Feldman-Fitzthum (4):
hw/acpi: Don't include auxiliary vcpus in ACPI tables
softmmu: Add pause_all_vcpus_except_aux
migration: Stop non-aux vcpus before copying the last pages
migration: Call migration handler cleanup routines
docs/confidential-guest-live-migration.rst | 142 ++++++++++++
docs/confidential-guest-support.txt | 5 +
docs/index.rst | 1 +
qapi/migration.json | 26 +++
include/exec/ram_addr.h | 197 ++++++++++++++++
include/exec/ramblock.h | 3 +
include/exec/ramlist.h | 3 +-
include/hw/boards.h | 3 +
include/hw/core/cpu.h | 2 +
include/hw/i386/x86.h | 2 +-
include/sysemu/cpus.h | 2 +
linux-headers/linux/kvm.h | 13 ++
migration/confidential-ram.h | 23 ++
accel/kvm/kvm-all.c | 43 ++++
hw/acpi/cpu.c | 10 +
hw/core/cpu.c | 1 +
hw/core/machine.c | 7 +
hw/i386/acpi-build.c | 5 +
hw/i386/acpi-common.c | 5 +
hw/i386/pc.c | 7 +
hw/i386/x86.c | 10 +-
hw/isa/lpc_ich9.c | 3 +-
migration/confidential-ram.c | 258 +++++++++++++++++++++
migration/migration.c | 18 +-
migration/ram.c | 135 ++++++++++-
migration/savevm.c | 13 +-
softmmu/cpus.c | 68 +++++-
softmmu/runstate.c | 1 +
softmmu/vl.c | 3 +
target/i386/machine.c | 9 +
target/i386/sev.c | 25 +-
migration/meson.build | 6 +-
migration/trace-events | 4 +
33 files changed, 1027 insertions(+), 26 deletions(-)
create mode 100644 docs/confidential-guest-live-migration.rst
create mode 100644 migration/confidential-ram.h
create mode 100644 migration/confidential-ram.c
base-commit: 00d8ba9e0d62ea1c7459c25aeabf9c8bb7659462
