On 8/27/26 10:18, Duan, Zhenzhong wrote:
>
>
>> -----Original Message-----
>> From: David Hildenbrand (Arm) <[email protected]>
>> Subject: Re: [RFCv2 PATCH 0/6] Support memory hotplug/unplug for TDX CoCo
>> guests
>>
>> On 6/23/26 12:17, Zhenzhong Duan wrote:
>>> This RFCv2 series implements comprehensive support for virtio-mem and ACPI
>>> DIMM memory hotplug/unplug in Intel TDX confidential computing guests.
>>> It explores the start-private memory approach utilizing the native
>>> TDG.MEM.PAGE.RELEASE API.
>>>
>>> We are seeking feedback from Kiryl on the CoCo guest implementation, MM
>>> experts on DIMM & virio-mem memory hotplug integration and broader
>>> virtio/CoCo community input on the overall approach. We are not seeking
>>> x86 maintainer review at this stage.
>>>
>>> == Changes from RFC v1 ==
>>>
>>> - Eliminated callback infrastructure: Dropped plug callback and replaced
>>> unplug callback with platform-level unaccept function into core MM
>>> hotplug and virtio-mem subsystems.
>>> - Added comprehensive bitmap tracking: Introduced a "plugged" bitmap
>>> alongside the unaccepted bitmap to track populated hotplug memory
>>> states to support load_unaligned_zeropad().
>>> - Enhanced SRAT parsing: Extended the EFI stub to parse ACPI SRAT tables
>>> early, ensuring hotpluggable ranges are tracked from initial boot.
>>>
>>> For more introduction about the background or other efforts in community,
>>> please check the RFCv1 cover letter [1].
>>>
>>> == Technical Approach ==
>>>
>>> - Early SRAT Integration: A lightweight EFI stub parser scans ACPI SRAT
>>> tables to identify hotpluggable ranges and adjust bitmap boundaries
>>> early, avoiding the overhead of the full ACPI subsystem.
>>> - Comprehensive Bitmap Tracking: Introduces a "plugged" bitmap right
>>> after the unaccepted bitmap. Both static and hotplugged memory are
>>> tracked, allowing the guest to map which ranges are populated by the
>>> VMM. This prevents acceptance beyond plugged memory boundaries due to
>>> load_unaligned_zeropad() operations.
>>> - Platform Extensibility: Exposes generic CoCo memory interfaces. Other
>>> confidential platforms (like AMD SEV-SNP) can easily adopt this by
>>> hooking their specific mechanisms into arch_unaccept_memory().
>>> - Hotplug & Guest Control: Integrates platform-level unaccept logic
>>> into ACPI hotplug and virtio-mem handlers. Uses TDG.MEM.PAGE.RELEASE
>>> for TDX to explicitly set memory to the "unaccepted" state during
>>> unplug, removing host hole-punching dependencies.
>>> - Kexec Handover: Leverages existing EFI mechanisms to seamlessly hand
>>> over both the extended unaccepted bitmap and the new plugged bitmap
>>> across kexec boundaries.
>>>
>>> == Testing ==
>>>
>>> - dimm and virtio-mem memory hotplug/unplug
>>> - lazy and eager accept
>>> - kexec/kdump with hotplugged memory
>>>
>>> This is tested with Marc-André Lureau's newest qemu series [2]
>>
>> What's the status of this?
>
> Marc's QEMU series is merged.
> For this series, following feedback from Kirill and Pratik, the preferred
> approach
> is updating the UEFI spec for hotplug memory ranges rather than parsing SRAT
> at the EFI stage. Pratik is already pushing this forward, I am currently
> waiting on
> his RFCs. If he hasn't taken over the entire implementation, I can rebase my
> remaining patches on top of his work.
>
> Hi Pratik, have you sent your UEFI RFC out yet? Just wanted to make sure
> I didn't miss your thread.
>
>>
>> I am still not sure whether we shouldn't perform acceptance from
>> move_pfn_range_to_zone() and from memory notifiers / generic_online_page.
>
> My understanding is that we already have full support for lazy and eager
> acceptance
> in generic_online_page() for static memory. We should be able to reuse that
> for
> hotplug memory and avoid adding acceptance logic in other places.
>
> All we need is extending unaccept_bitmap and adding new plugged_bitmap to
> support
> hotplug memory. I updated accept_memory() to check both bitmaps to determine
> which memory should be accepted.
The plugged bitmap is a very odd beast. I hate it, but I can see why it might
currently be required. I wonder if there is a better name for it because
"plugged"
is an overloaded term. What are the real semantics we want to express?
IIUC, unplug for virtio-mem requires prior conversion to shared memory. We
should have an intuitive mechanism for virtio-mem to just do the right thing
when unplugging memory (IOW, preparing for handback to the hypervisor).
>
>>
>> In particular, it's unclear to me how virtio-mem (which uses interfaces to
>> add/remove memory) interacts with unaccept_memory / coco bitmap.
>
> It works the same way as a physical DIMM: when memory is plugged, the
> corresponding bits in plugged_bitmap are set, and vice versa.
Well, no. When adding a Linux memory block through add_memory_resource() you do
coco_set_plugged_bitmap(). And in virtio_mem_send_plug_request() you do
coco_set_plugged_bitmap().
That's just super inconsistent and messy.
(coco_set_plugged_bitmap() and memory acceptance should *definitely not* be
open-coded like that in virtio_mem. There must be a clear abstraction layer with
clear, well documented semantics that virito-mem can iuse)
>
> Memory acceptance is already handled in generic_online_page(), so we
> do not need to do it inside virtio-mem. However, we do need to call
> unaccept_memory() during a memory unplug event in virtio-mem.
Again, I think we really need an abstraction that can just naturally be extended
for platforms that have to perform some work when returning memory to the
hypervisor. Open-coding x86's unaccept_memory() is not the way to go.
>
> Currently, tdx_unaccept_memory() can act as a no-op since QEMU handles
> hole-punching the private memory. That said, we still need to invoke
> unaccept_memory() to properly update the unaccept_bitmap bits.
>
>>
>> Can we have an overall design view on what happens at which stage when adding
>> /
>> removing memory through virtio-mem?
>
> I have put together a design view summary for virtio-mem below.
> Please let me know if this looks correct or if we should adjust the framing.
>
> Design Overview
> ---------------
> We maintain system stability and state safety using two metadata tracking
> layers during dynamic memory resizing operations:
> 1. plugged_bitmap: Explicitly tracks blocks plugged into the guest.
> This protects load_unaligned_zeropad() from reading omitted memory
> holes, preventing catastrophic guest crashes.
I hate load_unaligned_zeropad() so much at this point. We should finally rip it
out. I wish I would have more spare time to look into that.
The plugged bitmap is a clear sign that load_unaligned_zeropad() just has to go
instead of us hacking around it.
> 2. unaccept_bitmap: Explicitly tracks the secure page initialization state.
I didn't fully grasp the level of hackery we have to apply to make
load_unaligned_zeropad() not do stupid things.
Am I correct that we have to accept more memory, possibly falling into unplugged
virtio-mem ranges? What is the effect of that?
>
> Step-by-Step Lifecycle Stages
> -----------------------------
> Using sub-block hotplug of a new memory block with eager acceptance
> as an example:
>
> 1. Memory Addition (Plug) Stage
> a. Host notifies guest -> virtio-mem driver handles the plug event.
> b. Driver marks the allocated memory ranges in 'plugged_bitmap' by
> calling coco_set_plugged_bitmap(addr, size, true). During this stage,
> a plug request is also sent to the VMM.
> c. Driver adds memory blocks via add_memory_resource().
Assume you hotplug a single device block (e.g., 2M). virtio-mem will set the
plugged bitmap of that one block. But add_memory_resource() will set the plugged
bitmap of the entire Linux memory block.
That seems completely broken?
--
Cheers,
David
