== Motivation
guest_memfd is KVM's guest-first memory provider that was created for
CoCo VMs and will support all VMs.
Today, guest_memfd only supports the equivalent of anonymous
PAGE_SIZE-d memfd memory, and has a lot to catch up to (HugeTLBfs,
tmpfs, DAX, etc) since almost anything can be mmap()-ed and handed to
KVM in a memslot.
I'd like to find a generic interface that would fit different
providers.
== Overview
Think of guest_memfd as a permissions (shared/private) and
KVM-know-how layer wrapping memory allocated from some provider.
This RFC illustrates how a resource pool fd can be handed to
guest_memfd to describe how it should allocate memory.
I believe this is extensible to many different kinds of memory
providers. :)
I'd like to discuss this at LPC 2026's KVM MC [1]. See y'all there!
== How userspace will use this feature
Do one of these,
resource_fd = fsmount();
resource_fd = open("/root/of/mount");
Then hand it to KVM_CREATE_GUEST_MEMFD,
gmem_fd = ioctl(KVM_CREATE_GUEST_MEMFD,
GUEST_MEMFD_FLAG_USE_RESOURCE, resource_fd);
And then use gmem_fd as before.
== How to become a provider?
Implement this in the provider's struct super_operations,
struct guest_memfd_provider_operations {
void *(*attach)(struct file *resource_file);
void (*release)(void *provider);
struct folio *(*alloc_folio)(void *provider, pgoff_t index,
struct mempolicy *mpol);
void (*invalidate_folio)(void *provider, struct folio *folio);
};
And the existence of these ops when guest_memfd accesses resource_fd
will determine if the provider is supported.
One requirement of this design is to allow new providers at runtime.
== Questions you might have
=== Why implement in struct super_operations?
It's one of the options to allow guest_memfd to find ops, and for an
RFC I think this requires the least new code and makes a simpler
illustration.
We would progressively add support for kernel-internal providers by
implementing guest_memfd_provider_operations, and loadable filesystem
modules can also implement the ops and be picked up at runtime.
Another option would be to have some kind of registry. Please see
changelog of patch 1 for discussion of options.
=== What if my provider doesn't deal with pages?
Frank and David Woodhouse [2] have use cases for providers that use
PFNs, and this is definitely something a generic interface must
support.
Here are some options I can think of:
1. Don't define .alloc_folio(), instead define .alloc_pfn()
2. Refactor .alloc_folio() to .alloc_pfn()
Either way, I think guest_memfd should still be the primary manager of
the memory.
=== Why have guest_memfd be the primary manager of the memory?
==== For folio-based providers
I propose we allocate the folios based on how the allocator was
configured:
+ e.g. for tmpfs,
+ Use mount page size.
+ Respect mount size limits.
+ Use guest_memfd mpol and fall back to mount mpol.
+ Gregory, now you can set mpol upfront on a mount! [3])
+ Use all the config that your provider has for allocation, reuse all
those interfaces that your provider already has to do the config so
guest_memfd doesn't have to reinvent those.
Once allocated, follow guest_memfd's rules on:
+ Tracking number of allocated bytes for this guest_memfd
+ memcg charging
+ Mapping into stage 2 page tables
+ Shared/private conversions
+ memory failure handling
+ etc
For the above split, I had the provider allocate, and then insert the
folio into guest_memfd's filemap.
I think of guest_memfd as the primary manager in the sense that
folio->mapping points to guest_memfd's filemap, which will allow
guest_memfd to handle memory failure by guest_memfd's rules.
Memory failure handling by guest_memfd's rules is something core to
guest_memfd. When guest_memfd supports huge pages, we'd like to be
able to split and only unmap the very page with a memory failure from
stage 2, which will keep guests running longer.
The provider can perform more handling of memory failure; we could
work out the ordering between guest_memfd and the provider.
Other than memory failure, there are other users that use
folio->mapping to determine what action to take. For now AFAICT
guest_memfd doesn't support those, but maybe it will need to in
future.
==== For PFN-based providers
For PFN-based providers, guest_memfd can be taught to store PFNs in
the xarray (like DAX does, perhaps?).
=== But my provider must be able to revoke memory from the guest!
I think that can be achieved with a bidirectional .attach(), where the
provider saves a pointer to the guest_memfd instance.
When memory must be revoked, it can tell the guest_memfd instance to
invalidate the page from everywhere. This has the nice benefit that
guest_memfd already respects the KVM MMU invalidation protocol, so the
provider doesn't have to learn the protocol.
== About this RFC
I chose tmpfs because I think it's the closest provider to the
existing anonymous PAGE_SIZE folios that guest_memfd supports.
The patches in this RFC was written with the help of AI and with many
edits from me.
This was built on top of Sean's kvm-x86 coco branch.
[1] https://lpc.events/event/20/contributions/2551/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
Signed-off-by: Ackerley Tng <[email protected]>
---
Ackerley Tng (17):
mm: shmem: Implement guest_memfd provider operations for tmpfs
KVM: guest_memfd: Support provider folio allocation
KVM: guest_memfd: Support provider folio invalidation
KVM: guest_memfd: Add helper to attach resource provider file
KVM: selftests: Add helper to create guest_memfd with a resource file
KVM: selftests: Test negative validation of resource_fd argument
KVM: selftests: Test rejection of unsupported filesystem for resource_fd
KVM: selftests: Test rejection of tmpfs file for resource_fd
KVM: selftests: Test rejection of swap tmpfs mounts
KVM: selftests: Test rejection of hugepage tmpfs mounts
KVM: selftests: Test guest_memfd with anonymous fsmount() tmpfs pool
KVM: selftests: Test guest_memfd with mounted tmpfs root directory
KVM: selftests: Test guest_memfd resource pool sharing across instances
KVM: selftests: Test memory allocation against shared tmpfs resource pool
KVM: selftests: Test that shared tmpfs resource pool size limit is
respected
KVM: selftests: Test guest execution with tmpfs-backed guest_memfd
KVM: selftests: Document testing TODOs
Documentation/virt/kvm/api.rst | 28 +-
include/linux/fs/super_types.h | 4 +
include/linux/guest_memfd.h | 29 ++
include/linux/kvm_host.h | 2 +-
include/uapi/linux/kvm.h | 5 +-
mm/shmem.c | 112 +++++++
tools/include/uapi/linux/kvm.h | 5 +-
tools/testing/selftests/kvm/guest_memfd_test.c | 404 +++++++++++++++++++++++--
tools/testing/selftests/kvm/include/kvm_util.h | 23 +-
virt/kvm/guest_memfd.c | 118 +++++++-
10 files changed, 673 insertions(+), 57 deletions(-)
---
base-commit: 37e0791600e5f1e2837a270c885296a172f5825e
change-id: 20260924-gmem-tmpfs-backend-075e990294a1
Best regards,
--
Ackerley Tng <[email protected]>
