virtio-mem adds memory without MHP_MEMMAP_ON_MEMORY. With the memory
onlined to ZONE_MOVABLE, a VM that grows to many times its boot size
can run short of unmovable memory. I ran into this while working on
a guest-initiated memory negotiation project.
David wrote last year that memmap_on_memory support for virtio-mem had
not been worth the effort, because virtio-mem adds memory in small
blocks [1]. Each block is small, yes, but the memmap adds up when
scaling to many blocks.
This picks up Hui Zhu's RFC from 2021 [2] and follows David's review of
it [3][4]:
1. Big Block Mode first (patch 3). Since commit 6b8f0798b85a
("mm/memory_hotplug: split memmap_on_memory requests across
memblocks"), the core uses one altmap per memory block, so this also
works for big blocks spanning several memory blocks.
2. alloc_contig_range() with pageblock granularity, so subblocks can be
2 MiB: done since v5.19 (Zi Yan's series, 448b8ec3bf11).
3. Sub Block Mode only if the memmap spans complete subblocks (patch 5).
With larger device blocks, the driver warns and continues without
memmap_on_memory, as David suggested in 2023 [5].
The other review points on [2]:
- Plugging more than requested: if no more than the memmap subblocks
remain to be plugged, the driver stops short of the requested size, so
this doesn't happen.
- Error handling: if unplugging the memmap subblocks of a removed memory
block fails, the block is left in VIRTIO_MEM_SBM_MB_PLUGGED and
virtio_mem_cleanup_pending_mb() retries.
Georgi, Cc'ing you as you mentioned out-of-tree virtio-mem patches that
use memmap_on_memory [6].
Tested in QEMU/KVM (x86-64 guests) on v7.3: 128 MiB, 1 GiB
and 2 GiB memory blocks; device block sizes 2 MiB to 1 GiB;
ZONE_MOVABLE, kernel zones and offline blocks; force_bbm;
memory_hotplug.memmap_on_memory=force on a kernel without THP;
grow/fill/unplug cycles with memory held during unplug; 600
offline/online cycles from user space; random resizing under a memory
hog; injected failures when unplugging memmap subblocks; the
memory-hotplug selftest; virtio-mem as a module, reloaded. Also with
KASAN, lockdep, and DEBUG_VM. Built for x86-64, and the changed files
for arm64, riscv, and s390.
An LLM was used to review the series against the core, write the test
harness, and draft the cover letter and changelogs. I have reviewed all
of it.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3]
https://lore.kernel.org/all/CADFyXm7Ky+S2gq5ht2h_tX9bLsoOSuctCkG==vyah0uuhgp...@mail.gmail.com/
[4] https://lore.kernel.org/all/[email protected]/
[5] https://lore.kernel.org/all/[email protected]/
[6]
https://lore.kernel.org/all/[email protected]/
Signed-off-by: Nikolaos Karaolidis <[email protected]>
---
Nikolaos Karaolidis (6):
mm/memory_hotplug: add and export mhp_memmap_on_memory_nr_pages()
virtio-mem: prepare memory notifier for a self-hosted memmap
virtio-mem: support memmap_on_memory in Big Block Mode
virtio-mem: move virtio_mem_sbm_try_remove_unplugged_mb()
virtio-mem: support memmap_on_memory in Sub Block Mode
Documentation/admin-guide/mm: document virtio-mem memmap_on_memory
Documentation/admin-guide/mm/memory-hotplug.rst | 18 ++
drivers/virtio/virtio_mem.c | 268 +++++++++++++++++-------
include/linux/memory_hotplug.h | 6 +
mm/memory_hotplug.c | 15 ++
4 files changed, 236 insertions(+), 71 deletions(-)
---
base-commit: c03c1a14507dfaf8b856c2608f9ac216021fc222
change-id: 20261103-virtio-mem-memmap-f3ec7382772d
Best regards,
--
Nikolaos Karaolidis <[email protected]>
