Mooncake, the KV cache store used with vLLM and SGLang, backs each rank's host segment with MADV_HUGEPAGE memory and registers it with the EFA NIC, which long-term pins it. Its commit c3fa13ecaf10 [1] reports:
"under the default local NUMA policy every THP fault on a full node runs direct compaction before falling back. With 8 ranks x 190GB this made registration take ~38 minutes." Mooncake now binds the segment MPOL_INTERLEAVE across all allowed nodes, so that no node fills before the others. That gives up node-local placement, and a task that sets its own memory policy still gets the stall. A THP fault on a MADV_HUGEPAGE VMA tries the local node first, with a single round of async direct compaction, before falling back to other nodes. A failed async run is never deferred, so when the free memory of the local node is scattered between pages that can't be migrated, such as long-term pins, every 2M fault scans the zone again and fails. A node that is simply full doesn't pay this: compaction_suitable() turns compaction away before it scans anything. Patch 1 splits the deferral state by migration mode. Patch 2 defers async compaction on its own state after it fails, leaving sync compaction as it is. Populating a 4 GiB MADV_HUGEPAGE buffer on a node fragmented by pinned memory, direct compactions drop from 649-1473 to 9-41 and the time spent in compaction from 118.6-264.2 ms to 2.6-5.1 ms, with no fewer THPs on the local node. Details are in patch 2. [1] https://github.com/kvcache-ai/Mooncake/commit/c3fa13ecaf1038df933dcb784ff8ffe026381100 Signed-off-by: Qiliang Yuan <[email protected]> --- V1 -> V2: - Describe the Mooncake report in the cover letter - Emit mm_compaction_defer_reset for both states (Sashiko) - Reset the pageblock skip hints only from sync compaction (Sashiko) - Rebase on v7.4-rc1 and rerun the tests v1: https://lore.kernel.org/r/20261001-bug-mm-thp-async-compact-defer-v1-0-0174c7923...@gmail.com --- Qiliang Yuan (2): mm/compaction: keep compaction deferral state per migration mode mm/compaction: defer failed async direct compaction include/linux/mmzone.h | 7 ++-- include/trace/events/compaction.h | 27 +++++++------ mm/compaction.c | 82 ++++++++++++++++++++++----------------- 3 files changed, 66 insertions(+), 50 deletions(-) --- base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4 change-id: 20261001-bug-mm-thp-async-compact-defer-b83351bfb25d Best regards, -- Qiliang Yuan <[email protected]>
