A THP fault on a MADV_HUGEPAGE VMA first tries the local node only,
with __GFP_THISNODE | __GFP_NORETRY, and runs a single round of async
direct compaction there before falling back to other nodes. A failed
async run is never deferred.
When the local node has plenty of free memory but no free pageblock,
and what sits between the free pages can't be migrated, such as memory
long-term pinned for RDMA, each of these faults scans the zone again
and fails before falling back.
Defer the async state when an async run fails, and check it before
further async runs. Sync compaction keeps deferring on its own state,
and a successful compaction or allocation resets both. Reset the
pageblock skip hints only from sync compaction, as async relies on
them.
Populating a 4 GiB MADV_HUGEPAGE buffer from node 0 of a two-node VM,
with node 0 (24 GiB) fragmented by long-term pinning every other page,
median of 3 runs on each of two boots per kernel:
before after
direct compactions 649 / 1473 9 / 41
time in compaction 118.6 / 264.2 ms 2.6 / 5.1 ms
population time 531 / 504 ms 390 / 261 ms
THPs on node 0 1399 / 575 1478 / 727
With movable memory in place of the pins, compaction still succeeds
and both kernels get all 2048 THPs on node 0.
Signed-off-by: Qiliang Yuan <[email protected]>
---
mm/compaction.c | 24 ++++++++++++++++--------
1 file changed, 16 insertions(+), 8 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index e7a450b2be757..390171370b046 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -2600,9 +2600,12 @@ compact_zone(struct compact_control *cc, struct
capture_control *capc)
/*
* Clear pageblock skip if there were failures recently and compaction
- * is about to be retried after being deferred.
+ * is about to be retried after being deferred. Leave this to sync
+ * compaction: async compaction relies on the skip hints, and clearing
+ * them on its retries would rescan the whole zone each time.
*/
- if (compaction_restarting(cc->zone, cc->order))
+ if (cc->mode != MIGRATE_ASYNC &&
+ compaction_restarting(cc->zone, cc->order))
__reset_isolation_suitable(cc->zone);
/*
@@ -2858,8 +2861,10 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask,
unsigned int order,
!__cpuset_zone_allowed(zone, gfp_mask))
continue;
- if (prio > MIN_COMPACT_PRIORITY
- && compaction_deferred(zone, order,
true)) {
+ if (prio > MIN_COMPACT_PRIORITY &&
+ (compaction_deferred(zone, order, true) ||
+ (prio == COMPACT_PRIO_ASYNC &&
+ compaction_deferred(zone, order, false)))) {
rc = max_t(enum compact_result, COMPACT_DEFERRED, rc);
continue;
}
@@ -2890,14 +2895,17 @@ enum compact_result try_to_compact_pages(gfp_t
gfp_mask, unsigned int order,
break;
}
- if (prio != COMPACT_PRIO_ASYNC && (status == COMPACT_COMPLETE ||
- status == COMPACT_PARTIAL_SKIPPED))
+ if (status == COMPACT_COMPLETE ||
+ status == COMPACT_PARTIAL_SKIPPED)
/*
* We think that allocation won't succeed in this zone
* so we defer compaction there. If it ends up
- * succeeding after all, it will be reset.
+ * succeeding after all, it will be reset. Let a failed
+ * async run defer only further async runs, as sync
+ * compaction may succeed on pageblocks it skipped.
*/
- defer_compaction(zone, order, true);
+ defer_compaction(zone, order,
+ prio != COMPACT_PRIO_ASYNC);
/*
* We might have stopped compacting due to need_resched() in
--
2.43.0
