From: "Kiryl Shutsemau (Meta)" <[email protected]>
khugepaged collapses to mTHP orders since 7.2, and 7.3 added three
selftest cases for it: the generic collapse cases run at one order named
by -c, with the result detected by counting folios of that order.
That leaves the collapse path largely untested. The suite does not run
where a PMD is 512M. A folio count cannot say where a collapse landed.
Fixed sleeps cannot tell "not collapsed" from "not scanned yet". And
nothing exercises collapse under contention.
Close those gaps in order:
- Make the suite run at a 512M PMD: scale the collapse wait with the PMD
size, skip what such a PMD cannot serve, make the swapout the swap
cases depend on deterministic, and keep khugepaged out of the
MADV_COLLAPSE cases.
- Detect results per window rather than by count, with folio-order
helpers in vm_util that are checked against the kernel before any
collapse test trusts them.
- Drive khugepaged deterministically: a completion barrier that wakes the
daemon and waits for a full pass, and a check that one pass yields one
attributed collapse.
- Cover collapse at every supported order by default: which window
collapses, occupancy at both limits, sources that are already large
folios, and a fork-shared source under concurrent writes.
- Race collapse against everything that can touch its sources, at both
occupancy limits and over whole-table zaps, checked by content and by
the kernel's own assertions.
Everything passes on an unmodified kernel.
Changes since v5:
- Rebased onto mm-new: hugepage_settings lives in tools/lib/mm now, so
the new helpers and the three new binaries follow it there.
- The MADV_COLLAPSE cases set the per-order controls to "never" rather
than "inherit", which "mm: shmem: ignore sysfs configs for shmem
forced collapse" (in mm-new) made safe for shmem, and the Fixes tag
names the khugepaged commit that started the interference (Baolin
Wang).
- The "-c at or below -s" check moves from patch 14 to patch 12, where
-s first gains that meaning (Baolin Wang).
- Patch 08's changelog no longer claims check_huge_anon() reads smaps
(Baolin Wang).
- The two Sashiko fixups posted against v5 are folded in: the fork race
closes its pipe and checks waitpid(), and the race harness keeps its
mremap scratch address reserved.
- Baolin Wang's Reviewed-by on 01-05, 07, 10 and 14, Tested-by on 14.
Merging is_range_backed_by_order() with the check_huge_*() family stays a
follow-up, once Yeoreum's rewrite of those on top of pagemap and
kpageflags has landed.
Tested on mm-new 498ee28e5ec4 with KASAN, lockdep, DEBUG_VM and
page_table_check, in 16G guests:
x86-64/4K khugepaged all:all 104 pass, 11 skip, 0 fail
arm64/64K khugepaged all:all 102 pass, 3 skip, 0 fail
folio_order_check, khugepaged_sync_check and khugepaged_race pass on both.
The skips are structural: tmpfs cannot host some file cases, and mixed
sources has no order below the smallest. arm64 never registers the file
and shmem contexts, having no PMD-order page cache folio.
Range-diff against v5:
1: 759b73a5f738 ! 1: 14a0a1d44862 selftests/mm: raise the khugepaged
test-case cap
@@ Commit message
Acked-by: Usama Arif <[email protected]>
Acked-by: Lorenzo Stoakes (ARM) <[email protected]>
Reviewed-by: Mike Rapoport (Microsoft) <[email protected]>
+ Reviewed-by: Baolin Wang <[email protected]>
Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
## tools/testing/selftests/mm/khugepaged.c ##
2: 8160b5446e94 ! 2: 2265f32a677c selftests/mm: skip
collapse_compound_extreme() where the PMD is too large
@@ Commit message
Assisted-by: LLM
Acked-by: Lorenzo Stoakes (ARM) <[email protected]>
Reviewed-by: Mike Rapoport (Microsoft) <[email protected]>
+ Reviewed-by: Baolin Wang <[email protected]>
Tested-by: Muhammad Usama Anjum <[email protected]>
Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
3: c6a21ef8d798 ! 3: f14ae037a749 selftests/mm: scale khugepaged's collapse
wait with the PMD size
@@ Commit message
Assisted-by: LLM
Acked-by: Lorenzo Stoakes (ARM) <[email protected]>
Reviewed-by: Mike Rapoport (Microsoft) <[email protected]>
+ Reviewed-by: Baolin Wang <[email protected]>
Tested-by: Muhammad Usama Anjum <[email protected]>
Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
4: b039045f38b3 ! 4: cbe5dac4febc selftests/mm: skip khugepaged page cache
cases without a PMD folio
@@ Commit message
Assisted-by: LLM
Reviewed-by: Mike Rapoport (Microsoft) <[email protected]>
+ Reviewed-by: Baolin Wang <[email protected]>
Tested-by: Muhammad Usama Anjum <[email protected]>
Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
- ## tools/testing/selftests/mm/hugepage_settings.h ##
-@@ tools/testing/selftests/mm/hugepage_settings.h: void
thp_set_read_ahead_path(char *path);
+ ## tools/lib/mm/hugepage_settings.h ##
+@@ tools/lib/mm/hugepage_settings.h: void thp_set_read_ahead_path(char
*path);
unsigned long thp_supported_orders(void);
unsigned long thp_shmem_supported_orders(void);
5: 0ee82084bd2b ! 5: 021a1fcf711a selftests/mm: make the swap cases'
swapout reliable
@@ Commit message
Assisted-by: LLM
Reviewed-by: Muhammad Usama Anjum <[email protected]>
+ Reviewed-by: Baolin Wang <[email protected]>
Tested-by: Muhammad Usama Anjum <[email protected]>
Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
6: 4e48b431671e ! 6: e80ef0e8acc0 selftests/mm: stop khugepaged during the
MADV_COLLAPSE cases
@@ Commit message
collapses the very range the case is working on, and the case fails on
a
collapse that was interfered with rather than refused.
- Clear the per-order controls too, setting them to "inherit" rather than
- "never": khugepaged honours the global never and stays out, while
- MADV_COLLAPSE on shmem still finds an order to build.
+ Clear the per-order controls too. MADV_COLLAPSE does not consult them:
+ anon never did, and shmem stopped with "mm: shmem: ignore sysfs configs
+ for shmem forced collapse".
- Fixes: 9f0704eae8a4 ("selftests/mm/khugepaged: enlighten for
multi-size THP")
+ Fixes: b7f16963efe7 ("mm/khugepaged: run khugepaged for all orders")
Assisted-by: LLM
Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
@@ tools/testing/selftests/mm/khugepaged.c: static bool is_anon(struct
mem_ops *ops
ksft_print_msg("%s...", msg);
@@ tools/testing/selftests/mm/khugepaged.c: static void
__madvise_collapse(const char *msg, char *p, int nr_hpages,
- /*
- * Prevent khugepaged interference and tests that MADV_COLLAPSE
- * ignores /sys/kernel/mm/transparent_hugepage/enabled
-+ *
-+ * "inherit" rather than "never" so that MADV_COLLAPSE on shmem still
-+ * finds an order to build.
*/
settings.thp_enabled = THP_NEVER;
settings.shmem_enabled = SHMEM_NEVER;
+ for (i = 0; i < NR_ORDERS; i++) {
-+ settings.hugepages[i].enabled = THP_INHERIT;
-+ settings.shmem_hugepages[i].enabled = SHMEM_INHERIT;
++ settings.hugepages[i].enabled = THP_NEVER;
++ settings.shmem_hugepages[i].enabled = SHMEM_NEVER;
+ }
thp_push_settings(&settings);
7: 154a014dc5ad ! 7: e61798d22782 selftests/mm: move is_backed_by_folio()
into vm_util
@@ Commit message
Assisted-by: LLM
Acked-by: Mike Rapoport (Microsoft) <[email protected]>
Acked-by: Lorenzo Stoakes (ARM) <[email protected]>
+ Reviewed-by: Baolin Wang <[email protected]>
Tested-by: Muhammad Usama Anjum <[email protected]>
Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
@@ tools/testing/selftests/mm/split_huge_page_test.c: const char
*kpageflags_proc =
- return (pfn_flags & folio_tail_flags) != folio_tail_flags;
-fail:
- ksft_exit_fail_msg("Failed to get folio info\n");
-- return false;
-}
-
static int check_after_split_folio_orders(char *vaddr_start, size_t len,
@@ tools/testing/selftests/mm/vm_util.c: int pageflags_get(unsigned long
pfn, int k
+ return (pfn_flags & folio_tail_flags) != folio_tail_flags;
+fail:
+ ksft_exit_fail_msg("Failed to get folio info\n");
-+ return false;
+}
+
/* If `ioctls' non-NULL, the allowed ioctls will be returned into the var
*/
8: a23eab162641 ! 8: be5fb9342657 selftests/mm: add folio-order check for
address ranges
@@ Commit message
An mTHP collapse test needs to know that a range is backed by folios
of the
target order, and that they sit where a collapse would put them.
Nothing
- answers that today: is_backed_by_folio() classifies the folio behind a
- single page, and check_huge_anon() reads smaps AnonHugePages, which
only
- accounts PMD mappings.
+ answers both: is_backed_by_folio() classifies the folio behind a single
+ page, and check_huge_anon() counts the folios of an order in a range
+ without saying where they start.
Add is_range_backed_by_order(). It requires every folio-sized, folio-
aligned part of the range to map one folio of that order, head to tail,
@@ Commit message
## tools/testing/selftests/mm/vm_util.c ##
@@ tools/testing/selftests/mm/vm_util.c: bool is_backed_by_folio(char
*vaddr, int order, int pagemap_fd,
- return false;
+ ksft_exit_fail_msg("Failed to get folio info\n");
}
+/**
9: cd741198f81e ! 9: af0191f5548c selftests/mm: add folio-order detection
self-check
@@ tools/testing/selftests/mm/Makefile: TEST_GEN_FILES += guard-regions
TEST_GEN_FILES += rmap
TEST_GEN_FILES += folio_split_race_test
+TEST_GEN_FILES += folio_order_check
-
- ifneq ($(ARCH),arm64)
TEST_GEN_FILES += soft-dirty
+
+ ifeq ($(ARCH),x86_64)
## tools/testing/selftests/mm/folio_order_check.c (new) ##
@@
@@ tools/testing/selftests/mm/folio_order_check.c (new)
+
+#include "kselftest.h"
+#include "vm_util.h"
-+#include "hugepage_settings.h"
++#include <mm/hugepage_settings.h>
+
+static int pagemap_fd;
+static int kpageflags_fd;
@@ tools/testing/selftests/mm/run_vmtests.sh: CATEGORY="pfnmap" run_test
./pfnmap
## tools/testing/selftests/mm/vm_util.h ##
@@
- #include <linux/fs.h>
+ #include <mm/hugepage_settings.h>
#define BIT_ULL(nr) (1ULL << (nr))
+#define ALIGN(x, a) (((x) + (a) - 1) & ~((a) - 1))
10: 2099301ff2c1 ! 10: a105628f6e9b selftests/mm: add khugepaged completion
barrier helper
@@ Commit message
khugepaged knob only when its value changes.
Assisted-by: LLM
+ Reviewed-by: Baolin Wang <[email protected]>
Tested-by: Muhammad Usama Anjum <[email protected]>
Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
- ## tools/testing/selftests/mm/hugepage_settings.c ##
-@@ tools/testing/selftests/mm/hugepage_settings.c: void
thp_read_settings(struct thp_settings *settings)
+ ## tools/lib/mm/hugepage_settings.c ##
+@@ tools/lib/mm/hugepage_settings.c: void thp_read_settings(struct
thp_settings *settings)
}
}
@@ tools/testing/selftests/mm/hugepage_settings.c: void
thp_read_settings(struct th
void thp_write_settings(struct thp_settings *settings)
{
struct khugepaged_settings *khugepaged = &settings->khugepaged;
-@@ tools/testing/selftests/mm/hugepage_settings.c: void
thp_write_settings(struct thp_settings *settings)
+@@ tools/lib/mm/hugepage_settings.c: void thp_write_settings(struct
thp_settings *settings)
shmem_enabled_strings[settings->shmem_enabled]);
thp_write_num("use_zero_page", settings->use_zero_page);
@@ tools/testing/selftests/mm/hugepage_settings.c: void
thp_write_settings(struct t
+ thp_update_num("khugepaged/max_ptes_shared",
khugepaged->max_ptes_shared);
+ thp_update_num("khugepaged/pages_to_scan", khugepaged->pages_to_scan);
- if (dev_queue_read_ahead_path[0])
- write_num(dev_queue_read_ahead_path, settings->read_ahead_kb);
-@@ tools/testing/selftests/mm/hugepage_settings.c: void
thp_write_settings(struct thp_settings *settings)
+ if (dev_queue_read_ahead_path[0]) {
+ int ret = write_num(dev_queue_read_ahead_path,
+@@ tools/lib/mm/hugepage_settings.c: void thp_write_settings(struct
thp_settings *settings)
}
}
@@ tools/testing/selftests/mm/hugepage_settings.c: void
thp_write_settings(struct t
{
if (!settings_index) {
- ## tools/testing/selftests/mm/hugepage_settings.h ##
-@@ tools/testing/selftests/mm/hugepage_settings.h: static inline void
thp_save_settings(void)
+ ## tools/lib/mm/hugepage_settings.h ##
+@@ tools/lib/mm/hugepage_settings.h: static inline void
thp_save_settings(void)
hugepage_save_settings(/* thp = */ true, /* hugetlb = */ false);
}
11: 060448cb8a03 = 11: 27295f7decd1 selftests/mm: add order-parameterized
khugepaged collapse cases
12: 674fbf72a481 ! 12: 60e86026c227 selftests/mm: parameterize the
mixed-source collapse case by source order
@@ Commit message
supported mTHP order below the target. The other mTHP cases are
unaffected: mthp_push_target_order() enables only the target order.
+ A -c at or below -s is refused before any case runs: the sources would
+ already be the size being asked for. Without the check the generic
cases
+ fail on that one by one instead of saying why.
+
"-s 5 -c 7" on arm64/64K then collapses contpte-mapped sources into a
larger mTHP.
@@ tools/testing/selftests/mm/khugepaged.c: static void usage(void)
exit(1);
}
+@@ tools/testing/selftests/mm/khugepaged.c: int main(int argc, char **argv)
+ !(thp_supported_orders() & (1UL << collapse_order)))
+ ksft_exit_skip("Order %d is not a supported anon THP order\n",
+ collapse_order);
++ if (mthp_khugepaged_context && collapse_order <= anon_order)
++ ksft_exit_skip("-c %d needs a source order below it, -s says
%d\n",
++ collapse_order, anon_order);
+
+ if (mthp_khugepaged_context) {
+ pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
13: d2251908b73f ! 13: 39a895e66cf1 selftests/mm: cover a shared-source
collapse write race
@@ tools/testing/selftests/mm/khugepaged.c: static void
collapse_max_ptes_shared(st
+ close(sync[1]);
+ if (read(sync[0], &go, 1) != 1)
+ ksft_exit_fail_msg("child never reached the collapse\n");
++ close(sync[0]);
+
+ /*
+ * Unshare one page at a time: a burst would break CoW on the whole
@@ tools/testing/selftests/mm/khugepaged.c: static void
collapse_max_ptes_shared(st
+ */
+ i = 0;
+ for (;;) {
++ pid_t ret;
++
+ if (i < n)
+ ip[i * stride] = i + 0xbeef0000;
+ i++;
+ usleep(10 * 1000);
-+ if (waitpid(child, &wstatus, WNOHANG))
++ ret = waitpid(child, &wstatus, WNOHANG);
++ if (ret == child)
+ break;
++ if (ret < 0)
++ ksft_exit_fail_perror("waitpid()");
+ }
+
+ /* Finish whatever the paced sweep did not reach */
14: 30e316efbd9f ! 14: 8e214f9ef2ea selftests/mm: run every supported
collapse order by default
@@ Commit message
and "all". Also:
- -c still pins one order, and now says what is wrong instead of
- printing the usage text. An order at or below the -s source order
is
- skipped: the sources would already be the size being asked for.
+ printing the usage text.
- Both orders end up as array indices and shift counts, so -s and -c
are range-checked before they get there.
@@ Commit message
pinned order-4 line.
Assisted-by: LLM
+ Reviewed-by: Baolin Wang <[email protected]>
Tested-by: Muhammad Usama Anjum <[email protected]>
+ Tested-by: Baolin Wang <[email protected]>
Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
## tools/testing/selftests/mm/khugepaged.c ##
@@ tools/testing/selftests/mm/khugepaged.c: int main(int argc, char **argv)
- !(thp_supported_orders() & (1UL << collapse_order)))
- ksft_exit_skip("Order %d is not a supported anon THP order\n",
- collapse_order);
+- if (mthp_khugepaged_context && collapse_order <= anon_order)
+- ksft_exit_skip("-c %d needs a source order below it, -s says
%d\n",
+- collapse_order, anon_order);
+ if (mthp_khugepaged_context) {
+ unsigned long orders = thp_supported_orders();
+
15: e891ab107e3a ! 15: 95ae01ec50dd selftests/mm: check that one khugepaged
pass collapses one window
@@ tools/testing/selftests/mm/Makefile: TEST_GEN_FILES += merge
TEST_GEN_FILES += folio_split_race_test
TEST_GEN_FILES += folio_order_check
+TEST_GEN_FILES += khugepaged_sync_check
-
- ifneq ($(ARCH),arm64)
TEST_GEN_FILES += soft-dirty
+
+ ifeq ($(ARCH),x86_64)
## tools/testing/selftests/mm/khugepaged_sync_check.c (new) ##
@@
@@ tools/testing/selftests/mm/khugepaged_sync_check.c (new)
+
+#include "kselftest.h"
+#include "vm_util.h"
-+#include "hugepage_settings.h"
++#include <mm/hugepage_settings.h>
+
+#define BASE_ADDR ((void *)(1UL << 30))
+/* Smallest order khugepaged considers */
16: d7f8e59c19f1 ! 16: 343b8684a8bd selftests/mm: add khugepaged race harness
@@ tools/testing/selftests/mm/Makefile: TEST_GEN_FILES += rmap
TEST_GEN_FILES += folio_order_check
TEST_GEN_FILES += khugepaged_sync_check
+TEST_GEN_FILES += khugepaged_race
-
- ifneq ($(ARCH),arm64)
TEST_GEN_FILES += soft-dirty
+
+ ifeq ($(ARCH),x86_64)
## tools/testing/selftests/mm/khugepaged_race.c (new) ##
@@
@@ tools/testing/selftests/mm/khugepaged_race.c (new)
+
+#include "kselftest.h"
+#include "vm_util.h"
-+#include "hugepage_settings.h"
++#include <mm/hugepage_settings.h>
+#include "../../../../mm/gup_test.h"
+
+#ifndef FOLL_WRITE
@@ tools/testing/selftests/mm/khugepaged_race.c (new)
+ MREMAP_MAYMOVE | MREMAP_FIXED, mremap_area);
+ if (p == MAP_FAILED)
+ ksft_exit_fail_perror("mremap() back");
++ /* The move back unmapped the scratch address: claim it again */
++ if (mmap(mremap_scratch, hpage_pmd_size, PROT_NONE,
++ MAP_ANONYMOUS | MAP_PRIVATE | MAP_FIXED_NOREPLACE,
++ -1, 0) != (void *)mremap_scratch)
++ ksft_exit_fail_perror("mmap() mremap scratch");
+ usleep(rand_r(&seed) % 2000);
+ }
+ return NULL;
17: f85220ed8bb4 = 17: 09cfcb6afbc4 selftests/mm: race the collapse of
windows with holes
18: b3738ea322b7 = 18: b12f1f2d500c selftests/mm: add memory-pressure threads
to the khugepaged race harness
19: 66f5a400e015 = 19: ad0be4f7ec43 selftests/mm: zap whole PTE tables in the
khugepaged race harness
Kiryl Shutsemau (Meta) (19):
selftests/mm: raise the khugepaged test-case cap
selftests/mm: skip collapse_compound_extreme() where the PMD is too
large
selftests/mm: scale khugepaged's collapse wait with the PMD size
selftests/mm: skip khugepaged page cache cases without a PMD folio
selftests/mm: make the swap cases' swapout reliable
selftests/mm: stop khugepaged during the MADV_COLLAPSE cases
selftests/mm: move is_backed_by_folio() into vm_util
selftests/mm: add folio-order check for address ranges
selftests/mm: add folio-order detection self-check
selftests/mm: add khugepaged completion barrier helper
selftests/mm: add order-parameterized khugepaged collapse cases
selftests/mm: parameterize the mixed-source collapse case by source
order
selftests/mm: cover a shared-source collapse write race
selftests/mm: run every supported collapse order by default
selftests/mm: check that one khugepaged pass collapses one window
selftests/mm: add khugepaged race harness
selftests/mm: race the collapse of windows with holes
selftests/mm: add memory-pressure threads to the khugepaged race
harness
selftests/mm: zap whole PTE tables in the khugepaged race harness
tools/lib/mm/hugepage_settings.c | 60 +-
tools/lib/mm/hugepage_settings.h | 11 +
tools/testing/selftests/mm/Makefile | 3 +
.../testing/selftests/mm/folio_order_check.c | 122 ++++
tools/testing/selftests/mm/hmm-tests.c | 1 -
tools/testing/selftests/mm/khugepaged.c | 488 +++++++++++++++-
tools/testing/selftests/mm/khugepaged_race.c | 538 ++++++++++++++++++
.../selftests/mm/khugepaged_sync_check.c | 179 ++++++
tools/testing/selftests/mm/migration.c | 1 -
tools/testing/selftests/mm/run_vmtests.sh | 8 +-
.../selftests/mm/split_huge_page_test.c | 61 --
tools/testing/selftests/mm/vm_util.c | 146 +++++
tools/testing/selftests/mm/vm_util.h | 10 +
13 files changed, 1526 insertions(+), 102 deletions(-)
create mode 100644 tools/testing/selftests/mm/folio_order_check.c
create mode 100644 tools/testing/selftests/mm/khugepaged_race.c
create mode 100644 tools/testing/selftests/mm/khugepaged_sync_check.c
base-commit: 498ee28e5ec4727f829507c4a1bde3ab1b7704cd
--
2.54.0
