Ugh... So sorry, I forgot to turn off the chain-reply-to. resending V7 *facepalm*
On Wed, May 14, 2025 at 9:03 PM Nico Pache <npa...@redhat.com> wrote: > > The following series provides khugepaged and madvise collapse with the > capability to collapse anonymous memory regions to mTHPs. > > To achieve this we generalize the khugepaged functions to no longer depend > on PMD_ORDER. Then during the PMD scan, we keep track of chunks of pages > (defined by KHUGEPAGED_MTHP_MIN_ORDER) that are utilized. This info is > tracked using a bitmap. After the PMD scan is done, we do binary recursion > on the bitmap to find the optimal mTHP sizes for the PMD range. The > restriction on max_ptes_none is removed during the scan, to make sure we > account for the whole PMD range. When no mTHP size is enabled, the legacy > behavior of khugepaged is maintained. max_ptes_none will be scaled by the > attempted collapse order to determine how full a THP must be to be > eligible. If a mTHP collapse is attempted, but contains swapped out, or > shared pages, we dont perform the collapse. > > With the default max_ptes_none=511, the code should keep its most of its > original behavior. To exercise mTHP collapse we need to set > max_ptes_none<=255. With max_ptes_none > HPAGE_PMD_NR/2 you will > experience collapse "creep" and constantly promote mTHPs to the next > available size. This is due the fact that it will introduce at least 2x > the number of pages, and on a future scan will satisfy that condition once > again. > > Patch 1: Refactor/rename hpage_collapse > Patch 2: Some refactoring to combine madvise_collapse and khugepaged > Patch 3-5: Generalize khugepaged functions for arbitrary orders > Patch 6-9: The mTHP patches > Patch 10-11: Tracing/stats > Patch 12: Documentation > > --------- > Testing > --------- > - Built for x86_64, aarch64, ppc64le, and s390x > - selftests mm > - I created a test script that I used to push khugepaged to its limits > while monitoring a number of stats and tracepoints. The code is > available here[1] (Run in legacy mode for these changes and set mthp > sizes to inherit) > The summary from my testings was that there was no significant > regression noticed through this test. In some cases my changes had > better collapse latencies, and was able to scan more pages in the same > amount of time/work, but for the most part the results were consistent. > - redis testing. I tested these changes along with my defer changes > (see followup post for more details). > - some basic testing on 64k page size. > - lots of general use. > > V6 Changes: > - Dont release the anon_vma_lock early (like in the PMD case), as not all > pages are isolated. > - Define the PTE as null to avoid a uninitilized condition > - minor nits and newline cleanup > - make sure to unmap and unlock the pte for the swapin case > - change the revalidation to always check the PMD order (as this will make > sure that no other VMA spans it) > > V5 Changes [2]: > - switched the order of patches 1 and 2 > - fixed some edge cases on the unified madvise_collapse and khugepaged > - Explained the "creep" some more in the docs > - fix EXCEED_SHARED vs EXCEED_SWAP accounting issue > - fix potential highmem issue caused by a early unmap of the PTE > > V4 Changes: > - Rebased onto mm-unstable > - small changes to Documentation > > V3 Changes: > - corrected legacy behavior for khugepaged and madvise_collapse > - added proper mTHP stat tracking > - Minor changes to prevent a nested lock on non-split-lock arches > - Took Devs version of alloc_charge_folio as it has the proper stats > - Skip cases were trying to collapse to a lower order would still fail > - Fixed cases were the bitmap was not being updated properly > - Moved Documentation update to this series instead of the defer set > - Minor bugs discovered during testing and review > - Minor "nit" cleanup > > V2 Changes: > - Minor bug fixes discovered during review and testing > - removed dynamic allocations for bitmaps, and made them stack based > - Adjusted bitmap offset from u8 to u16 to support 64k pagesize. > - Updated trace events to include collapsing order info. > - Scaled max_ptes_none by order rather than scaling to a 0-100 scale. > - No longer require a chunk to be fully utilized before setting the bit. > Use the same max_ptes_none scaling principle to achieve this. > - Skip mTHP collapse that requires swapin or shared handling. This helps > prevent some of the "creep" that was discovered in v1. > > [1] - https://gitlab.com/npache/khugepaged_mthp_test > [2] - https://lore.kernel.org/all/20250428181218.85925-1-npa...@redhat.com/ > > Dev Jain (1): > khugepaged: generalize alloc_charge_folio() > > Nico Pache (11): > khugepaged: rename hpage_collapse_* to khugepaged_* > introduce khugepaged_collapse_single_pmd to unify khugepaged and > madvise_collapse > khugepaged: generalize hugepage_vma_revalidate for mTHP support > khugepaged: generalize __collapse_huge_page_* for mTHP support > khugepaged: introduce khugepaged_scan_bitmap for mTHP support > khugepaged: add mTHP support > khugepaged: skip collapsing mTHP to smaller orders > khugepaged: avoid unnecessary mTHP collapse attempts > khugepaged: improve tracepoints for mTHP orders > khugepaged: add per-order mTHP khugepaged stats > Documentation: mm: update the admin guide for mTHP collapse > > Documentation/admin-guide/mm/transhuge.rst | 14 +- > include/linux/huge_mm.h | 5 + > include/linux/khugepaged.h | 4 + > include/trace/events/huge_memory.h | 34 +- > mm/huge_memory.c | 11 + > mm/khugepaged.c | 472 ++++++++++++++------- > 6 files changed, 382 insertions(+), 158 deletions(-) > > -- > 2.49.0 >
