Both hugetlb and thp locate on the same migration type of pageblock, since they are allocated from a free_list[]. Based on this fact, it is enough to check on a single subpage to decide the migration type of the whole huge page. By this way, it saves (2M/4K - 1) times loop for pmd_huge on x86, similar on other archs.
Furthermore, when executing isolate_huge_page(), it avoid taking global hugetlb_lock many times, and meanless remove/add to the local link list cma_page_list. Signed-off-by: Pingfan Liu <[email protected]> Cc: Andrew Morton <[email protected]> Cc: Ira Weiny <[email protected]> Cc: Mike Rapoport <[email protected]> Cc: "Kirill A. Shutemov" <[email protected]> Cc: Thomas Gleixner <[email protected]> Cc: John Hubbard <[email protected]> Cc: "Aneesh Kumar K.V" <[email protected]> Cc: Christoph Hellwig <[email protected]> Cc: Keith Busch <[email protected]> Cc: Mike Kravetz <[email protected]> Cc: [email protected] --- mm/gup.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index ddde097..544f5de 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1342,19 +1342,22 @@ static long check_and_migrate_cma_pages(struct task_struct *tsk, LIST_HEAD(cma_page_list); check_again: - for (i = 0; i < nr_pages; i++) { + for (i = 0; i < nr_pages;) { + + struct page *head = compound_head(pages[i]); + long step = 1; + + if (PageCompound(head)) + step = compound_order(head) - (pages[i] - head); /* * If we get a page from the CMA zone, since we are going to * be pinning these entries, we might as well move them out * of the CMA zone if possible. */ - if (is_migrate_cma_page(pages[i])) { - - struct page *head = compound_head(pages[i]); - - if (PageHuge(head)) { + if (is_migrate_cma_page(head)) { + if (PageHuge(head)) isolate_huge_page(head, &cma_page_list); - } else { + else { if (!PageLRU(head) && drain_allow) { lru_add_drain_all(); drain_allow = false; @@ -1369,6 +1372,8 @@ static long check_and_migrate_cma_pages(struct task_struct *tsk, } } } + + i += step; } if (!list_empty(&cma_page_list)) { -- 2.7.5
