From: Dave Hansen <[email protected]> There is an existing variable (direct_gbpages) that says whether the kernel can and should use 1G pages in the direct map. It is driven by a bunch of other machinery. At least:
1. Hardware support for 1G pages 2. Kconfig support for 1G direct mappings 3. Kernel command line overrides Most code just checks the 'direct_gbpages' variable itself. But there are cases where 1G mappings are compile-time disabled (via X86_DIRECT_GBPAGES) and 'direct_gbpages' is always 0. Unfortunately, that constraint is invisible to the compiler. This opacity has been historically functionally harmless; it only leaves a bit of dead code. But, there are plans to tigthen up the compile-time checks around folded page table levels. The build will break if that dead code appears reachable to the compiler. Making the compile-time config visible to the compiler fixes the build. Add a helper to replace 'direct_gbpages' checks. Check the Kconfig option and base CPU support before looking at the variable. This lets the compiler optimize things better, especially collapse_pud_page() where most of the function can now be optimized out when the PUD level is folded. Notes: Use boot_cpu_has() instead of cpu_feature_enabled(). There's no required/disabled features for 1G pages themselves (X86_DIRECT_GBPAGES is for kernel mappings only) and the static_cpu_has() infrastructure is just gets in the compiler's way. This makes 64-bit build marginally larger (20 bytes in one compile) and 32-bit builds less marginally _smaller_ (~700 bytes). Signed-off-by: Dave Hansen <[email protected]> Reviewed-by: Yeoreum Yun <[email protected]> Tested-by: Yeoreum Yun <[email protected]> Link: https://lore.kernel.org/all/[email protected]/ [1] Reviewed-by: Sohil Mehta <[email protected]> --- arch/x86/include/asm/pgtable.h | 14 ++++++++++++++ arch/x86/kernel/cpu/common.c | 2 +- arch/x86/kernel/machine_kexec_64.c | 2 +- arch/x86/mm/init.c | 2 +- arch/x86/mm/pat/set_memory.c | 6 +++--- 5 files changed, 20 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index d5f4917c1edcb..c7adc00d49863 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1163,6 +1163,20 @@ static inline int pgd_none(pgd_t pgd) #ifndef __ASSEMBLER__ extern int direct_gbpages; +static inline bool direct_gbpages_enabled(void) +{ + /* Check the direct map config option: */ + if (!IS_ENABLED(CONFIG_X86_DIRECT_GBPAGES)) + return false; + + /* Check the CPU feature: */ + if (!boot_cpu_has(X86_FEATURE_GBPAGES)) + return false; + + /* Check the command-line and early setup variable: */ + return direct_gbpages; +} + void init_mem_mapping(void); void early_alloc_pgt_buf(void); void __init poking_init(void); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index c7352827f491d..4693ba98ca380 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2660,7 +2660,7 @@ void __init arch_cpu_finalize_init(void) * Right now we don't do that with gbpages because there seems * very little benefit for that case. */ - if (!direct_gbpages) + if (!direct_gbpages_enabled()) set_memory_4k((unsigned long)__va(0), 1); } else { fpu__init_check_bugs(); diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c index c3f4a389992da..0da0e89f2611a 100644 --- a/arch/x86/kernel/machine_kexec_64.c +++ b/arch/x86/kernel/machine_kexec_64.c @@ -257,7 +257,7 @@ static int init_pgtable(struct kimage *image, unsigned long control_page) info.kernpg_flag |= _PAGE_ENC; } - if (direct_gbpages) + if (direct_gbpages_enabled()) info.direct_gbpages = true; for (i = 0; i < nr_pfn_mapped; i++) { diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index 079f8c7e9e3cd..f0f4a06584c61 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -251,7 +251,7 @@ static void __init probe_page_size_mask(void) __default_kernel_pte_mask &= ~_PAGE_GLOBAL; /* Enable 1 GB linear kernel mappings if available: */ - if (direct_gbpages && boot_cpu_has(X86_FEATURE_GBPAGES)) { + if (direct_gbpages_enabled()) { printk(KERN_INFO "Using GB pages for direct mapping\n"); page_size_mask |= 1 << PG_LEVEL_1G; } else { diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index a1a061d995b31..261dda5f9f57c 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -128,7 +128,7 @@ void arch_report_meminfo(struct seq_file *m) seq_printf(m, "DirectMap4M: %8lu kB\n", direct_pages_count[PG_LEVEL_2M] << 12); #endif - if (direct_gbpages) + if (direct_gbpages_enabled()) seq_printf(m, "DirectMap1G: %8lu kB\n", direct_pages_count[PG_LEVEL_1G] << 20); } @@ -1315,7 +1315,7 @@ static int collapse_pud_page(pud_t *pud, unsigned long addr, pmd_t *pmd, first; int i; - if (!direct_gbpages) + if (!direct_gbpages_enabled()) return 0; addr &= PUD_MASK; @@ -1697,7 +1697,7 @@ static int populate_pud(struct cpa_data *cpa, unsigned long start, p4d_t *p4d, /* * Map everything starting from the Gb boundary, possibly with 1G pages */ - while (boot_cpu_has(X86_FEATURE_GBPAGES) && end - start >= PUD_SIZE) { + while (direct_gbpages_enabled() && end - start >= PUD_SIZE) { set_pud(pud, pud_mkhuge(pfn_pud(cpa->pfn, canon_pgprot(pud_pgprot)))); -- 2.43.0
