On 26.10.20 18:33, Vlastimil Babka wrote:
Commit 11c9c7edae06 ("mm/page_poison.c: replace bool variable with static key")
changed page_poisoning_enabled() to a static key check. However, the function
is not inlined, so each check still involves a function call with overhead not
eliminated when page poisoning is disabled.
Analogically to how debug_pagealloc is handled, this patch converts
page_poisoning_enabled() back to boolean check, and introduces
page_poisoning_enabled_static() for fast paths. Both functions are inlined.
Also optimize the check that enables page poisoning instead of debug_pagealloc
for architectures without proper debug_pagealloc support. Move the check to
init_mem_debugging() to enable a single static key instead of having two
static branches in page_poisoning_enabled_static().
Signed-off-by: Vlastimil Babka <[email protected]>
[...]
+/*
+ * For use in fast paths after init_mem_debugging() has run, or when a
+ * false negative result is not harmful when called too early.
+ */
+static inline bool page_poisoning_enabled_static(void)
+{
+ return (static_branch_unlikely(&_page_poisoning_enabled));
return static_branch_unlikely(&_page_poisoning_enabled);
+}
#else
static inline bool page_poisoning_enabled(void) { return false; }
+static inline bool page_poisoning_enabled_static(void) { return false; }
static inline void kernel_poison_pages(struct page *page, int numpages,
int enable) { }
#endif
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index b168c58ef337..2a1be197649d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -777,6 +777,17 @@ void init_mem_debugging()
}
}
+#ifdef CONFIG_PAGE_POISONING
+ /*
+ * Page poisoning is debug page alloc for some arches. If
+ * either of those options are enabled, enable poisoning.
+ */
+ if (page_poisoning_enabled() ||
+ (!IS_ENABLED(CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC) &&
+ debug_pagealloc_enabled()))
Weird indentation
if (page_poisoning_enabled() ||
(!IS_ENABLED(CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC) &&
debug_pagealloc_enabled()))
+ static_branch_enable(&_page_poisoning_enabled);
+#endif
+
#ifdef CONFIG_DEBUG_PAGEALLOC
if (!debug_pagealloc_enabled())
return;
@@ -2208,7 +2219,7 @@ static inline int check_new_page(struct page *page)
static inline bool free_pages_prezeroed(void)
{
return (IS_ENABLED(CONFIG_PAGE_POISONING_ZERO) &&
- page_poisoning_enabled()) || want_init_on_free();
+ page_poisoning_enabled_static()) || want_init_on_free();
}
Apart from that LGTM.
--
Thanks,
David / dhildenb
