uaccess_without_mm() is inlined into every user copy, including every
copy_{to,from}_user() call site on architectures that define
INLINE_COPY_USER, such as arm64. Each inlined WARN_ON_ONCE() adds its
own __bug_table entry.
Keep the NULL current->mm check inline, but move the warning into a
single out-of-line, __cold function, uaccess_warn_without_mm().
The trade-off is that only the first offending access in the system
warns, rather than the first one from each call site. The stack trace
still identifies the caller, and the access is refused either way.
Compared to the base of this series, with defconfig + KVM:
inline WARN_ON_ONCE() out-of-line warning
x86_64 total +5,884 B (+0.014%) +9,714 B (+0.023%)
arm64 total +40,948 B (+0.10%) +24,609 B (+0.06%)
On arm64, this removes the ~16 KB of __bug_table growth, while text
size is unchanged, as a call is the same size as the inline brk. On
x86_64, the call is larger than the inline ud2 and causes more .cold
splitting, so this is a slight regression.
Signed-off-by: James Houghton <[email protected]>
---
NOTE: This patch is included for reference only, to show the cost of the
inline warning.
---
include/linux/uaccess.h | 8 +++++++-
lib/usercopy.c | 6 ++++++
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h
index 7d26d2de9599..6a5b6c2b3388 100644
--- a/include/linux/uaccess.h
+++ b/include/linux/uaccess.h
@@ -66,9 +66,15 @@
* that keeps user addresses belonging to a specific mm must check
* current->mm against that mm itself (see e.g. KVM's kvm->mm checks).
*/
+void __cold uaccess_warn_without_mm(void);
+
static __always_inline bool uaccess_without_mm(void)
{
- return WARN_ON_ONCE(!current->mm);
+ if (likely(current->mm))
+ return false;
+ /* Warn out of line to keep inlined user copies small. */
+ uaccess_warn_without_mm();
+ return true;
}
/*
diff --git a/lib/usercopy.c b/lib/usercopy.c
index 9146c729c47a..3d547ecb74df 100644
--- a/lib/usercopy.c
+++ b/lib/usercopy.c
@@ -26,6 +26,12 @@ unsigned long _copy_to_user(void __user *to, const void
*from, unsigned long n)
EXPORT_SYMBOL(_copy_to_user);
#endif
+void __cold uaccess_warn_without_mm(void)
+{
+ WARN_ON_ONCE(1);
+}
+EXPORT_SYMBOL(uaccess_warn_without_mm);
+
/**
* check_zeroed_user: check if a userspace buffer only contains zero bytes
* @from: Source address, in userspace.
--
2.56.0.307.g5fe7bd680c-goog
