The rcutree.do_rcu_barrier test hook is intended to prevent deferred RCU
callbacks from one stress test spilling into the next. Since kfree_rcu()
sheaves were added, an object can remain deferred without appearing on an
ordinary RCU callback list. rcu_barrier() therefore no longer fulfils the
hook's stated purpose by itself.
Drain kfree_rcu sheaves and kvfree_rcu batches before completing the
ordinary RCU barrier. Keep the explicit rcu_barrier() because the hook's
original ordinary-callback contract should not depend on the current,
undocumented fact that kvfree_rcu_barrier() includes one internally.
Keep the existing throttling because this remains a deliberately expensive
test-only action. Do not coalesce requests based on the ordinary
rcu_barrier() sequence: an unrelated ordinary barrier does not prove that
sheaves were drained.
A reproducer creates a private SLAB_NO_MERGE cache whose first allocation
populates a 60-object slab. It queues that object with kfree_rcu(), invokes
the hook, and reads the active-object count from /proc/slabinfo. In four
fresh VM pairs, the parent retained the object (60 to 60). The patched hook
drained it (60 to 59).
Fixes: ec66e0d59952 ("slab: add sheaf support for batching kfree_rcu()
operations")
Signed-off-by: Matthias Goergens <[email protected]>
---
.../admin-guide/kernel-parameters.txt | 7 ++---
kernel/rcu/tree.c | 27 ++++++++++++-------
2 files changed, 21 insertions(+), 13 deletions(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt
b/Documentation/admin-guide/kernel-parameters.txt
index 68647ff4bdd2..244a53166249 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5699,9 +5699,10 @@ Kernel parameters
there is an ongoing too-long CSD-lock wait.
rcutree.do_rcu_barrier= [KNL]
- Request a call to rcu_barrier(). This is
- throttled so that userspace tests can safely
- hammer on the sysfs variable if they so choose.
+ Request that deferred kfree_rcu() objects and
+ ordinary call_rcu() callbacks be drained. This is
+ throttled so that userspace tests can safely hammer
+ on the sysfs variable if they so choose.
If triggered before the RCU grace-period machinery
is fully active, this will error out with EAGAIN.
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 96848fc1f02b..014e28ec3bd3 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3989,12 +3989,12 @@ EXPORT_SYMBOL_GPL(rcu_barrier);
static unsigned long rcu_barrier_last_throttle;
/**
- * rcu_barrier_throttled - Do rcu_barrier(), but limit to one per second
+ * rcu_barrier_throttled - Drain deferred RCU frees, but rate-limit starts
*
- * This can be thought of as guard rails around rcu_barrier() that
- * permits unrestricted userspace use, at least assuming the hardware's
- * try_cmpxchg() is robust. There will be at most one call per second to
- * rcu_barrier() system-wide from use of this function, which means that
+ * This can be thought of as guard rails around the deferred-free barriers
+ * that permit unrestricted userspace use, at least assuming the hardware's
+ * try_cmpxchg() is robust. There will be at most one drain operation started
+ * per sixteenth of a second from use of this function, which means that
* callers might needlessly wait a second or three.
*
* This is intended for use by test suites to avoid OOM by flushing RCU
@@ -4011,18 +4011,25 @@ static void rcu_barrier_throttled(void)
{
unsigned long j = jiffies;
unsigned long old = READ_ONCE(rcu_barrier_last_throttle);
- unsigned long s = rcu_seq_snap(&rcu_state.barrier_sequence);
while (time_in_range(j, old, old + HZ / 16) ||
!try_cmpxchg(&rcu_barrier_last_throttle, &old, j)) {
schedule_timeout_idle(HZ / 16);
- if (rcu_seq_done(&rcu_state.barrier_sequence, s)) {
- smp_mb(); /* caller's subsequent code after above
check. */
- return;
- }
j = jiffies;
old = READ_ONCE(rcu_barrier_last_throttle);
}
+ /*
+ * kfree_rcu() can retain objects outside the ordinary callback lists in
+ * per-CPU SLUB sheaves and kvfree_rcu batches. Test suites use this
hook
+ * to prevent deferred frees from spilling into the following test, so
+ * drain those queues as well as ordinary call_rcu() callbacks.
+ *
+ * kvfree_rcu_barrier() currently includes an ordinary barrier, but that
+ * is not part of its documented API. Keep the explicit rcu_barrier()
so
+ * this hook's original contract does not depend on slab implementation
+ * details.
+ */
+ kvfree_rcu_barrier();
rcu_barrier();
}
--
2.55.0
