On Fri, Jun 22, 2018 at 03:23:51PM +0900, Byungchul Park wrote: > On Fri, Jun 22, 2018 at 03:12:06PM +0900, Byungchul Park wrote: > > When passing through irq or NMI contexts, the current code uses > > ->dynticks_nmi_nesting to detect if it's in the ourmost at the moment. > > > > Here, the thing is that all the related functions, rcu_irq_enter(), > > rcu_nmi_enter(), rcu_irq_exit() and rcu_nmi_exit() are carrying out > > the check within each under the following call relationship so there > > must be redundant conditional branches: > > > > rcu_irq_enter() > > /* A conditional branch with ->dynticks_nmi_nesting */ > > rcu_nmi_enter() > > /* A conditional branch with ->dynticks_nmi_nesting */ > ^ > Precisely, ->dynticks > > /* A conditional branch with ->dynticks_nmi_nesting */ > > > > rcu_irq_exit() > > /* A conditional branch with ->dynticks_nmi_nesting */ > > rcu_nmi_exit() > > /* A conditional branch with ->dynticks_nmi_nesting */ > > /* A conditional branch with ->dynticks_nmi_nesting */ > > > > rcu_nmi_enter() > > /* A conditional branch with ->dynticks_nmi_nesting */ > ^ > Precisely, ->dynticks > > > > rcu_nmi_exit() > > /* A conditional branch with ->dynticks_nmi_nesting */ > > > > In order to make the code more readable and remove redundant > > conditional branches, we need to refactor rcu_{nmi,irq}_{enter,exit}() > > so they use a common function for each like: > > > > rcu_irq_enter() inlining rcu_nmi_enter_common(irq) > > /* A conditional branch with ->dynticks_nmi_nesting */ > ^ > Precisely, ->dynticks > > > > rcu_irq_exit() inlining rcu_nmi_exit_common(irq) > > /* A conditional branch with ->dynticks_nmi_nesting */ > > > > rcu_nmi_enter() inlining rcu_nmi_enter_common(nmi) > > /* A conditional branch with ->dynticks_nmi_nesting */ > ^ > Precisely, ->dynticks > > > > rcu_nmi_exit() inlining rcu_nmi_exit_common(nmi) > > /* A conditional branch with ->dynticks_nmi_nesting */
I queued this for testing and further review, updating the commit log as follows. Please let me know if I messed up something. Thanx, Paul ------------------------------------------------------------------------ commit 5e5ea52645b197fb7ae2f59f7927079b91e91aa0 Author: Byungchul Park <byungchul.p...@lge.com> Date: Fri Jun 22 15:12:06 2018 +0900 rcu: Refactor rcu_{nmi,irq}_{enter,exit}() When entering or exiting irq or NMI handlers, the current code uses ->dynticks_nmi_nesting to detect if it is in the outermost handler, that is, the one interrupting or returning to an RCU-idle context (the idle loop or nohz_full usermode execution). When entering the outermost handler via an interrupt (as opposed to NMI), it is necessary to invoke rcu_dynticks_task_exit() just before the CPU is marked non-idle from an RCU perspective and to invoke rcu_cleanup_after_idle() just after the CPU is marked non-idle. Similarly, when exiting the outermost handler via an interrupt, it is necessary to invoke rcu_prepare_for_idle() just before marking the CPU idle and to invoke rcu_dynticks_task_enter() just after marking the CPU idle. The decision to execute these four functions is currently taken in rcu_irq_enter() and rcu_irq_exit() as follows: rcu_irq_enter() /* A conditional branch with ->dynticks_nmi_nesting */ rcu_nmi_enter() /* A conditional branch with ->dynticks */ /* A conditional branch with ->dynticks_nmi_nesting */ rcu_irq_exit() /* A conditional branch with ->dynticks_nmi_nesting */ rcu_nmi_exit() /* A conditional branch with ->dynticks_nmi_nesting */ /* A conditional branch with ->dynticks_nmi_nesting */ rcu_nmi_enter() /* A conditional branch with ->dynticks */ rcu_nmi_exit() /* A conditional branch with ->dynticks_nmi_nesting */ This works, but the conditional branches in rcu_irq_enter() and rcu_irq_exit() are redundant with those in rcu_nmi_enter() and rcu_nmi_exit(), respectively. Redundant branches are not something we want in the to/from-idle fastpaths, so this commit refactors rcu_{nmi,irq}_{enter,exit}() so they use a common inlined function passed a constant argument as follows: rcu_irq_enter() inlining rcu_nmi_enter_common(irq=true) /* A conditional branch with ->dynticks */ rcu_irq_exit() inlining rcu_nmi_exit_common(irq=true) /* A conditional branch with ->dynticks_nmi_nesting */ rcu_nmi_enter() inlining rcu_nmi_enter_common(irq=false) /* A conditional branch with ->dynticks */ rcu_nmi_exit() inlining rcu_nmi_exit_common(irq=false) /* A conditional branch with ->dynticks_nmi_nesting */ The combination of the constant function argument and the inlining allows the compiler to discard the conditionals that previously controlled execution of rcu_dynticks_task_exit(), rcu_cleanup_after_idle(), rcu_prepare_for_idle(), and rcu_dynticks_task_enter(). This reduces both the to-idle and from-idle path lengths by two conditional branches each, and improves readability as well. Suggested-by: Paul E. McKenney <paul...@linux.vnet.ibm.com> Signed-off-by: Byungchul Park <byungchul.p...@lge.com> Signed-off-by: Paul E. McKenney <paul...@linux.vnet.ibm.com> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 8788ddbc0d13..4ed74f10c852 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -773,17 +773,17 @@ void rcu_user_enter(void) #endif /* CONFIG_NO_HZ_FULL */ /** - * rcu_nmi_exit - inform RCU of exit from NMI context + * rcu_nmi_exit_common - inform RCU of exit from NMI context * * If we are returning from the outermost NMI handler that interrupted an * RCU-idle period, update rdtp->dynticks and rdtp->dynticks_nmi_nesting * to let the RCU grace-period handling know that the CPU is back to * being RCU-idle. * - * If you add or remove a call to rcu_nmi_exit(), be sure to test + * If you add or remove a call to rcu_nmi_exit_common(), be sure to test * with CONFIG_RCU_EQS_DEBUG=y. */ -void rcu_nmi_exit(void) +static __always_inline void rcu_nmi_exit_common(bool irq) { struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks); @@ -809,7 +809,22 @@ void rcu_nmi_exit(void) /* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */ trace_rcu_dyntick(TPS("Startirq"), rdtp->dynticks_nmi_nesting, 0, rdtp->dynticks); WRITE_ONCE(rdtp->dynticks_nmi_nesting, 0); /* Avoid store tearing. */ + + if (irq) + rcu_prepare_for_idle(); + rcu_dynticks_eqs_enter(); + + if (irq) + rcu_dynticks_task_enter(); +} + +/** + * rcu_nmi_exit - inform RCU of exit from NMI context + */ +void rcu_nmi_exit(void) +{ + rcu_nmi_exit_common(false); } /** @@ -833,14 +848,8 @@ void rcu_nmi_exit(void) */ void rcu_irq_exit(void) { - struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks); - lockdep_assert_irqs_disabled(); - if (rdtp->dynticks_nmi_nesting == 1) - rcu_prepare_for_idle(); - rcu_nmi_exit(); - if (rdtp->dynticks_nmi_nesting == 0) - rcu_dynticks_task_enter(); + rcu_nmi_exit_common(true); } /* @@ -923,7 +932,7 @@ void rcu_user_exit(void) #endif /* CONFIG_NO_HZ_FULL */ /** - * rcu_nmi_enter - inform RCU of entry to NMI context + * rcu_nmi_enter_common - inform RCU of entry to NMI context * * If the CPU was idle from RCU's viewpoint, update rdtp->dynticks and * rdtp->dynticks_nmi_nesting to let the RCU grace-period handling know @@ -931,10 +940,10 @@ void rcu_user_exit(void) * long as the nesting level does not overflow an int. (You will probably * run out of stack space first.) * - * If you add or remove a call to rcu_nmi_enter(), be sure to test + * If you add or remove a call to rcu_nmi_enter_common(), be sure to test * with CONFIG_RCU_EQS_DEBUG=y. */ -void rcu_nmi_enter(void) +static __always_inline void rcu_nmi_enter_common(bool irq) { struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks); long incby = 2; @@ -951,7 +960,15 @@ void rcu_nmi_enter(void) * period (observation due to Andy Lutomirski). */ if (rcu_dynticks_curr_cpu_in_eqs()) { + + if (irq) + rcu_dynticks_task_exit(); + rcu_dynticks_eqs_exit(); + + if (irq) + rcu_cleanup_after_idle(); + incby = 1; } trace_rcu_dyntick(incby == 1 ? TPS("Endirq") : TPS("++="), @@ -962,6 +979,14 @@ void rcu_nmi_enter(void) barrier(); } +/** + * rcu_nmi_enter - inform RCU of entry to NMI context + */ +void rcu_nmi_enter(void) +{ + rcu_nmi_enter_common(false); +} + /** * rcu_irq_enter - inform RCU that current CPU is entering irq away from idle * @@ -986,14 +1011,8 @@ void rcu_nmi_enter(void) */ void rcu_irq_enter(void) { - struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks); - lockdep_assert_irqs_disabled(); - if (rdtp->dynticks_nmi_nesting == 0) - rcu_dynticks_task_exit(); - rcu_nmi_enter(); - if (rdtp->dynticks_nmi_nesting == 1) - rcu_cleanup_after_idle(); + rcu_nmi_enter_common(true); } /*