From: "Gautham R. Shenoy" <gautham.she...@amd.com> Currently TIF_NEED_RESCHED is being overloaded, to wakeup an idle CPU in TIF_POLLING mode to service an IPI even if there are no new tasks being woken up on the said CPU.
In preparation of a proper fix, introduce a new helper "need_resched_or_ipi()" which is intended to return true if either the TIF_NEED_RESCHED flag or if TIF_NOTIFY_IPI flag is set. Use this helper function in place of need_resched() in idle loops where TIF_POLLING_NRFLAG is set. To preserve bisectibility and avoid unbreakable idle loops, all the need_resched() checks within TIF_POLLING_NRFLAGS sections, have been replaced tree-wide with the need_resched_or_ipi() check. [ prateek: Replaced some of the missed out occurrences of need_resched() within a TIF_POLLING sections with need_resched_or_ipi() ] Cc: Richard Henderson <richard.hender...@linaro.org> Cc: Ivan Kokshaysky <i...@jurassic.park.msu.ru> Cc: Matt Turner <matts...@gmail.com> Cc: Russell King <li...@armlinux.org.uk> Cc: Guo Ren <guo...@kernel.org> Cc: Michal Simek <mon...@monstr.eu> Cc: Dinh Nguyen <dingu...@kernel.org> Cc: Jonas Bonn <jo...@southpole.se> Cc: Stefan Kristiansson <stefan.kristians...@saunalahti.fi> Cc: Stafford Horne <sho...@gmail.com> Cc: "James E.J. Bottomley" <james.bottom...@hansenpartnership.com> Cc: Helge Deller <del...@gmx.de> Cc: Michael Ellerman <m...@ellerman.id.au> Cc: Nicholas Piggin <npig...@gmail.com> Cc: Christophe Leroy <christophe.le...@csgroup.eu> Cc: "Aneesh Kumar K.V" <aneesh.ku...@kernel.org> Cc: "Naveen N. Rao" <naveen.n....@linux.ibm.com> Cc: Yoshinori Sato <ys...@users.sourceforge.jp> Cc: Rich Felker <dal...@libc.org> Cc: John Paul Adrian Glaubitz <glaub...@physik.fu-berlin.de> Cc: "David S. Miller" <da...@davemloft.net> Cc: Thomas Gleixner <t...@linutronix.de> Cc: Ingo Molnar <mi...@redhat.com> Cc: Borislav Petkov <b...@alien8.de> Cc: Dave Hansen <dave.han...@linux.intel.com> Cc: "H. Peter Anvin" <h...@zytor.com> Cc: "Rafael J. Wysocki" <raf...@kernel.org> Cc: Daniel Lezcano <daniel.lezc...@linaro.org> Cc: Peter Zijlstra <pet...@infradead.org> Cc: Juri Lelli <juri.le...@redhat.com> Cc: Vincent Guittot <vincent.guit...@linaro.org> Cc: Dietmar Eggemann <dietmar.eggem...@arm.com> Cc: Steven Rostedt <rost...@goodmis.org> Cc: Ben Segall <bseg...@google.com> Cc: Mel Gorman <mgor...@suse.de> Cc: Daniel Bristot de Oliveira <bris...@redhat.com> Cc: Valentin Schneider <vschn...@redhat.com> Cc: Al Viro <v...@zeniv.linux.org.uk> Cc: Linus Walleij <linus.wall...@linaro.org> Cc: Ard Biesheuvel <a...@kernel.org> Cc: Andrew Donnellan <a...@linux.ibm.com> Cc: Nicholas Miehlbradt <nicho...@linux.ibm.com> Cc: Andrew Morton <a...@linux-foundation.org> Cc: Arnd Bergmann <a...@arndb.de> Cc: Josh Poimboeuf <jpoim...@kernel.org> Cc: "Kirill A. Shutemov" <kirill.shute...@linux.intel.com> Cc: Rick Edgecombe <rick.p.edgeco...@intel.com> Cc: Tony Battersby <to...@cybernetics.com> Cc: Brian Gerst <brge...@gmail.com> Cc: Tim Chen <tim.c.c...@linux.intel.com> Cc: David Vernet <v...@manifault.com> Cc: x...@kernel.org Cc: linux-ker...@vger.kernel.org Cc: linux-alpha@vger.kernel.org Cc: linux-arm-ker...@lists.infradead.org Cc: linux-c...@vger.kernel.org Cc: linux-openr...@vger.kernel.org Cc: linux-par...@vger.kernel.org Cc: linuxppc-...@lists.ozlabs.org Cc: linux...@vger.kernel.org Cc: sparcli...@vger.kernel.org Cc: linux...@vger.kernel.org Signed-off-by: Gautham R. Shenoy <gautham.she...@amd.com> Co-developed-by: K Prateek Nayak <kprateek.na...@amd.com> Signed-off-by: K Prateek Nayak <kprateek.na...@amd.com> --- arch/x86/include/asm/mwait.h | 2 +- arch/x86/kernel/process.c | 2 +- drivers/cpuidle/cpuidle-powernv.c | 2 +- drivers/cpuidle/cpuidle-pseries.c | 2 +- drivers/cpuidle/poll_state.c | 2 +- include/linux/sched.h | 5 +++++ include/linux/sched/idle.h | 4 ++-- kernel/sched/idle.c | 7 ++++--- 8 files changed, 16 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/mwait.h b/arch/x86/include/asm/mwait.h index 778df05f8539..ac1370143407 100644 --- a/arch/x86/include/asm/mwait.h +++ b/arch/x86/include/asm/mwait.h @@ -115,7 +115,7 @@ static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned lo } __monitor((void *)¤t_thread_info()->flags, 0, 0); - if (!need_resched()) + if (!need_resched_or_ipi()) __mwait(eax, ecx); } current_clr_polling(); diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index b6f4e8399fca..ca6cb7e28cba 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -925,7 +925,7 @@ static __cpuidle void mwait_idle(void) } __monitor((void *)¤t_thread_info()->flags, 0, 0); - if (!need_resched()) { + if (!need_resched_or_ipi()) { __sti_mwait(0, 0); raw_local_irq_disable(); } diff --git a/drivers/cpuidle/cpuidle-powernv.c b/drivers/cpuidle/cpuidle-powernv.c index 9ebedd972df0..77c3bb371f56 100644 --- a/drivers/cpuidle/cpuidle-powernv.c +++ b/drivers/cpuidle/cpuidle-powernv.c @@ -79,7 +79,7 @@ static int snooze_loop(struct cpuidle_device *dev, dev->poll_time_limit = false; ppc64_runlatch_off(); HMT_very_low(); - while (!need_resched()) { + while (!need_resched_or_ipi()) { if (likely(snooze_timeout_en) && get_tb() > snooze_exit_time) { /* * Task has not woken up but we are exiting the polling diff --git a/drivers/cpuidle/cpuidle-pseries.c b/drivers/cpuidle/cpuidle-pseries.c index 14db9b7d985d..4f2b490f8b73 100644 --- a/drivers/cpuidle/cpuidle-pseries.c +++ b/drivers/cpuidle/cpuidle-pseries.c @@ -46,7 +46,7 @@ int snooze_loop(struct cpuidle_device *dev, struct cpuidle_driver *drv, snooze_exit_time = get_tb() + snooze_timeout; dev->poll_time_limit = false; - while (!need_resched()) { + while (!need_resched_or_ipi()) { HMT_low(); HMT_very_low(); if (likely(snooze_timeout_en) && get_tb() > snooze_exit_time) { diff --git a/drivers/cpuidle/poll_state.c b/drivers/cpuidle/poll_state.c index 9b6d90a72601..225f37897e45 100644 --- a/drivers/cpuidle/poll_state.c +++ b/drivers/cpuidle/poll_state.c @@ -26,7 +26,7 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev, limit = cpuidle_poll_time(drv, dev); - while (!need_resched()) { + while (!need_resched_or_ipi()) { cpu_relax(); if (loop_count++ < POLL_IDLE_RELAX_COUNT) continue; diff --git a/include/linux/sched.h b/include/linux/sched.h index 03bfe9ab2951..63451f6f25b9 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2258,6 +2258,11 @@ static __always_inline bool need_resched(void) return unlikely(tif_need_resched()); } +static __always_inline bool need_resched_or_ipi(void) +{ + return unlikely(tif_need_resched() || tif_notify_ipi()); +} + /* * Wrappers for p->thread_info->cpu access. No-op on UP. */ diff --git a/include/linux/sched/idle.h b/include/linux/sched/idle.h index 478084f9105e..d739ab810e00 100644 --- a/include/linux/sched/idle.h +++ b/include/linux/sched/idle.h @@ -63,7 +63,7 @@ static __always_inline bool __must_check current_set_polling_and_test(void) */ smp_mb__after_atomic(); - return unlikely(tif_need_resched()); + return unlikely(need_resched_or_ipi()); } static __always_inline bool __must_check current_clr_polling_and_test(void) @@ -76,7 +76,7 @@ static __always_inline bool __must_check current_clr_polling_and_test(void) */ smp_mb__after_atomic(); - return unlikely(tif_need_resched()); + return unlikely(need_resched_or_ipi()); } #else diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c index 31231925f1ec..fcc734f45a2a 100644 --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -57,7 +57,7 @@ static noinline int __cpuidle cpu_idle_poll(void) ct_cpuidle_enter(); raw_local_irq_enable(); - while (!tif_need_resched() && + while (!need_resched_or_ipi() && (cpu_idle_force_poll || tick_check_broadcast_expired())) cpu_relax(); raw_local_irq_disable(); @@ -153,7 +153,7 @@ static void cpuidle_idle_call(void) * Check if the idle task must be rescheduled. If it is the * case, exit the function after re-enabling the local irq. */ - if (need_resched()) { + if (need_resched_or_ipi()) { local_irq_enable(); return; } @@ -255,7 +255,7 @@ static void do_idle(void) __current_set_polling(); tick_nohz_idle_enter(); - while (!need_resched()) { + while (!need_resched_or_ipi()) { rmb(); /* @@ -336,6 +336,7 @@ static void do_idle(void) * RCU relies on this call to be done outside of an RCU read-side * critical section. */ + current_clr_notify_ipi(); flush_smp_call_function_queue(); schedule_idle(); -- 2.34.1