On Mon, Apr 08, 2013 at 05:57:01PM +0200, Ingo Molnar wrote:
>
> * Robin Holt <h...@sgi.com> wrote:
>
> > We noticed that recently, reboot of a 1024 cpu machine takes approx 16
> > minutes of just stopping the cpus. The slowdown was tracked to commit
> > f96972f which went into v3.7 and then to the stable trees.
> >
> > x86 does not need to be running the boot cpu to pull reset and I don't
> > think it is really needed for shutdown either.
> >
> > I decided to go the "simple" way and make this a config option that is
> > selected by the x86 arch. I don't know which other arch's would also
> > benefit, if any.
> >
> > Signed-off-by: Robin Holt <h...@sgi.com>
> > To: Andrew Morton <a...@linux-foundation.org>
> > Cc: Russ Anderson <r...@sgi.com>
> > Cc: Thomas Gleixner <t...@linutronix.de>
> > Cc: Ingo Molnar <mi...@redhat.com>
> > Cc: "H. Peter Anvin" <h...@zytor.com>
> > Cc: Shawn Guo <shawn....@linaro.org>
> > Cc: <sta...@vger.kernel.org>
> >
> > ---
> > arch/x86/Kconfig | 3 +++
> > kernel/Kconfig.shutdown | 3 +++
> > kernel/sys.c | 4 ++++
> > 3 files changed, 10 insertions(+)
> > create mode 100644 kernel/Kconfig.shutdown
> >
> > diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> > index 70c0f3d..9611942 100644
> > --- a/arch/x86/Kconfig
> > +++ b/arch/x86/Kconfig
> > @@ -120,6 +120,7 @@ config X86
> > select OLD_SIGSUSPEND3 if X86_32 || IA32_EMULATION
> > select OLD_SIGACTION if X86_32
> > select COMPAT_OLD_SIGACTION if IA32_EMULATION
> > + select ARCH_SHUTDOWN_TO_ANY_CPU
> >
> > config INSTRUCTION_DECODER
> > def_bool y
> > @@ -839,6 +840,8 @@ config SCHED_MC
> > making when dealing with multi-core CPU chips at a cost of slightly
> > increased overhead in some places. If unsure say N here.
> >
> > +source "kernel/Kconfig.shutdown"
> > +
> > source "kernel/Kconfig.preempt"
> >
> > config X86_UP_APIC
> > diff --git a/kernel/Kconfig.shutdown b/kernel/Kconfig.shutdown
> > new file mode 100644
> > index 0000000..d79fc04
> > --- /dev/null
> > +++ b/kernel/Kconfig.shutdown
> > @@ -0,0 +1,3 @@
> > +
> > +config ARCH_SHUTDOWN_TO_ANY_CPU
> > + bool
> > diff --git a/kernel/sys.c b/kernel/sys.c
> > index 39c9c4a..c0b8880 100644
> > --- a/kernel/sys.c
> > +++ b/kernel/sys.c
> > @@ -369,7 +369,9 @@ EXPORT_SYMBOL(unregister_reboot_notifier);
> > void kernel_restart(char *cmd)
> > {
> > kernel_restart_prepare(cmd);
> > +#ifndef CONFIG_ARCH_SHUTDOWN_TO_ANY_CPU
> > disable_nonboot_cpus();
> > +#endif
> > if (!cmd)
> > printk(KERN_EMERG "Restarting system.\n");
> > else
> > @@ -413,7 +415,9 @@ void kernel_power_off(void)
> > kernel_shutdown_prepare(SYSTEM_POWER_OFF);
> > if (pm_power_off_prepare)
> > pm_power_off_prepare();
> > +#ifndef CONFIG_ARCH_SHUTDOWN_TO_ANY_CPU
> > disable_nonboot_cpus();
> > +#endif
> > syscore_shutdown();
> > printk(KERN_EMERG "Power down.\n");
> > kmsg_dump(KMSG_DUMP_POWEROFF);
>
> Hm, the 'fix' is a pretty ugly workaround that does not fix much IMHO.
>
> I think the original commit:
>
> f96972f2dc63 kernel/sys.c: call disable_nonboot_cpus() in kernel_restart()
>
> actually regressed your 1024 CPU systems, and should possibly be reverted or
> fixed
> in some other fashion - such as by migrating to the primary CPU (on
> architectures
> that require that), instead of hotplug offlining every secondary CPU on every
> architecture!
>
> Alternatively, disable_nonboot_cpus() could perhaps be improved to down CPUs
> in
> parallel: issue the CPU-down requests to every CPU, then wait for them to
> complete
> - instead of the loop over every CPU?
>
> This would be the conceptual counter part to parallel boot up of CPUs -
> something
> SGI might be interested in as well?
Interested, but even more so interested in parellelizing memory setup. ;)
How can we proceed with this?
Robin
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
