Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Tejun Heo writes: > Hello, Michael. > > On Tue, Oct 18, 2016 at 03:37:42PM +1100, Michael Ellerman wrote: >> That doesn't compile, wq doesn't exist. >> >> I guessed that you meant: >> >> + wq_numa_init(); >> + list_for_each_entry(wq, , list) >> +

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Tejun Heo writes: > Hello, Michael. > > On Tue, Oct 18, 2016 at 03:37:42PM +1100, Michael Ellerman wrote: >> That doesn't compile, wq doesn't exist. >> >> I guessed that you meant: >> >> + wq_numa_init(); >> + list_for_each_entry(wq, , list) >> +

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, Michael. On Tue, Oct 18, 2016 at 03:37:42PM +1100, Michael Ellerman wrote: > That doesn't compile, wq doesn't exist. > > I guessed that you meant: > > + wq_numa_init(); > + list_for_each_entry(wq, , list) > + wq_update_unbound_numa(wq, smp_processor_id(), true);

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, Michael. On Tue, Oct 18, 2016 at 03:37:42PM +1100, Michael Ellerman wrote: > That doesn't compile, wq doesn't exist. > > I guessed that you meant: > > + wq_numa_init(); > + list_for_each_entry(wq, , list) > + wq_update_unbound_numa(wq, smp_processor_id(), true);

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Balbir Singh writes: > On 17/10/16 23:24, Michael Ellerman wrote: >> That happened because we haven't yet called set_cpu_numa_node() for the >> non-boot >> cpus, because that happens in smp_prepare_cpus(), and >> workqueue_init_early() is called much earlier than that. >>

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Balbir Singh writes: > On 17/10/16 23:24, Michael Ellerman wrote: >> That happened because we haven't yet called set_cpu_numa_node() for the >> non-boot >> cpus, because that happens in smp_prepare_cpus(), and >> workqueue_init_early() is called much earlier than that. >> >> This doesn't

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, Michael. Other NUMA archs are lazy-initializing cpu to node mapping too, so we need to fix it from workqueue side. This also means that we've been getting NUMA node wrong for percpu pools on those archs. Can you please try the following patch and if it resolves the issue, report the

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, Michael. Other NUMA archs are lazy-initializing cpu to node mapping too, so we need to fix it from workqueue side. This also means that we've been getting NUMA node wrong for percpu pools on those archs. Can you please try the following patch and if it resolves the issue, report the

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, Michael. On Mon, Oct 17, 2016 at 11:24:34PM +1100, Michael Ellerman wrote: > The bad case (where we hit the BUG_ON I added above) is where we are > creating a wq for node 1. > > In wq_calc_node_cpumask() we do: > > cpumask_and(cpumask, attrs->cpumask,

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, Michael. On Mon, Oct 17, 2016 at 11:24:34PM +1100, Michael Ellerman wrote: > The bad case (where we hit the BUG_ON I added above) is where we are > creating a wq for node 1. > > In wq_calc_node_cpumask() we do: > > cpumask_and(cpumask, attrs->cpumask,

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, On Sat, Oct 15, 2016 at 08:48:01PM +1100, Michael Ellerman wrote: > > Hmm... it doesn't reproduce it here and can't see how the commit would > > affect this given that it doesn't really change when the kworker > > kthreads are being created. > > Try turning on CONFIG_DEBUG_PER_CPU_MAPS=y

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, On Sat, Oct 15, 2016 at 08:48:01PM +1100, Michael Ellerman wrote: > > Hmm... it doesn't reproduce it here and can't see how the commit would > > affect this given that it doesn't really change when the kworker > > kthreads are being created. > > Try turning on CONFIG_DEBUG_PER_CPU_MAPS=y

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On 17/10/16 23:24, Michael Ellerman wrote: > Tejun Heo writes: > >> Hello, Michael. >> >> On Tue, Oct 11, 2016 at 10:22:13PM +1100, Michael Ellerman wrote: >>> The oops happens because we're in enqueue_task_fair() and p->se->cfs_rq >>> is NULL. >>> >>> The cfs_rq is NULL

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On 17/10/16 23:24, Michael Ellerman wrote: > Tejun Heo writes: > >> Hello, Michael. >> >> On Tue, Oct 11, 2016 at 10:22:13PM +1100, Michael Ellerman wrote: >>> The oops happens because we're in enqueue_task_fair() and p->se->cfs_rq >>> is NULL. >>> >>> The cfs_rq is NULL because we did

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Tejun Heo writes: > Hello, Michael. > > On Tue, Oct 11, 2016 at 10:22:13PM +1100, Michael Ellerman wrote: >> The oops happens because we're in enqueue_task_fair() and p->se->cfs_rq >> is NULL. >> >> The cfs_rq is NULL because we did set_task_rq(p, 2048), where 2048 is >>

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Tejun Heo writes: > Hello, Michael. > > On Tue, Oct 11, 2016 at 10:22:13PM +1100, Michael Ellerman wrote: >> The oops happens because we're in enqueue_task_fair() and p->se->cfs_rq >> is NULL. >> >> The cfs_rq is NULL because we did set_task_rq(p, 2048), where 2048 is >> NR_CPUS. That causes us

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Tejun Heo writes: > Hello, Michael. > > On Tue, Oct 11, 2016 at 10:22:13PM +1100, Michael Ellerman wrote: >> The oops happens because we're in enqueue_task_fair() and p->se->cfs_rq >> is NULL. >> >> The cfs_rq is NULL because we did set_task_rq(p, 2048), where 2048 is >>

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Tejun Heo writes: > Hello, Michael. > > On Tue, Oct 11, 2016 at 10:22:13PM +1100, Michael Ellerman wrote: >> The oops happens because we're in enqueue_task_fair() and p->se->cfs_rq >> is NULL. >> >> The cfs_rq is NULL because we did set_task_rq(p, 2048), where 2048 is >> NR_CPUS. That causes us

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On 15/10/16 02:08, Tejun Heo wrote: > Hello, Balbir. > > On Tue, Oct 11, 2016 at 11:21:09PM +1100, Balbir Singh wrote: >> FYI: I see the samething on my cpu as well, its just that I get lucky >> and cpu_online(cpu) returns false. > > Are you seeing this on x86 or is your test setup also a

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On 15/10/16 02:08, Tejun Heo wrote: > Hello, Balbir. > > On Tue, Oct 11, 2016 at 11:21:09PM +1100, Balbir Singh wrote: >> FYI: I see the samething on my cpu as well, its just that I get lucky >> and cpu_online(cpu) returns false. > > Are you seeing this on x86 or is your test setup also a

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On 15/10/16 02:07, Tejun Heo wrote: > Hello, Michael. > > On Tue, Oct 11, 2016 at 10:22:13PM +1100, Michael Ellerman wrote: >> The oops happens because we're in enqueue_task_fair() and p->se->cfs_rq >> is NULL. >> >> The cfs_rq is NULL because we did set_task_rq(p, 2048), where 2048 is >>

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On 15/10/16 02:07, Tejun Heo wrote: > Hello, Michael. > > On Tue, Oct 11, 2016 at 10:22:13PM +1100, Michael Ellerman wrote: >> The oops happens because we're in enqueue_task_fair() and p->se->cfs_rq >> is NULL. >> >> The cfs_rq is NULL because we did set_task_rq(p, 2048), where 2048 is >>

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, Balbir. On Tue, Oct 11, 2016 at 11:21:09PM +1100, Balbir Singh wrote: > FYI: I see the samething on my cpu as well, its just that I get lucky > and cpu_online(cpu) returns false. Are you seeing this on x86 or is your test setup also a power machine? Thanks. -- tejun

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, Balbir. On Tue, Oct 11, 2016 at 11:21:09PM +1100, Balbir Singh wrote: > FYI: I see the samething on my cpu as well, its just that I get lucky > and cpu_online(cpu) returns false. Are you seeing this on x86 or is your test setup also a power machine? Thanks. -- tejun

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, Michael. On Tue, Oct 11, 2016 at 10:22:13PM +1100, Michael Ellerman wrote: > The oops happens because we're in enqueue_task_fair() and p->se->cfs_rq > is NULL. > > The cfs_rq is NULL because we did set_task_rq(p, 2048), where 2048 is > NR_CPUS. That causes us to index past the end of the

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, Michael. On Tue, Oct 11, 2016 at 10:22:13PM +1100, Michael Ellerman wrote: > The oops happens because we're in enqueue_task_fair() and p->se->cfs_rq > is NULL. > > The cfs_rq is NULL because we did set_task_rq(p, 2048), where 2048 is > NR_CPUS. That causes us to index past the end of the

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On 11/10/16 22:22, Michael Ellerman wrote: > Tejun Heo writes: > >> Hello, Michael. >> >> On Mon, Oct 10, 2016 at 09:22:55PM +1100, Michael Ellerman wrote: >>> This patch seems to be causing one of my Power8 boxes not to boot. >>> >>> Specifically commit 3347fa092821

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On 11/10/16 22:22, Michael Ellerman wrote: > Tejun Heo writes: > >> Hello, Michael. >> >> On Mon, Oct 10, 2016 at 09:22:55PM +1100, Michael Ellerman wrote: >>> This patch seems to be causing one of my Power8 boxes not to boot. >>> >>> Specifically commit 3347fa092821 ("workqueue: make

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Tejun Heo writes: > Hello, Michael. > > On Mon, Oct 10, 2016 at 09:22:55PM +1100, Michael Ellerman wrote: >> This patch seems to be causing one of my Power8 boxes not to boot. >> >> Specifically commit 3347fa092821 ("workqueue: make workqueue available >> early during boot") in

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Tejun Heo writes: > Hello, Michael. > > On Mon, Oct 10, 2016 at 09:22:55PM +1100, Michael Ellerman wrote: >> This patch seems to be causing one of my Power8 boxes not to boot. >> >> Specifically commit 3347fa092821 ("workqueue: make workqueue available >> early during boot") in linux-next. >>

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On 10/10/16 23:53, Tejun Heo wrote: > On Mon, Oct 10, 2016 at 10:17:16PM +1100, Balbir Singh wrote: >> rest_init() >> { >> ... >> kernel_thread(kernel_init, NULL, CLONE_FS); >> numa_default_policy(); >> pid = kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES); >>

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On 10/10/16 23:53, Tejun Heo wrote: > On Mon, Oct 10, 2016 at 10:17:16PM +1100, Balbir Singh wrote: >> rest_init() >> { >> ... >> kernel_thread(kernel_init, NULL, CLONE_FS); >> numa_default_policy(); >> pid = kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES); >>

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On Mon, Oct 10, 2016 at 09:02:53AM -0400, Tejun Heo wrote: > Hello, Michael. > > On Mon, Oct 10, 2016 at 09:22:55PM +1100, Michael Ellerman wrote: > > This patch seems to be causing one of my Power8 boxes not to boot. > > > > Specifically commit 3347fa092821 ("workqueue: make workqueue available

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On Mon, Oct 10, 2016 at 09:02:53AM -0400, Tejun Heo wrote: > Hello, Michael. > > On Mon, Oct 10, 2016 at 09:22:55PM +1100, Michael Ellerman wrote: > > This patch seems to be causing one of my Power8 boxes not to boot. > > > > Specifically commit 3347fa092821 ("workqueue: make workqueue available

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, Michael. On Mon, Oct 10, 2016 at 09:22:55PM +1100, Michael Ellerman wrote: > This patch seems to be causing one of my Power8 boxes not to boot. > > Specifically commit 3347fa092821 ("workqueue: make workqueue available > early during boot") in linux-next. > > If I revert this on top of

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

Hello, Michael. On Mon, Oct 10, 2016 at 09:22:55PM +1100, Michael Ellerman wrote: > This patch seems to be causing one of my Power8 boxes not to boot. > > Specifically commit 3347fa092821 ("workqueue: make workqueue available > early during boot") in linux-next. > > If I revert this on top of

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On Mon, Oct 10, 2016 at 10:17:16PM +1100, Balbir Singh wrote: > rest_init() > { > ... > kernel_thread(kernel_init, NULL, CLONE_FS); > numa_default_policy(); > pid = kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES); > rcu_read_lock(); > kthreadd_task =

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On Mon, Oct 10, 2016 at 10:17:16PM +1100, Balbir Singh wrote: > rest_init() > { > ... > kernel_thread(kernel_init, NULL, CLONE_FS); > numa_default_policy(); > pid = kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES); > rcu_read_lock(); > kthreadd_task =

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On 10/10/16 21:22, Michael Ellerman wrote: > Hi Tejun, > > Tejun Heo writes: >> From f85002f627f7fdc7b3cda526863f5c9a8d36b997 Mon Sep 17 00:00:00 2001 >> From: Tejun Heo >> Date: Fri, 16 Sep 2016 15:49:32 -0400 >> Subject: [PATCH] workqueue: make workqueue

Re: Oops on Power8 (was Re: [PATCH v2 1/7] workqueue: make workqueue available early during boot)

On 10/10/16 21:22, Michael Ellerman wrote: > Hi Tejun, > > Tejun Heo writes: >> From f85002f627f7fdc7b3cda526863f5c9a8d36b997 Mon Sep 17 00:00:00 2001 >> From: Tejun Heo >> Date: Fri, 16 Sep 2016 15:49:32 -0400 >> Subject: [PATCH] workqueue: make workqueue available early during boot >> >>

[PATCH v2 1/7] workqueue: make workqueue available early during boot

>From f85002f627f7fdc7b3cda526863f5c9a8d36b997 Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Fri, 16 Sep 2016 15:49:32 -0400 Subject: [PATCH] workqueue: make workqueue available early during boot Workqueue is currently initialized in an early init call; however, there are cases

[PATCH v2 1/7] workqueue: make workqueue available early during boot

>From f85002f627f7fdc7b3cda526863f5c9a8d36b997 Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Fri, 16 Sep 2016 15:49:32 -0400 Subject: [PATCH] workqueue: make workqueue available early during boot Workqueue is currently initialized in an early init call; however, there are cases where early boot