On Wed, Dec 23, 2020 at 5:39 AM Dexuan-Linux Cui <dexuan.li...@gmail.com> wrote:
>
> On Fri, Dec 18, 2020 at 8:11 AM Lai Jiangshan <jiangshan...@gmail.com> wrote:
> >
> > From: Lai Jiangshan <la...@linux.alibaba.com>
> >
> > 06249738a41a ("workqueue: Manually break affinity on hotplug")
> > said that scheduler will not force break affinity for us.
> >
> > But workqueue highly depends on the old behavior. Many parts of the codes
> > relies on it, 06249738a41a ("workqueue: Manually break affinity on hotplug")
> > is not enough to change it, and the commit has flaws in itself too.
> >
> > It doesn't handle for worker detachment.
> > It doesn't handle for worker attachement, mainly worker creation
> > which is handled by Valentin Schneider's patch [1].
> > It doesn't handle for unbound workers which might be possible
> > per-cpu-kthread.
> >
> > We need to thoroughly update the way workqueue handles affinity
> > in cpu hot[un]plug, what is this patchset intends to do and
> > replace the Valentin Schneider's patch [1]. The equivalent patch
> > is patch 10.
> >
> > Patch 1 fixes a flaw reported by Hillf Danton <hdan...@sina.com>.
> > I have to include this fix because later patches depends on it.
> >
> > The patchset is based on tip/master rather than workqueue tree,
> > because the patchset is a complement for 06249738a41a ("workqueue:
> > Manually break affinity on hotplug") which is only in tip/master by now.
> >
> > And TJ acked to route the series through tip.
> >
> > Changed from V1:
> > Add TJ's acked-by for the whole patchset
> >
> > Add more words to the comments and the changelog, mainly derived
> > from discussion with Peter.
> >
> > Update the comments as TJ suggested.
> >
> > Update a line of code as Valentin suggested.
> >
> > Add Valentin's ack for patch 10 because "Seems alright to me." and
> > add Valentin's comments to the changelog which is integral.
> >
> > [1]:
> > https://lore.kernel.org/r/ff62e3ee994efb3620177bf7b19fab16f4866845.ca...@redhat.com
> > [V1 patcheset]:
> > https://lore.kernel.org/lkml/20201214155457.3430-1-jiangshan...@gmail.com/
> >
> > Cc: Hillf Danton <hdan...@sina.com>
> > Cc: Valentin Schneider <valentin.schnei...@arm.com>
> > Cc: Qian Cai <c...@redhat.com>
> > Cc: Peter Zijlstra <pet...@infradead.org>
> > Cc: Vincent Donnefort <vincent.donnef...@arm.com>
> > Cc: Tejun Heo <t...@kernel.org>
> >
> > Lai Jiangshan (10):
> > workqueue: restore unbound_workers' cpumask correctly
> > workqueue: use cpu_possible_mask instead of cpu_active_mask to break
> > affinity
> > workqueue: Manually break affinity on pool detachment
> > workqueue: don't set the worker's cpumask when kthread_bind_mask()
> > workqueue: introduce wq_online_cpumask
> > workqueue: use wq_online_cpumask in restore_unbound_workers_cpumask()
> > workqueue: Manually break affinity on hotplug for unbound pool
> > workqueue: reorganize workqueue_online_cpu()
> > workqueue: reorganize workqueue_offline_cpu() unbind_workers()
> > workqueue: Fix affinity of kworkers when attaching into pool
> >
> > kernel/workqueue.c | 214 ++++++++++++++++++++++++++++-----------------
> > 1 file changed, 132 insertions(+), 82 deletions(-)
> >
> > --
> > 2.19.1.6.gb485710b
>
> Hi,
> I tested this patchset on today's tip.git's master branch
> (981316394e35 ("Merge branch 'locking/urgent'")).
>
> Every time the kernel boots with 32 CPUs (I'm running the Linux VM on
> Hyper-V), I get the below warning.
> (BTW, with 8 or 16 CPUs, I don't see the warning).
> By printing the cpumasks with "%*pbl", I know the warning happens because:
> new_mask = 16-31
> cpu_online_mask= 0-16
> cpu_active_mask= 0-15
> p->nr_cpus_allowed=16
>
> 2374 if (p->flags & PF_KTHREAD) {
> 2375 /*
> 2376 * For kernel threads that do indeed end up on online &&
> 2377 * !active we want to ensure they are strict
> per-CPU threads.
> 2378 */
> 2379 WARN_ON(cpumask_intersects(new_mask, cpu_online_mask) &&
> 2380 !cpumask_intersects(new_mask, cpu_active_mask) &&
> 2381 p->nr_cpus_allowed != 1);
> 2382 }
> 2383
>
Hello, Dexuan
Could you omit patch4 of the patchset and test it again, please?
("workqueue: don't set the worker's cpumask when kthread_bind_mask()")
kthread_bind_mask() set the worker task to the pool's cpumask without
any check. And set_cpus_allowed_ptr() finds that the task's cpumask
is unchanged (already set by kthread_bind_mask()) and skips all the checks.
And I found that numa=fake=2U seems broken on cpumask_of_node() in my box.
Thanks,
Lai
