Re: cpu stopper threads and setaffinity leads to deadlock

[Peter Zijlstra]

On Wed, Aug 01, 2018 at 06:34:40PM -0700, Sodagudi Prasad wrote:
> Due to cross migration of tasks between cpu7 and cpu3, migration/7 has
> started executing and waits for the migration/3 task, so that they can
> proceed within the multi cpu stop state machine together.
> Unfortunately stress-ng-affin is affine to cpu7, and since migration 7 has
> started running, and has monopolized cpu7’s execution, stress-ng will never
> run on cpu7, and cpu3’s migration task is never woken up.

> diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
> index e190d1e..f932e1e 100644
> --- a/kernel/stop_machine.c
> +++ b/kernel/stop_machine.c
> @@ -87,9 +87,9 @@ static bool cpu_stop_queue_work(unsigned int cpu, struct
> cpu_stop_work *work)
>                 __cpu_stop_queue_work(stopper, work, &wakeq);
>         else if (work->done)
>                 cpu_stop_signal_done(work->done);
> -       raw_spin_unlock_irqrestore(&stopper->lock, flags);
> 
>         wake_up_q(&wakeq);
> +       raw_spin_unlock_irqrestore(&stopper->lock, flags);
> 

So why didn't you do the 'obvious' parallel to what you did for
cpu_stop_queue_two_works(), namely:

--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -81,6 +81,7 @@ static bool cpu_stop_queue_work(unsigned
        unsigned long flags;
        bool enabled;
 
+       preempt_disable();
        raw_spin_lock_irqsave(&stopper->lock, flags);
        enabled = stopper->enabled;
        if (enabled)
@@ -90,6 +91,7 @@ static bool cpu_stop_queue_work(unsigned
        raw_spin_unlock_irqrestore(&stopper->lock, flags);
 
        wake_up_q(&wakeq);
+       preempt_enable();
 
        return enabled;
 }