Make smp_call_function_single_async() a safer and more convenient interface by using an atomic op for setting CSD_FLAG_LOCK. This allows safe concurrent use of this function as would be expected by the -EBUSY return value.
Signed-off-by: Peter Zijlstra (Intel) <[email protected]> --- kernel/smp.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-) --- a/kernel/smp.c +++ b/kernel/smp.c @@ -297,6 +297,13 @@ static void flush_smp_call_function_queu void *info = csd->info; csd_unlock(csd); + /* + * Ensures any LOAD of func() will be after + * the above UNLOCK, this guarantees that + * func() will observe any state prior + * to _async() returning -EBUSY. + */ + smp_mb(); func(info); } else if (type == CSD_TYPE_IRQ_WORK) { irq_work_single(csd); @@ -397,16 +404,18 @@ EXPORT_SYMBOL(smp_call_function_single); * can thus be done from contexts with disabled interrupts. * * The caller passes his own pre-allocated data structure - * (ie: embedded in an object) and is responsible for synchronizing it - * such that the IPIs performed on the @csd are strictly serialized. + * and is responsible for it's life-time, it must not be re-used + * until csd->node.u_flags == 0. * * If the function is called with one csd which has not yet been * processed by previous call to smp_call_function_single_async(), the * function will return immediately with -EBUSY showing that the csd * object is still in progress. * - * NOTE: Be careful, there is unfortunately no current debugging facility to - * validate the correctness of this serialization. + * When -EBUSY is returned, any invocation of csd->func() is guaranteed to see + * the state prior to this call. + * + * Also, consider using irq_work_queue_remote() if at all possible. */ int smp_call_function_single_async(int cpu, call_single_data_t *csd) { @@ -414,13 +423,16 @@ int smp_call_function_single_async(int c preempt_disable(); - if (csd->node.u_flags & CSD_FLAG_LOCK) { + /* + * We still need RELEASE like semantics, even when the cmpxchg() fails. + * Pairs with the smp_mb() in flush_smp_call_function_queue(). + */ + smp_mb__before_atomic(); + if (cmpxchg_relaxed(&csd->node.u_flags, 0, CSD_FLAG_LOCK) != 0) { err = -EBUSY; goto out; } - - csd->node.u_flags = CSD_FLAG_LOCK; - smp_wmb(); + /* ctrl-dep orders later stores */ err = generic_exec_single(cpu, csd);
