Re: [PATCH 05/10] smp: Fast path check on IPI list
On Tue, Jul 29, 2014 at 02:07:39PM +0200, Peter Zijlstra wrote: > On Mon, Jul 28, 2014 at 07:37:31PM +0200, Frederic Weisbecker wrote: > > When we enqueue a remote irq work, we trigger the same IPI as those > > raised by smp_call_function_*() family. > > > > So when we receive such IPI, we check both irq_work and smp_call_function > > queues. Thus if we trigger a remote irq work, we'll likely find the > > smp_call_function queue empty unless we collide with concurrent enqueuers > > but the probability is low. > > > > Meanwhile, checking the smp_call_function queue can be costly because > > we use llist_del_all() which relies on cmpxchg(). > > xchg() It's still costly though as it's an atomic read/write with full barrier. I'll update the comment meanwhile. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 05/10] smp: Fast path check on IPI list
On Tue, Jul 29, 2014 at 02:07:39PM +0200, Peter Zijlstra wrote: > On Mon, Jul 28, 2014 at 07:37:31PM +0200, Frederic Weisbecker wrote: > > When we enqueue a remote irq work, we trigger the same IPI as those > > raised by smp_call_function_*() family. > > > > So when we receive such IPI, we check both irq_work and smp_call_function > > queues. Thus if we trigger a remote irq work, we'll likely find the > > smp_call_function queue empty unless we collide with concurrent enqueuers > > but the probability is low. > > > > Meanwhile, checking the smp_call_function queue can be costly because > > we use llist_del_all() which relies on cmpxchg(). > > xchg() Ah, right! -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 05/10] smp: Fast path check on IPI list
On Mon, Jul 28, 2014 at 07:37:31PM +0200, Frederic Weisbecker wrote: > When we enqueue a remote irq work, we trigger the same IPI as those > raised by smp_call_function_*() family. > > So when we receive such IPI, we check both irq_work and smp_call_function > queues. Thus if we trigger a remote irq work, we'll likely find the > smp_call_function queue empty unless we collide with concurrent enqueuers > but the probability is low. > > Meanwhile, checking the smp_call_function queue can be costly because > we use llist_del_all() which relies on cmpxchg(). xchg() -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 05/10] smp: Fast path check on IPI list
On Mon, Jul 28, 2014 at 07:37:31PM +0200, Frederic Weisbecker wrote: When we enqueue a remote irq work, we trigger the same IPI as those raised by smp_call_function_*() family. So when we receive such IPI, we check both irq_work and smp_call_function queues. Thus if we trigger a remote irq work, we'll likely find the smp_call_function queue empty unless we collide with concurrent enqueuers but the probability is low. Meanwhile, checking the smp_call_function queue can be costly because we use llist_del_all() which relies on cmpxchg(). xchg() -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 05/10] smp: Fast path check on IPI list
On Tue, Jul 29, 2014 at 02:07:39PM +0200, Peter Zijlstra wrote: On Mon, Jul 28, 2014 at 07:37:31PM +0200, Frederic Weisbecker wrote: When we enqueue a remote irq work, we trigger the same IPI as those raised by smp_call_function_*() family. So when we receive such IPI, we check both irq_work and smp_call_function queues. Thus if we trigger a remote irq work, we'll likely find the smp_call_function queue empty unless we collide with concurrent enqueuers but the probability is low. Meanwhile, checking the smp_call_function queue can be costly because we use llist_del_all() which relies on cmpxchg(). xchg() Ah, right! -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 05/10] smp: Fast path check on IPI list
On Tue, Jul 29, 2014 at 02:07:39PM +0200, Peter Zijlstra wrote: On Mon, Jul 28, 2014 at 07:37:31PM +0200, Frederic Weisbecker wrote: When we enqueue a remote irq work, we trigger the same IPI as those raised by smp_call_function_*() family. So when we receive such IPI, we check both irq_work and smp_call_function queues. Thus if we trigger a remote irq work, we'll likely find the smp_call_function queue empty unless we collide with concurrent enqueuers but the probability is low. Meanwhile, checking the smp_call_function queue can be costly because we use llist_del_all() which relies on cmpxchg(). xchg() It's still costly though as it's an atomic read/write with full barrier. I'll update the comment meanwhile. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH 05/10] smp: Fast path check on IPI list
When we enqueue a remote irq work, we trigger the same IPI as those
raised by smp_call_function_*() family.
So when we receive such IPI, we check both irq_work and smp_call_function
queues. Thus if we trigger a remote irq work, we'll likely find the
smp_call_function queue empty unless we collide with concurrent enqueuers
but the probability is low.
Meanwhile, checking the smp_call_function queue can be costly because
we use llist_del_all() which relies on cmpxchg().
We can reduce this overhead by doing a fast path check with llist_empty().
Given the implicit IPI ordering:
EnqueuerDequeuer
-
llist_add(csd, queue) get_IPI() {
send_IPI() if (llist_empty(queue)
...
When the IPI is sent, we are guaranteed that the IPI receiver will
see the new csd.
So lets do the fast path check to optimize non smp_call_function() related
jobs.
Cc: Ingo Molnar
Cc: Nicolas Pitre
Cc: Paul E. McKenney
Cc: Peter Zijlstra
Cc: Steven Rostedt
Cc: Thomas Gleixner
Cc: Viresh Kumar
Signed-off-by: Frederic Weisbecker
---
kernel/smp.c | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/kernel/smp.c b/kernel/smp.c
index 12c5bea..09ddc92 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -184,11 +184,19 @@ static int generic_exec_single(int cpu, struct
call_single_data *csd,
*/
void generic_smp_call_function_single_interrupt(void)
{
+ struct llist_head *head = &__get_cpu_var(call_single_queue);
struct llist_node *entry;
struct call_single_data *csd, *csd_next;
static bool warned;
- entry = llist_del_all(&__get_cpu_var(call_single_queue));
+ /*
+* Fast check: in case of irq work remote queue, the IPI list
+* is likely empty. We can spare the expensive llist_del_all().
+*/
+ if (llist_empty(head))
+ goto irq_work;
+
+ entry = llist_del_all(head);
entry = llist_reverse_order(entry);
/*
@@ -212,6 +220,7 @@ void generic_smp_call_function_single_interrupt(void)
csd_unlock(csd);
}
+irq_work:
/*
* Handle irq works queued remotely by irq_work_queue_on().
* Smp functions above are typically synchronous so they
--
1.8.3.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[PATCH 05/10] smp: Fast path check on IPI list
When we enqueue a remote irq work, we trigger the same IPI as those
raised by smp_call_function_*() family.
So when we receive such IPI, we check both irq_work and smp_call_function
queues. Thus if we trigger a remote irq work, we'll likely find the
smp_call_function queue empty unless we collide with concurrent enqueuers
but the probability is low.
Meanwhile, checking the smp_call_function queue can be costly because
we use llist_del_all() which relies on cmpxchg().
We can reduce this overhead by doing a fast path check with llist_empty().
Given the implicit IPI ordering:
EnqueuerDequeuer
-
llist_add(csd, queue) get_IPI() {
send_IPI() if (llist_empty(queue)
...
When the IPI is sent, we are guaranteed that the IPI receiver will
see the new csd.
So lets do the fast path check to optimize non smp_call_function() related
jobs.
Cc: Ingo Molnar mi...@kernel.org
Cc: Nicolas Pitre nicolas.pi...@linaro.org
Cc: Paul E. McKenney paul...@linux.vnet.ibm.com
Cc: Peter Zijlstra pet...@infradead.org
Cc: Steven Rostedt rost...@goodmis.org
Cc: Thomas Gleixner t...@linutronix.de
Cc: Viresh Kumar viresh.ku...@linaro.org
Signed-off-by: Frederic Weisbecker fweis...@gmail.com
---
kernel/smp.c | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/kernel/smp.c b/kernel/smp.c
index 12c5bea..09ddc92 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -184,11 +184,19 @@ static int generic_exec_single(int cpu, struct
call_single_data *csd,
*/
void generic_smp_call_function_single_interrupt(void)
{
+ struct llist_head *head = __get_cpu_var(call_single_queue);
struct llist_node *entry;
struct call_single_data *csd, *csd_next;
static bool warned;
- entry = llist_del_all(__get_cpu_var(call_single_queue));
+ /*
+* Fast check: in case of irq work remote queue, the IPI list
+* is likely empty. We can spare the expensive llist_del_all().
+*/
+ if (llist_empty(head))
+ goto irq_work;
+
+ entry = llist_del_all(head);
entry = llist_reverse_order(entry);
/*
@@ -212,6 +220,7 @@ void generic_smp_call_function_single_interrupt(void)
csd_unlock(csd);
}
+irq_work:
/*
* Handle irq works queued remotely by irq_work_queue_on().
* Smp functions above are typically synchronous so they
--
1.8.3.1
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[PATCH 05/10] smp: Fast path check on IPI list
When we enqueue a remote irq work, we trigger the same IPI as those
raised by smp_call_function_*() family.
So when we receive such IPI, we check both irq_work and smp_call_function
queues. Thus if we trigger a remote irq work, we'll likely find the
smp_call_function queue empty unless we collide with concurrent enqueuers
but the probability is low.
Meanwhile, checking the smp_call_function queue can be costly because
we use llist_del_all() which relies on cmpxchg().
We can reduce this overhead by doing a fast path check with llist_empty().
Given the implicit IPI ordering:
EnqueuerDequeuer
-
llist_add(csd, queue) get_IPI() {
send_IPI() if (llist_empty(queue)
...
When the IPI is sent, we are guaranteed that the IPI receiver will
see the new csd.
So lets do the fast path check to optimize non smp_call_function() related
jobs.
Cc: Ingo Molnar
Cc: Paul E. McKenney
Cc: Peter Zijlstra
Cc: Steven Rostedt
Cc: Thomas Gleixner
Cc: Viresh Kumar
Signed-off-by: Frederic Weisbecker
---
kernel/smp.c | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/kernel/smp.c b/kernel/smp.c
index a1812d1..34378d4 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -184,11 +184,19 @@ static int generic_exec_single(int cpu, struct
call_single_data *csd,
*/
void generic_smp_call_function_single_interrupt(void)
{
+ struct llist_head *head = &__get_cpu_var(call_single_queue);
struct llist_node *entry;
struct call_single_data *csd, *csd_next;
static bool warned;
- entry = llist_del_all(&__get_cpu_var(call_single_queue));
+ /*
+* Fast check: in case of irq work remote queue, the IPI list
+* is likely empty. We can spare the expensive llist_del_all().
+*/
+ if (llist_empty(head))
+ goto irq_work;
+
+ entry = llist_del_all(head);
entry = llist_reverse_order(entry);
/*
@@ -212,6 +220,7 @@ void generic_smp_call_function_single_interrupt(void)
csd_unlock(csd);
}
+irq_work:
/*
* Handle irq works queued remotely by irq_work_queue_on().
* Smp functions above are typically synchronous so they
--
1.8.3.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[PATCH 05/10] smp: Fast path check on IPI list
When we enqueue a remote irq work, we trigger the same IPI as those
raised by smp_call_function_*() family.
So when we receive such IPI, we check both irq_work and smp_call_function
queues. Thus if we trigger a remote irq work, we'll likely find the
smp_call_function queue empty unless we collide with concurrent enqueuers
but the probability is low.
Meanwhile, checking the smp_call_function queue can be costly because
we use llist_del_all() which relies on cmpxchg().
We can reduce this overhead by doing a fast path check with llist_empty().
Given the implicit IPI ordering:
EnqueuerDequeuer
-
llist_add(csd, queue) get_IPI() {
send_IPI() if (llist_empty(queue)
...
When the IPI is sent, we are guaranteed that the IPI receiver will
see the new csd.
So lets do the fast path check to optimize non smp_call_function() related
jobs.
Cc: Ingo Molnar mi...@kernel.org
Cc: Paul E. McKenney paul...@linux.vnet.ibm.com
Cc: Peter Zijlstra pet...@infradead.org
Cc: Steven Rostedt rost...@goodmis.org
Cc: Thomas Gleixner t...@linutronix.de
Cc: Viresh Kumar viresh.ku...@linaro.org
Signed-off-by: Frederic Weisbecker fweis...@gmail.com
---
kernel/smp.c | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/kernel/smp.c b/kernel/smp.c
index a1812d1..34378d4 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -184,11 +184,19 @@ static int generic_exec_single(int cpu, struct
call_single_data *csd,
*/
void generic_smp_call_function_single_interrupt(void)
{
+ struct llist_head *head = __get_cpu_var(call_single_queue);
struct llist_node *entry;
struct call_single_data *csd, *csd_next;
static bool warned;
- entry = llist_del_all(__get_cpu_var(call_single_queue));
+ /*
+* Fast check: in case of irq work remote queue, the IPI list
+* is likely empty. We can spare the expensive llist_del_all().
+*/
+ if (llist_empty(head))
+ goto irq_work;
+
+ entry = llist_del_all(head);
entry = llist_reverse_order(entry);
/*
@@ -212,6 +220,7 @@ void generic_smp_call_function_single_interrupt(void)
csd_unlock(csd);
}
+irq_work:
/*
* Handle irq works queued remotely by irq_work_queue_on().
* Smp functions above are typically synchronous so they
--
1.8.3.1
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
