Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-10-11 Thread qiaozhou
Hi Will, On 2017年10月02日 22:14, Will Deacon wrote: Hi Qiao, On Mon, Sep 25, 2017 at 07:02:03PM +0800, qiaozhou wrote: Will this bodging patch be merged? It can solve the livelock issue on arm64 platforms(at least improve a lot). Whilst it seemed to help in some cases, I'm not keen to merge

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-10-11 Thread qiaozhou
Hi Will, On 2017年10月02日 22:14, Will Deacon wrote: Hi Qiao, On Mon, Sep 25, 2017 at 07:02:03PM +0800, qiaozhou wrote: Will this bodging patch be merged? It can solve the livelock issue on arm64 platforms(at least improve a lot). Whilst it seemed to help in some cases, I'm not keen to merge

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-10-02 Thread Will Deacon
Hi Qiao, On Mon, Sep 25, 2017 at 07:02:03PM +0800, qiaozhou wrote: > Will this bodging patch be merged? It can solve the livelock issue on arm64 > platforms(at least improve a lot). Whilst it seemed to help in some cases, I'm not keen to merge it until we've identified a sensible way to

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-10-02 Thread Will Deacon
Hi Qiao, On Mon, Sep 25, 2017 at 07:02:03PM +0800, qiaozhou wrote: > Will this bodging patch be merged? It can solve the livelock issue on arm64 > platforms(at least improve a lot). Whilst it seemed to help in some cases, I'm not keen to merge it until we've identified a sensible way to

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-09-25 Thread qiaozhou
Hi Will, Will this bodging patch be merged? It can solve the livelock issue on arm64 platforms(at least improve a lot). I suspected that CCI-freq might impact the contention between little and big core, but on my platform, it impacts little. In fact the frequency of external DDR controller

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-09-25 Thread qiaozhou
Hi Will, Will this bodging patch be merged? It can solve the livelock issue on arm64 platforms(at least improve a lot). I suspected that CCI-freq might impact the contention between little and big core, but on my platform, it impacts little. In fact the frequency of external DDR controller

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-09-06 Thread qiaozhou
On 2017年08月29日 07:12, Vikram Mulukutla wrote: Well here's something interesting. I tried a different platform and found that the workaround doesn't help much at all, similar to Qiao's observation on his b.L chipset. Something to do with the WFE implementation or event-stream? Hi Vikram,

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-09-06 Thread qiaozhou
On 2017年08月29日 07:12, Vikram Mulukutla wrote: Well here's something interesting. I tried a different platform and found that the workaround doesn't help much at all, similar to Qiao's observation on his b.L chipset. Something to do with the WFE implementation or event-stream? Hi Vikram,

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-28 Thread Vikram Mulukutla
Hi Will, On 2017-08-25 12:48, Vikram Mulukutla wrote: Hi Will, On 2017-08-15 11:40, Will Deacon wrote: Hi Vikram, On Thu, Aug 03, 2017 at 04:25:12PM -0700, Vikram Mulukutla wrote: On 2017-07-31 06:13, Will Deacon wrote: >On Fri, Jul 28, 2017 at 12:09:38PM -0700, Vikram Mulukutla wrote: >>On

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-28 Thread Vikram Mulukutla
Hi Will, On 2017-08-25 12:48, Vikram Mulukutla wrote: Hi Will, On 2017-08-15 11:40, Will Deacon wrote: Hi Vikram, On Thu, Aug 03, 2017 at 04:25:12PM -0700, Vikram Mulukutla wrote: On 2017-07-31 06:13, Will Deacon wrote: >On Fri, Jul 28, 2017 at 12:09:38PM -0700, Vikram Mulukutla wrote: >>On

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-25 Thread Vikram Mulukutla
On 2017-08-25 12:48, Vikram Mulukutla wrote: If I understand the code correctly, the upper 32 bits of an ARM64 virtual address will overflow when 1 is added to it, and so we'll keep WFE'ing on every subsequent cpu_relax invoked from the same PC, until we cross the hard-coded threshold,

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-25 Thread Vikram Mulukutla
On 2017-08-25 12:48, Vikram Mulukutla wrote: If I understand the code correctly, the upper 32 bits of an ARM64 virtual address will overflow when 1 is added to it, and so we'll keep WFE'ing on every subsequent cpu_relax invoked from the same PC, until we cross the hard-coded threshold,

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-25 Thread Vikram Mulukutla
Hi Will, On 2017-08-15 11:40, Will Deacon wrote: Hi Vikram, On Thu, Aug 03, 2017 at 04:25:12PM -0700, Vikram Mulukutla wrote: On 2017-07-31 06:13, Will Deacon wrote: >On Fri, Jul 28, 2017 at 12:09:38PM -0700, Vikram Mulukutla wrote: >>On 2017-07-28 02:28, Will Deacon wrote: >>>On Thu, Jul

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-25 Thread Vikram Mulukutla
Hi Will, On 2017-08-15 11:40, Will Deacon wrote: Hi Vikram, On Thu, Aug 03, 2017 at 04:25:12PM -0700, Vikram Mulukutla wrote: On 2017-07-31 06:13, Will Deacon wrote: >On Fri, Jul 28, 2017 at 12:09:38PM -0700, Vikram Mulukutla wrote: >>On 2017-07-28 02:28, Will Deacon wrote: >>>On Thu, Jul

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-15 Thread Will Deacon
Hi Vikram, On Thu, Aug 03, 2017 at 04:25:12PM -0700, Vikram Mulukutla wrote: > On 2017-07-31 06:13, Will Deacon wrote: > >On Fri, Jul 28, 2017 at 12:09:38PM -0700, Vikram Mulukutla wrote: > >>On 2017-07-28 02:28, Will Deacon wrote: > >>>On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-15 Thread Will Deacon
Hi Vikram, On Thu, Aug 03, 2017 at 04:25:12PM -0700, Vikram Mulukutla wrote: > On 2017-07-31 06:13, Will Deacon wrote: > >On Fri, Jul 28, 2017 at 12:09:38PM -0700, Vikram Mulukutla wrote: > >>On 2017-07-28 02:28, Will Deacon wrote: > >>>On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-03 Thread qiaozhou
On 2017年08月04日 07:32, Vikram Mulukutla wrote: Hi Qiao, On 2017-08-01 00:37, qiaozhou wrote: On 2017年07月31日 19:20, qiaozhou wrote: = Also apply Vikram's patch and have a test. cpu2: a53, 832MHz, cpu7: a73, 1.75Hz Without

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-03 Thread qiaozhou
On 2017年08月04日 07:32, Vikram Mulukutla wrote: Hi Qiao, On 2017-08-01 00:37, qiaozhou wrote: On 2017年07月31日 19:20, qiaozhou wrote: = Also apply Vikram's patch and have a test. cpu2: a53, 832MHz, cpu7: a73, 1.75Hz Without

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-03 Thread Vikram Mulukutla
Hi Qiao, On 2017-08-01 00:37, qiaozhou wrote: On 2017年07月31日 19:20, qiaozhou wrote: = Also apply Vikram's patch and have a test. cpu2: a53, 832MHz, cpu7: a73, 1.75Hz Without cpu_relax bodging patch

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-03 Thread Vikram Mulukutla
Hi Qiao, On 2017-08-01 00:37, qiaozhou wrote: On 2017年07月31日 19:20, qiaozhou wrote: = Also apply Vikram's patch and have a test. cpu2: a53, 832MHz, cpu7: a73, 1.75Hz Without cpu_relax bodging patch

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-03 Thread Vikram Mulukutla
Hi Will, On 2017-07-31 06:13, Will Deacon wrote: Hi Vikram, On Fri, Jul 28, 2017 at 12:09:38PM -0700, Vikram Mulukutla wrote: On 2017-07-28 02:28, Will Deacon wrote: >On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: > This does seem to help. Here's some data after 5 runs

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-03 Thread Vikram Mulukutla
Hi Will, On 2017-07-31 06:13, Will Deacon wrote: Hi Vikram, On Fri, Jul 28, 2017 at 12:09:38PM -0700, Vikram Mulukutla wrote: On 2017-07-28 02:28, Will Deacon wrote: >On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: > This does seem to help. Here's some data after 5 runs

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-01 Thread qiaozhou
On 2017年07月31日 19:20, qiaozhou wrote: On 2017年07月29日 03:09, Vikram Mulukutla wrote: On 2017-07-28 02:28, Will Deacon wrote: On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: Does bodging cpu_relax to back-off to wfe after a while help? The event stream will wake it up

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-08-01 Thread qiaozhou
On 2017年07月31日 19:20, qiaozhou wrote: On 2017年07月29日 03:09, Vikram Mulukutla wrote: On 2017-07-28 02:28, Will Deacon wrote: On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: Does bodging cpu_relax to back-off to wfe after a while help? The event stream will wake it up

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-31 Thread Will Deacon
Hi Vikram, On Fri, Jul 28, 2017 at 12:09:38PM -0700, Vikram Mulukutla wrote: > On 2017-07-28 02:28, Will Deacon wrote: > >On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: > > > > >> > >>I think we should have this discussion now - I brought this up earlier > >>[1] > >>and I

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-31 Thread Will Deacon
Hi Vikram, On Fri, Jul 28, 2017 at 12:09:38PM -0700, Vikram Mulukutla wrote: > On 2017-07-28 02:28, Will Deacon wrote: > >On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: > > > > >> > >>I think we should have this discussion now - I brought this up earlier > >>[1] > >>and I

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-31 Thread qiaozhou
On 2017年07月29日 03:09, Vikram Mulukutla wrote: On 2017-07-28 02:28, Will Deacon wrote: On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: I think we should have this discussion now - I brought this up earlier [1] and I promised a test case that I completely forgot about

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-31 Thread qiaozhou
On 2017年07月29日 03:09, Vikram Mulukutla wrote: On 2017-07-28 02:28, Will Deacon wrote: On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: I think we should have this discussion now - I brought this up earlier [1] and I promised a test case that I completely forgot about

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-28 Thread Vikram Mulukutla
On 2017-07-28 02:28, Peter Zijlstra wrote: On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: I think we should have this discussion now - I brought this up earlier [1] and I promised a test case that I completely forgot about - but here it is (attached). Essentially a Big CPU

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-28 Thread Vikram Mulukutla
On 2017-07-28 02:28, Peter Zijlstra wrote: On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: I think we should have this discussion now - I brought this up earlier [1] and I promised a test case that I completely forgot about - but here it is (attached). Essentially a Big CPU

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-28 Thread Vikram Mulukutla
On 2017-07-28 02:28, Will Deacon wrote: On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: I think we should have this discussion now - I brought this up earlier [1] and I promised a test case that I completely forgot about - but here it is (attached). Essentially a Big

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-28 Thread Vikram Mulukutla
On 2017-07-28 02:28, Will Deacon wrote: On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: I think we should have this discussion now - I brought this up earlier [1] and I promised a test case that I completely forgot about - but here it is (attached). Essentially a Big

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-28 Thread Will Deacon
On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: > On 2017-07-26 18:29, qiaozhou wrote: > >On 2017年07月26日 22:16, Thomas Gleixner wrote: > >>On Wed, 26 Jul 2017, qiaozhou wrote: > >>For that particular timer case we can clear base->running_timer w/o the > >>lock held (see patch

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-28 Thread Peter Zijlstra
On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: > I think we should have this discussion now - I brought this up earlier [1] > and I promised a test case that I completely forgot about - but here it > is (attached). Essentially a Big CPU in an acquire-check-release loop > will

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-28 Thread Will Deacon
On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: > On 2017-07-26 18:29, qiaozhou wrote: > >On 2017年07月26日 22:16, Thomas Gleixner wrote: > >>On Wed, 26 Jul 2017, qiaozhou wrote: > >>For that particular timer case we can clear base->running_timer w/o the > >>lock held (see patch

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-28 Thread Peter Zijlstra
On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: > I think we should have this discussion now - I brought this up earlier [1] > and I promised a test case that I completely forgot about - but here it > is (attached). Essentially a Big CPU in an acquire-check-release loop > will

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-27 Thread Vikram Mulukutla
cc: Sudeep Holla On 2017-07-26 18:29, qiaozhou wrote: On 2017年07月26日 22:16, Thomas Gleixner wrote: On Wed, 26 Jul 2017, qiaozhou wrote: Cc'ed ARM folks. For that particular timer case we can clear base->running_timer w/o the lock held (see patch below), but this kind of lock

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-27 Thread Vikram Mulukutla
cc: Sudeep Holla On 2017-07-26 18:29, qiaozhou wrote: On 2017年07月26日 22:16, Thomas Gleixner wrote: On Wed, 26 Jul 2017, qiaozhou wrote: Cc'ed ARM folks. For that particular timer case we can clear base->running_timer w/o the lock held (see patch below), but this kind of lock

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-27 Thread Thomas Gleixner
On Thu, 27 Jul 2017, Will Deacon wrote: > On Thu, Jul 27, 2017 at 09:29:20AM +0800, qiaozhou wrote: > > On 2017年07月26日 22:16, Thomas Gleixner wrote: > > >--- a/kernel/time/timer.c > > >+++ b/kernel/time/timer.c > > >@@ -1301,10 +1301,12 @@ static void expire_timers(struct timer_b > > >

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-27 Thread Thomas Gleixner
On Thu, 27 Jul 2017, Will Deacon wrote: > On Thu, Jul 27, 2017 at 09:29:20AM +0800, qiaozhou wrote: > > On 2017年07月26日 22:16, Thomas Gleixner wrote: > > >--- a/kernel/time/timer.c > > >+++ b/kernel/time/timer.c > > >@@ -1301,10 +1301,12 @@ static void expire_timers(struct timer_b > > >

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-27 Thread Will Deacon
On Thu, Jul 27, 2017 at 09:29:20AM +0800, qiaozhou wrote: > On 2017年07月26日 22:16, Thomas Gleixner wrote: > >--- a/kernel/time/timer.c > >+++ b/kernel/time/timer.c > >@@ -1301,10 +1301,12 @@ static void expire_timers(struct timer_b > > if (timer->flags & TIMER_IRQSAFE) { > >

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-27 Thread Will Deacon
On Thu, Jul 27, 2017 at 09:29:20AM +0800, qiaozhou wrote: > On 2017年07月26日 22:16, Thomas Gleixner wrote: > >--- a/kernel/time/timer.c > >+++ b/kernel/time/timer.c > >@@ -1301,10 +1301,12 @@ static void expire_timers(struct timer_b > > if (timer->flags & TIMER_IRQSAFE) { > >

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-26 Thread qiaozhou
On 2017年07月26日 22:16, Thomas Gleixner wrote: On Wed, 26 Jul 2017, qiaozhou wrote: Cc'ed ARM folks. I want to ask you for suggestions about how to fix one contention between expire_timers and try_to_del_timer_sync. Thanks in advance. The issue is a hard-lockup issue detected on our

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-26 Thread qiaozhou
On 2017年07月26日 22:16, Thomas Gleixner wrote: On Wed, 26 Jul 2017, qiaozhou wrote: Cc'ed ARM folks. I want to ask you for suggestions about how to fix one contention between expire_timers and try_to_del_timer_sync. Thanks in advance. The issue is a hard-lockup issue detected on our

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-26 Thread Thomas Gleixner
On Wed, 26 Jul 2017, qiaozhou wrote: Cc'ed ARM folks. > I want to ask you for suggestions about how to fix one contention between > expire_timers and try_to_del_timer_sync. Thanks in advance. > The issue is a hard-lockup issue detected on our platform(arm64, one cluster > with 4 a53, and the

Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

2017-07-26 Thread Thomas Gleixner
On Wed, 26 Jul 2017, qiaozhou wrote: Cc'ed ARM folks. > I want to ask you for suggestions about how to fix one contention between > expire_timers and try_to_del_timer_sync. Thanks in advance. > The issue is a hard-lockup issue detected on our platform(arm64, one cluster > with 4 a53, and the