Re: [PATCH] sched/fair: use dst group while checking imbalance for NUMA balancer

at 11:50:04PM +0200, Jirka Hladky wrote: > > 1) Threads stability has improved a lot. We see much fewer threads > > migrations. Check for example this heatmap based on the mpstat results > > collected while running sp.C test from the NAS benchmark. sp.C was run > > with 1

Re: [PATCH] sched/fair: use dst group while checking imbalance for NUMA balancer

Hi Hilf and Mel, thanks a lot for bringing this to my attention. We have tested the proposed patch and we are getting excellent results so far! 1) Threads stability has improved a lot. We see much fewer threads migrations. Check for example this heatmap based on the mpstat results collected

Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load balancer v6

Wed, May 20, 2020 at 3:58 PM Jirka Hladky wrote: > > Hi Hillf, Mel and all, > > thanks for the patch! It has produced really GOOD results. > > 1) It has fixed performance problems with 5.7 vanilla kernel for > single-tenant workload and low system load scenarios, without &

Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load balancer v6

would like to check the patch thoroughly to make sure it does not hurt performance in other areas. Thanks a lot! Jirka On Tue, May 19, 2020 at 6:32 AM Hillf Danton wrote: > > > Hi Jirka > > On Mon, 18 May 2020 16:52:52 +0200 Jirka Hladky wrote: > > > >

Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load balancer v6

ote: > > > On Thu, 7 May 2020 13:49:34 Phil Auld wrote: > > > > On Thu, May 07, 2020 at 06:29:44PM +0200 Jirka Hladky wrote: > > > Hi Mel, > > > > > > we are not targeting just OMP applications. We see the performance > > > degradation also for o

Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load balancer v6

man wrote: > > On Wed, May 13, 2020 at 04:57:15PM +0200, Jirka Hladky wrote: > > Hi Mel, > > > > we have tried the kernel with adjust_numa_imbalance() crippled to just > > return the imbalance it's given. > > > > It has solved all the performance problems

Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load balancer v6

THANK YOU! On Thu, May 14, 2020 at 1:50 PM Mel Gorman wrote: > > On Thu, May 14, 2020 at 12:22:05PM +0200, Jirka Hladky wrote: > > Thanks! > > > > Do you have a link? I cannot find it on github > > (https://github.com/gormanm/mmtests, searched for > > confi

Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load balancer v6

Thanks! Do you have a link? I cannot find it on github (https://github.com/gormanm/mmtests, searched for config-network-netperf-cstate-small-cross-socket) On Thu, May 14, 2020 at 12:08 PM Mel Gorman wrote: > > On Thu, May 14, 2020 at 11:58:36AM +0200, Jirka Hladky wrote: > > Th

Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load balancer v6

://github.com/gormanm/mmtests and we use lots of the same benchmarks but I'm not sure if we cover this particular scenario. Jirka On Wed, May 13, 2020 at 5:30 PM Mel Gorman wrote: > > On Wed, May 13, 2020 at 04:57:15PM +0200, Jirka Hladky wrote: > > Hi Mel, > > > >

Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load balancer v6

Any suggestions on how to proceed? One approach is to turn "imbalance_min" into the kernel tunable. Any other ideas? https://github.com/torvalds/linux/blob/4f8a3cc1183c442daee6cc65360e3385021131e4/kernel/sched/fair.c#L8914 Thanks a lot! Jirka On Fri, May 8, 2020 at 12:40 PM Jirka Hladky wrot

Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load balancer v6

Hi Mel, thanks for hints! We will try it. @Phil - could you please prepare a kernel build for me to test? Thank you! Jirka On Fri, May 8, 2020 at 11:22 AM Mel Gorman wrote: > > On Thu, May 07, 2020 at 06:29:44PM +0200, Jirka Hladky wrote: > > Hi Mel, > > > > we ar

Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load balancer v6

then evaluate if it's the way to go. Thoughts? Thanks a lot! Jirka On Thu, May 7, 2020 at 5:54 PM Mel Gorman wrote: > > On Thu, May 07, 2020 at 05:24:17PM +0200, Jirka Hladky wrote: > > Hi Mel, > > > > > > Yes, it's indeed OMP. With low threads count, I mean up to 2x n

Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load balancer v6

profiles [1]. What do you think about this approach? Thanks a lot! Jirka [1] https://tuned-project.org On Fri, Mar 20, 2020 at 5:38 PM Mel Gorman wrote: > > On Fri, Mar 20, 2020 at 04:30:08PM +0100, Jirka Hladky wrote: > > > > > > MPI or OMP and what is a low thre

Group Imbalance Bug - performance drop by factor 10x on NUMA boxes with cgroups

Hi Mel and Srikar, I would like to ask you if you could look into the Group Imbalance Bug described in this paper http://www.ece.ubc.ca/~sasha/papers/eurosys16-final29.pdf in chapter 3.1. See also comment [1]. The paper describes the bug on workload which involves different ssh sessions and it

Group Imbalance Bug - performance drop by factor 10x on NUMA boxes with cgroups

Hi Mel and Srikar, I would like to ask you if you could look into the Group Imbalance Bug described in this paper http://www.ece.ubc.ca/~sasha/papers/eurosys16-final29.pdf in chapter 3.1. See also comment [1]. The paper describes the bug on workload which involves different ssh sessions and it

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

em. Sounds great, thank you! If you want me to retest the rebased patch set, just let me know, we would be more than happy to run the tests on our side.  Jirka On Mon, Sep 17, 2018 at 3:06 PM, Mel Gorman wrote: > On Fri, Sep 14, 2018 at 04:50:20PM +0200, Jirka Hladky wrote: >>

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

em. Sounds great, thank you! If you want me to retest the rebased patch set, just let me know, we would be more than happy to run the tests on our side.  Jirka On Mon, Sep 17, 2018 at 3:06 PM, Mel Gorman wrote: > On Fri, Sep 14, 2018 at 04:50:20PM +0200, Jirka Hladky wrote: >>

Re: [4.17 regression] Performance drop on kernel-4.17 visible on Stream, Linpack and NAS parallel benchmarks

be really great if both series could be merged together. Removing NUMA migration rate limit helps performance. Thanks a lot for your help on this! Jirka On Fri, Sep 7, 2018 at 10:09 AM, Jirka Hladky wrote: >> Maybe 305c1fac3225dfa7eeb89bfe91b7335a6edd5172. That introduces a weird >&g

Re: [4.17 regression] Performance drop on kernel-4.17 visible on Stream, Linpack and NAS parallel benchmarks

be really great if both series could be merged together. Removing NUMA migration rate limit helps performance. Thanks a lot for your help on this! Jirka On Fri, Sep 7, 2018 at 10:09 AM, Jirka Hladky wrote: >> Maybe 305c1fac3225dfa7eeb89bfe91b7335a6edd5172. That introduces a weird >&g

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

tchset could be merged into the upstream kernel? Thanks a lot! Jirka On Sun, Sep 9, 2018 at 4:03 PM, Jirka Hladky wrote: > > Hi Peter and Srikar, > > thanks a lot for the information and for the patches to test! > > > I have bounced the 5 patches to you, (one of the 6 has not bee

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

tchset could be merged into the upstream kernel? Thanks a lot! Jirka On Sun, Sep 9, 2018 at 4:03 PM, Jirka Hladky wrote: > > Hi Peter and Srikar, > > thanks a lot for the information and for the patches to test! > > > I have bounced the 5 patches to you, (one of the 6 has not bee

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

Hi Peter and Srikar, thanks a lot for the information and for the patches to test! > I have bounced the 5 patches to you, (one of the 6 has not been applied by > Peter) so I have skipped that. > They can also be fetched from >

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

Hi Peter and Srikar, thanks a lot for the information and for the patches to test! > I have bounced the 5 patches to you, (one of the 6 has not been applied by > Peter) so I have skipped that. > They can also be fetched from >

[SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

Hi Srikar, I work at Red Hat in the Kernel Performance Team. I would like to ask you for help. We have detected a significant performance drop (20% and more) with 4.19rc1 relatively to 4.18 vanilla. We see the regression on different 2 NUMA and 4 NUMA boxes with pretty much all the benchmarks we

[SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

Hi Srikar, I work at Red Hat in the Kernel Performance Team. I would like to ask you for help. We have detected a significant performance drop (20% and more) with 4.19rc1 relatively to 4.18 vanilla. We see the regression on different 2 NUMA and 4 NUMA boxes with pretty much all the benchmarks we

Re: [4.17 regression] Performance drop on kernel-4.17 visible on Stream, Linpack and NAS parallel benchmarks

> On Thu, Sep 06, 2018 at 10:16:28AM +0200, Jirka Hladky wrote: >> Hi Mel, >> >> we have results with 2d4056fafa196e1ab4e7161bae4df76f9602d56d reverted. >> >> * Compared to 4.18, there is still performance regression - >> especially with NAS (sp_C_

Re: [4.17 regression] Performance drop on kernel-4.17 visible on Stream, Linpack and NAS parallel benchmarks

> On Thu, Sep 06, 2018 at 10:16:28AM +0200, Jirka Hladky wrote: >> Hi Mel, >> >> we have results with 2d4056fafa196e1ab4e7161bae4df76f9602d56d reverted. >> >> * Compared to 4.18, there is still performance regression - >> especially with NAS (sp_C_

Re: [4.17 regression] Performance drop on kernel-4.17 visible on Stream, Linpack and NAS parallel benchmarks

contact Srikar and ask for the advice or should we contact him directly? Thanks a lot! Jirka On Tue, Sep 4, 2018 at 12:07 PM, Jirka Hladky wrote: > Hi Mel, > > thanks for sharing the background information! We will check if > 2d4056fafa196e1ab4e7161bae4df76f9602d56d is causing

Re: [4.17 regression] Performance drop on kernel-4.17 visible on Stream, Linpack and NAS parallel benchmarks

contact Srikar and ask for the advice or should we contact him directly? Thanks a lot! Jirka On Tue, Sep 4, 2018 at 12:07 PM, Jirka Hladky wrote: > Hi Mel, > > thanks for sharing the background information! We will check if > 2d4056fafa196e1ab4e7161bae4df76f9602d56d is causing

Re: [4.17 regression] Performance drop on kernel-4.17 visible on Stream, Linpack and NAS parallel benchmarks

200, Jirka Hladky wrote: >> Resending in the plain text mode. >> >> > My own testing completed and the results are within expectations and I >> > saw no red flags. Unfortunately, I consider it unlikely they'll be merged >> > for 4.18. Srikar Dronamraju's series i

Re: [4.17 regression] Performance drop on kernel-4.17 visible on Stream, Linpack and NAS parallel benchmarks

200, Jirka Hladky wrote: >> Resending in the plain text mode. >> >> > My own testing completed and the results are within expectations and I >> > saw no red flags. Unfortunately, I consider it unlikely they'll be merged >> > for 4.18. Srikar Dronamraju's series i

Re: [4.17 regression] Performance drop on kernel-4.17 visible on Stream, Linpack and NAS parallel benchmarks

ode-v1r12" to be merged or should we start looking at what has caused the drop in performance going from 4.19rc1 to 4.18? We would appreciate any guidance on how to proceed. Thanks a lot! Jirka On Mon, Sep 3, 2018 at 5:04 PM, Jirka Hladky wrote: >> My own testing completed and t

Re: [4.17 regression] Performance drop on kernel-4.17 visible on Stream, Linpack and NAS parallel benchmarks

ode-v1r12" to be merged or should we start looking at what has caused the drop in performance going from 4.19rc1 to 4.18? We would appreciate any guidance on how to proceed. Thanks a lot! Jirka On Mon, Sep 3, 2018 at 5:04 PM, Jirka Hladky wrote: >> My own testing completed and t

Group Imbalance bug - performance drop upto factor 10x

Hello, we observe that group imbalance bug can cause performance degradation upto factor 10x on 4 NUMA server. I have opened Bug 194231 https://bugzilla.kernel.org/show_bug.cgi?id=194231 for this issue. The problem was first described in this paper

Group Imbalance bug - performance drop upto factor 10x

Hello, we observe that group imbalance bug can cause performance degradation upto factor 10x on 4 NUMA server. I have opened Bug 194231 https://bugzilla.kernel.org/show_bug.cgi?id=194231 for this issue. The problem was first described in this paper

Re: [PATCH] x86/topology: Fallback to SMT level only once

Hi Peter, yes, initially I have reported the issue to occur on Intel E5v3 CPU (that CPU does not have CoD) but it has turned to be a fluctuation of results. After repeating the test 10 times it has turned out that Intel E5v3 CPU is not affected. I'm sorry for that. I have then rerun the test on

Re: [PATCH] x86/topology: Fallback to SMT level only once

Hi Peter, yes, initially I have reported the issue to occur on Intel E5v3 CPU (that CPU does not have CoD) but it has turned to be a fluctuation of results. After repeating the test 10 times it has turned out that Intel E5v3 CPU is not affected. I'm sorry for that. I have then rerun the test on

Re: Kernel v4.7-rc5 - performance degradation upto 40% after disabling and re-enabling a core

On Tue, Jul 12, 2016 at 11:04 AM, Jirka Hladky <jhla...@redhat.com> wrote: > Hi Peter, > > have you a chance to look into this? Is there anything I can do to > help you to fix it? > > Thanks a lot! > Jirka > > > On Wed, Jun 29, 2016 at 11:58 AM, Peter Zijlstra <p

Re: Kernel v4.7-rc5 - performance degradation upto 40% after disabling and re-enabling a core

On Tue, Jul 12, 2016 at 11:04 AM, Jirka Hladky wrote: > Hi Peter, > > have you a chance to look into this? Is there anything I can do to > help you to fix it? > > Thanks a lot! > Jirka > > > On Wed, Jun 29, 2016 at 11:58 AM, Peter Zijlstra wrote: >> On Wed, Jun

Re: Kernel v4.7-rc5 - performance degradation upto 40% after disabling and re-enabling a core

Hi Peter, have you a chance to look into this? Is there anything I can do to help you to fix it? Thanks a lot! Jirka On Wed, Jun 29, 2016 at 11:58 AM, Peter Zijlstra <pet...@infradead.org> wrote: > On Wed, Jun 29, 2016 at 11:47:56AM +0200, Jirka Hladky wrote: >> Hi Peter, >&g

Re: Kernel v4.7-rc5 - performance degradation upto 40% after disabling and re-enabling a core

Hi Peter, have you a chance to look into this? Is there anything I can do to help you to fix it? Thanks a lot! Jirka On Wed, Jun 29, 2016 at 11:58 AM, Peter Zijlstra wrote: > On Wed, Jun 29, 2016 at 11:47:56AM +0200, Jirka Hladky wrote: >> Hi Peter, >> >> I think Clus

Kernel v4.7-rc5 - performance degradation upto 40% after disabling and re-enabling a core

Hello, on NUMA enabled server equipped with 4 Intel E5-4610 v2 CPUs we observe following performance degradation: Runtime to run "lu.C.x" test from NAS Parallel Benchmarks after booting the kernel: real 1m57.834s user 113m51.520s Then we disable and re-enable one core: echo 0 >

Kernel v4.7-rc5 - performance degradation upto 40% after disabling and re-enabling a core

Hello, on NUMA enabled server equipped with 4 Intel E5-4610 v2 CPUs we observe following performance degradation: Runtime to run "lu.C.x" test from NAS Parallel Benchmarks after booting the kernel: real 1m57.834s user 113m51.520s Then we disable and re-enable one core: echo 0 >

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

Hi Peter, I have compiled your version of linux kernel and run the SPECjvm2008 tests. Results are fine, performance is at the level of 4.6 kernel. $ git rev-parse HEAD 02548776ded1185e6e16ad0a475481e982741ee9 Jirka On Fri, Jun 24, 2016 at 5:54 PM, Peter Zijlstra

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

Hi Peter, I have compiled your version of linux kernel and run the SPECjvm2008 tests. Results are fine, performance is at the level of 4.6 kernel. $ git rev-parse HEAD 02548776ded1185e6e16ad0a475481e982741ee9 Jirka On Fri, Jun 24, 2016 at 5:54 PM, Peter Zijlstra wrote: > On Fri, Jun 24,

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

te: >> On Fri, Jun 24, 2016 at 09:44:41AM +0200, Jirka Hladky wrote: >>> Hi Peter, >>> >>> thanks a lot for looking into it! >>> >>> I have tried to disable autogroups >>> >>> sysctl -w kernel.sched_autogroup_enabled=0 >>

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

Hi Peter, the proposed patch has fixed the performance issue. I have applied the patch to v4.7-rc4 Jirka On Fri, Jun 24, 2016 at 2:44 PM, Vincent Guittot wrote: > Hi Peter, > > On 24 June 2016 at 14:02, Peter Zijlstra wrote: >> On Fri, Jun 24, 2016 at 09:44:41AM +0200, Jir

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

OK, I have applied to v4.7-rc4 via git am Compiling kernel, should have the results soon. Jirka On Fri, Jun 24, 2016 at 2:09 PM, Jirka Hladky <jhla...@redhat.com> wrote: > Thank you Peter! > > Should I apply it to v4.7-rc4 ? > > Jirka > > On Fri, Jun 24, 2016 at 2

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

OK, I have applied to v4.7-rc4 via git am Compiling kernel, should have the results soon. Jirka On Fri, Jun 24, 2016 at 2:09 PM, Jirka Hladky wrote: > Thank you Peter! > > Should I apply it to v4.7-rc4 ? > > Jirka > > On Fri, Jun 24, 2016 at 2:02 PM, Peter Zijlstra wro

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

Thank you Peter! Should I apply it to v4.7-rc4 ? Jirka On Fri, Jun 24, 2016 at 2:02 PM, Peter Zijlstra <pet...@infradead.org> wrote: > On Fri, Jun 24, 2016 at 09:44:41AM +0200, Jirka Hladky wrote: >> Hi Peter, >> >> thanks a lot for looking into it! >> >

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

Thank you Peter! Should I apply it to v4.7-rc4 ? Jirka On Fri, Jun 24, 2016 at 2:02 PM, Peter Zijlstra wrote: > On Fri, Jun 24, 2016 at 09:44:41AM +0200, Jirka Hladky wrote: >> Hi Peter, >> >> thanks a lot for looking into it! >> >> I have tried to d

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

I had a look and CONFIG_SCHED_AUTOGROUP=y is used both in RHEL6 and RHEL7. We compile the upstream kernels with config derived from RHEL7 config file. Jirka On Fri, Jun 24, 2016 at 10:08 AM, Peter Zijlstra <pet...@infradead.org> wrote: > On Fri, Jun 24, 2016 at 09:44:41AM +0200, Jir

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

I had a look and CONFIG_SCHED_AUTOGROUP=y is used both in RHEL6 and RHEL7. We compile the upstream kernels with config derived from RHEL7 config file. Jirka On Fri, Jun 24, 2016 at 10:08 AM, Peter Zijlstra wrote: > On Fri, Jun 24, 2016 at 09:44:41AM +0200, Jirka Hladky wrote: >&g

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

ON both in 4.6 and 4.7 kernel. Jirka On Thu, Jun 23, 2016 at 8:43 PM, Peter Zijlstra <pet...@infradead.org> wrote: > On Thu, Jun 23, 2016 at 08:33:18PM +0200, Peter Zijlstra wrote: >> On Fri, Jun 17, 2016 at 01:04:23AM +0200, Jirka Hladky wrote: >> >> > > What kind of c

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

ON both in 4.6 and 4.7 kernel. Jirka On Thu, Jun 23, 2016 at 8:43 PM, Peter Zijlstra wrote: > On Thu, Jun 23, 2016 at 08:33:18PM +0200, Peter Zijlstra wrote: >> On Fri, Jun 17, 2016 at 01:04:23AM +0200, Jirka Hladky wrote: >> >> > > What kind of config and userspace s

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

/core, to pick up fixes before applying new changes This commit is bad: 2159197 - Peter Zijlstra, 8 weeks ago : sched/core: Enable increased load resolution on 64-bit kernels Could you please have a look? Thanks a lot! Jirka On Wed, Jun 22, 2016 at 2:46 PM, Jirka Hladky <jhla...@redhat.

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

/core, to pick up fixes before applying new changes This commit is bad: 2159197 - Peter Zijlstra, 8 weeks ago : sched/core: Enable increased load resolution on 64-bit kernels Could you please have a look? Thanks a lot! Jirka On Wed, Jun 22, 2016 at 2:46 PM, Jirka Hladky wrote: > OK, I h

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

, Jun 22, 2016 at 2:37 PM, Jirka Hladky <jhla...@redhat.com> wrote: > Hi Peter, > > crap - I have done bisecting manually (not using git bisect) and I > have probably done some mistake. > > Commits (git checkout ) for which I got BAD results: > > 2159197d66

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

, Jun 22, 2016 at 2:37 PM, Jirka Hladky wrote: > Hi Peter, > > crap - I have done bisecting manually (not using git bisect) and I > have probably done some mistake. > > Commits (git checkout ) for which I got BAD results: > > 2159197d66770ec0

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

results: 21e96f88776deead303ecd30a17d1d7c2a1776e3 64b7aad579847852e110878ccaae4c3aaa34 e7904a28f5331c21d17af638cb477c83662e3cb6 I will try to use git bisect now.  Jirka On Wed, Jun 22, 2016 at 1:12 PM, Peter Zijlstra <pet...@infradead.org> wrote: > On Wed, Jun 22, 2016 at 11:52:45AM +02

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

results: 21e96f88776deead303ecd30a17d1d7c2a1776e3 64b7aad579847852e110878ccaae4c3aaa34 e7904a28f5331c21d17af638cb477c83662e3cb6 I will try to use git bisect now.  Jirka On Wed, Jun 22, 2016 at 1:12 PM, Peter Zijlstra wrote: > On Wed, Jun 22, 2016 at 11:52:45AM +0200, Jirka Hladky wrote: &g

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

Hi Peter, the performance regression has been caused by this commit = commit 6ecdd74962f246dfe8750b7bea481a1c0816315d Author: Yuyang Du Date: Tue Apr 5 12:12:26 2016 +0800 sched/fair: Generalize the load/util averages

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

Hi Peter, the performance regression has been caused by this commit = commit 6ecdd74962f246dfe8750b7bea481a1c0816315d Author: Yuyang Du Date: Tue Apr 5 12:12:26 2016 +0800 sched/fair: Generalize the load/util averages resolution definition

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

imo...@gmail.com> wrote: > Could it be related to this: > > https://www.phoronix.com/scan.php?page=news_item=P-State-Possible-4.6-Regression > > > On Thu, 16 Jun 2016 18:40:01 +0200 > Jirka Hladky <jhla...@redhat.com> wrote: > >> Hello, >> >> we see p

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

uld it be related to this: > > https://www.phoronix.com/scan.php?page=news_item=P-State-Possible-4.6-Regression > > > On Thu, 16 Jun 2016 18:40:01 +0200 > Jirka Hladky wrote: > >> Hello, >> >> we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008 >> benc

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

xargs grep -H Score | grep xml.validation | grep "[0-9]\{4\}[.][0-9]\{2\} ops/m" Jirka On Wed, Jun 22, 2016 at 9:16 AM, Peter Zijlstra <pet...@infradead.org> wrote: > On Fri, Jun 17, 2016 at 01:04:23AM +0200, Jirka Hladky wrote: >> > > we see performance dr

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

xargs grep -H Score | grep xml.validation | grep "[0-9]\{4\}[.][0-9]\{2\} ops/m" Jirka On Wed, Jun 22, 2016 at 9:16 AM, Peter Zijlstra wrote: > On Fri, Jun 17, 2016 at 01:04:23AM +0200, Jirka Hladky wrote: >> > > we see performance drop 30-40% for SPECjbb2005 and SPECjvm

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

5ca3726af7f66a8cc71ce4414cfeb86deb784491 sched/cpuacct: Show all possible CPUs in cpuacct output On Fri, Jun 17, 2016 at 1:04 AM, Jirka Hladky <jhla...@redhat.com> wrote: >> > we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008 >> Blergh, of course I don't have those

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

5ca3726af7f66a8cc71ce4414cfeb86deb784491 sched/cpuacct: Show all possible CPUs in cpuacct output On Fri, Jun 17, 2016 at 1:04 AM, Jirka Hladky wrote: >> > we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008 >> Blergh, of course I don't have those.. :/ > > SPECjvm2008

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

r Zijlstra <pet...@infradead.org> wrote: > On Thu, Jun 16, 2016 at 06:38:50PM +0200, Jirka Hladky wrote: >> Hello, >> >> we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008 > > Blergh, of course I don't have those.. :/ > >> benchmarks starting from

Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

Zijlstra wrote: > On Thu, Jun 16, 2016 at 06:38:50PM +0200, Jirka Hladky wrote: >> Hello, >> >> we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008 > > Blergh, of course I don't have those.. :/ > >> benchmarks starting from 4.7.0-0.rc0 kernel compared

Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

Hello, we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks starting from 4.7.0-0.rc0 kernel compared to 4.6 kernel. We have tested kernels 4.7.0-0.rc1 and 4.7.0-0.rc3 and these are as well affected. We have observed the drop on variety of different x86_64 servers with

Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

Hello, we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks starting from 4.7.0-0.rc0 kernel compared to 4.6 kernel. We have tested kernels 4.7.0-0.rc1 and 4.7.0-0.rc3 and these are as well affected. We have observed the drop on variety of different x86_64 servers with

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

fixed for 2a595721a1fa6b684c1c818f379bef834ac3d65e ? I think I will need to start the bisecting from there. Thanks Jirka > > > On Wed, Dec 16, 2015 at 6:04 PM, Jirka Hladky wrote: >> >> Hi Peter, >> >> you are right the kernel 4.4-rc4 has it already fixed

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

fixed for 2a595721a1fa6b684c1c818f379bef834ac3d65e ? I think I will need to start the bisecting from there. Thanks Jirka > > > On Wed, Dec 16, 2015 at 6:04 PM, Jirka Hladky <jhla...@redhat.com> wrote: >> >> Hi Peter, >> >> you are right the kernel 4

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

Bisecting: 32 revisions left to test after this (roughly 5 steps) [da7142e2ed735e1c1bef5a757dc55de35c65fbd6] sched/core: Simplify preempt_count tests I will let you know the outcome. Jirka On Wed, Dec 16, 2015 at 2:50 PM, Peter Zijlstra wrote: > On Wed, Dec 16, 2015 at 01:56:17PM +0100, Jirka Hla

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

the NUMA sched_feature git bisect good 2b49d84b259fc18e131026e5d38e7855352f71b9 # first bad commit: [2a595721a1fa6b684c1c818f379bef834ac3d65e] sched/numa: Convert sched_numa_balancing to a static_branch On Tue, Dec 15, 2015 at 9:49 AM, Jirka Hladky wrote: > Hi Rik, > > I have reviewed

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

at 01:56:17PM +0100, Jirka Hladky wrote: >> Hi Rik, >> >> I have redone the bisecting and have new results: >> >> # first bad commit: [2a595721a1fa6b684c1c818f379bef834ac3d65e] >> sched/numa: Convert sched_numa_balancing to a static_branch >> >> Co

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

the NUMA sched_feature git bisect good 2b49d84b259fc18e131026e5d38e7855352f71b9 # first bad commit: [2a595721a1fa6b684c1c818f379bef834ac3d65e] sched/numa: Convert sched_numa_balancing to a static_branch On Tue, Dec 15, 2015 at 9:49 AM, Jirka Hladky <jhla...@redhat.com> wrote: > Hi Rik, &

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

placement. I will try to replay the bisect once again. Jirka On Tue, Dec 15, 2015 at 3:12 AM, Rik van Riel wrote: > On 12/14/2015 06:52 PM, Jirka Hladky wrote: >> Hi all, >> >> I have the results of bisecting: >> >> first bad commit: [973759c80db96ed4b4c5cb85ac7d481

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

placement. I will try to replay the bisect once again. Jirka On Tue, Dec 15, 2015 at 3:12 AM, Rik van Riel <r...@redhat.com> wrote: > On 12/14/2015 06:52 PM, Jirka Hladky wrote: >> Hi all, >> >> I have the results of bisecting: >> >> first bad commit: [973759c8

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

at 3:37 PM, Mike Galbraith wrote: > On Sat, 2015-12-12 at 15:16 +0100, Jirka Hladky wrote: >> > A bisection doesn't require any special skills, but may give busy >> > maintainers a single change to eyeball vs the entire lot. >> >> They have been couple of merges

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

at 3:37 PM, Mike Galbraith wrote: > On Sat, 2015-12-12 at 15:16 +0100, Jirka Hladky wrote: >> > A bisection doesn't require any special skills, but may give busy >> > maintainers a single change to eyeball vs the entire lot. >> >> They have been couple of merges

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

at 3:37 PM, Mike Galbraith <umgwanakikb...@gmail.com> wrote: > On Sat, 2015-12-12 at 15:16 +0100, Jirka Hladky wrote: >> > A bisection doesn't require any special skills, but may give busy >> > maintainers a single change to eyeball vs the entire lot. >> >> They

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

at 3:37 PM, Mike Galbraith <umgwanakikb...@gmail.com> wrote: > On Sat, 2015-12-12 at 15:16 +0100, Jirka Hladky wrote: >> > A bisection doesn't require any special skills, but may give busy >> > maintainers a single change to eyeball vs the entire lot. >> >> They

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

e19159 -- sched and let you the outcome. On Sat, Dec 12, 2015 at 8:04 AM, Mike Galbraith wrote: > (it's always a good idea to CC subsystem maintainers when reporting) > > On Fri, 2015-12-11 at 15:17 +0100, Jirka Hladky wrote: >> Hello, >> >> we are doing performance te

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

e19159 -- sched and let you the outcome. On Sat, Dec 12, 2015 at 8:04 AM, Mike Galbraith <umgwanakikb...@gmail.com> wrote: > (it's always a good idea to CC subsystem maintainers when reporting) > > On Fri, 2015-12-11 at 15:17 +0100, Jirka Hladky wrote: >> Hello, >> &g

sched : performance regression 24% between 4.4rc4 and 4.3 kernel

Hello, we are doing performance testing of the new kernel scheduler (commit 53528695ff6d8b77011bc818407c13e30914a946). In most cases we see performance improvements compared to 4.3 kernel with the exception of stream benchmark when running on 4 NUMA node server. When we run 4 stream benchmark

sched : performance regression 24% between 4.4rc4 and 4.3 kernel

Hello, we are doing performance testing of the new kernel scheduler (commit 53528695ff6d8b77011bc818407c13e30914a946). In most cases we see performance improvements compared to 4.3 kernel with the exception of stream benchmark when running on 4 NUMA node server. When we run 4 stream benchmark

Re: [PATCH] numa,sched: only consider less busy nodes as numa balancing destination

rying to move to the preferred node, or to another node. Signed-off-by: Rik van Riel Reported-by: Artem Bityutski Reported-by: Jirka Hladky --- kernel/sched/fair.c | 30 -- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel

Re: [PATCH] numa,sched: only consider less busy nodes as numa balancing destination

the task is trying to move to the preferred node, or to another node. Signed-off-by: Rik van Riel r...@redhat.com Reported-by: Artem Bityutski dedeki...@gmail.com Reported-by: Jirka Hladky jhla...@redhat.com --- kernel/sched/fair.c | 30 -- 1 file changed, 28 insertions

Re: [LKP] [sched/numa] a43455a1d57: +94.1% proc-vmstat.numa_hint_faults_local

On 08/02/2014 06:17 AM, Rik van Riel wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/01/2014 05:30 PM, Jirka Hladky wrote: I see the regression only on this box. It has 4 "Ivy Bridge-EX" Xeon E7-4890 v2 CPUs. http://ark.intel.com/products/75251 http://en.wikipedi

Re: [LKP] [sched/numa] a43455a1d57: +94.1% proc-vmstat.numa_hint_faults_local

On 08/01/2014 10:46 PM, Davidlohr Bueso wrote: On Thu, 2014-07-31 at 18:16 +0200, Jirka Hladky wrote: Peter, I'm seeing regressions for SINGLE SPECjbb instance for number of warehouses being the same as total number of cores in the box. Example: 4 NUMA node box, each CPU has 6 cores

Re: [LKP] [sched/numa] a43455a1d57: +94.1% proc-vmstat.numa_hint_faults_local

On 08/01/2014 10:46 PM, Davidlohr Bueso wrote: On Thu, 2014-07-31 at 18:16 +0200, Jirka Hladky wrote: Peter, I'm seeing regressions for SINGLE SPECjbb instance for number of warehouses being the same as total number of cores in the box. Example: 4 NUMA node box, each CPU has 6 cores = biggest

Re: [LKP] [sched/numa] a43455a1d57: +94.1% proc-vmstat.numa_hint_faults_local

On 08/02/2014 06:17 AM, Rik van Riel wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/01/2014 05:30 PM, Jirka Hladky wrote: I see the regression only on this box. It has 4 Ivy Bridge-EX Xeon E7-4890 v2 CPUs. http://ark.intel.com/products/75251 http://en.wikipedia.org/wiki

Re: [LKP] [sched/numa] a43455a1d57: +94.1% proc-vmstat.numa_hint_faults_local

On 07/31/2014 06:27 PM, Peter Zijlstra wrote: On Thu, Jul 31, 2014 at 06:16:26PM +0200, Jirka Hladky wrote: On 07/31/2014 05:57 PM, Peter Zijlstra wrote: On Thu, Jul 31, 2014 at 12:42:41PM +0200, Peter Zijlstra wrote: On Tue, Jul 29, 2014 at 02:39:40AM -0400, Rik van Riel wrote: On Tue, 29

Re: [LKP] [sched/numa] a43455a1d57: +94.1% proc-vmstat.numa_hint_faults_local

6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 74 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_local Hi Aaron, Jirka Hladky has reported a regression with that changeset as well, and I ha

Re: [LKP] [sched/numa] a43455a1d57: +94.1% proc-vmstat.numa_hint_faults_local

% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 74 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_local Hi Aaron, Jirka Hladky has reported a regression with that changeset as well, and I

Re: [LKP] [sched/numa] a43455a1d57: +94.1% proc-vmstat.numa_hint_faults_local

On 07/31/2014 06:27 PM, Peter Zijlstra wrote: On Thu, Jul 31, 2014 at 06:16:26PM +0200, Jirka Hladky wrote: On 07/31/2014 05:57 PM, Peter Zijlstra wrote: On Thu, Jul 31, 2014 at 12:42:41PM +0200, Peter Zijlstra wrote: On Tue, Jul 29, 2014 at 02:39:40AM -0400, Rik van Riel wrote: On Tue, 29