Re: [LKP] [lkp] [x86/platform/UV] 71854cb812: will-it-scale.per_thread_ops -2.3% regression
On Mon, 31 Oct 2016, Fengguang Wu wrote: > On Fri, Oct 28, 2016 at 12:37:45AM +0200, Thomas Gleixner wrote: > > I have no explanation for this either, but you really should try to figure > > out what's going on here. > > Xiaolong, how about doing a small debug patch (a WARN_ONCE() line may > be enough) to verify whether the code path is executed? > > It'd also help to compare vmlinux according to Thomas' reasoning: > > > The only difference between plain rc1 and rc + this patch is the resulting > > text size and therefor some other unrelated stuff moving to different > > places in memory which has some yet to figure out side effects. So one other thing you might try is using rc3 as a base line and then revert the patch in question on top of rc3 and see whether that weird behaviour persists. > > From bisect POV, the below graphs show the user_time and system_time > are clearly and consistently different before/after commit 71854cb812. > So this commit must impacted something. I agree. I just have no idea what happens. Thanks, tglx
Re: [LKP] [lkp] [x86/platform/UV] 71854cb812: will-it-scale.per_thread_ops -2.3% regression
Hi Thomas, It's been a big challenge that we'll occasionally run into such bisect whose data show clear changes, however cannot be easily explained by looking at the code logic. On Fri, Oct 28, 2016 at 12:37:45AM +0200, Thomas Gleixner wrote: On Thu, 27 Oct 2016, Ye Xiaolong wrote: Yes, this is weird, the per_thread_ops change is small and should be run to run variation, the actual significant change is will-it-scale.time.user_time -27% decrease, but the patch seems not relevant, we can't interpret it. :( We've tried to queue the jobs (4 times) for 71854cb812ec23bfe5f63d52217e6b9e6cb901f5 and v4.9-rc1 with new kconfig (added CONFIG_DEBUG_INFO_REDUCED), and result shows user_time change is quite stable. v4.9-rc1 71854cb812ec23bfe5f63d5221 -- %stddev %change %stddev \ |\ 1670068 ± 0% -3.8%1606650 ± 1% will-it-scale.per_thread_ops 9749 ± 2% +1328.0% 139222 ±105% will-it-scale.time.involuntary_context_switches ^^ This is massive I have no explanation for this either, but you really should try to figure out what's going on here. Xiaolong, how about doing a small debug patch (a WARN_ONCE() line may be enough) to verify whether the code path is executed? It'd also help to compare vmlinux according to Thomas' reasoning: The only difference between plain rc1 and rc + this patch is the resulting text size and therefor some other unrelated stuff moving to different places in memory which has some yet to figure out side effects. Yeah that's possible. From bisect POV, the below graphs show the user_time and system_time are clearly and consistently different before/after commit 71854cb812. So this commit must impacted something. Legend: [*] bisect good samples (eg. tests run on commits before 71854cb812) [o] bisect bad samples (eg. tests run on commits since 71854cb812) will-it-scale.time.user_time 85 ++-+ | .*.*.. .*..*..*..*. .*.. .*..*. .* | 80 *+ *.*..* *..* *..*..** *..*.*.| | | | | 75 ++ | | | 70 ++ | | | 65 ++ | | | |OO| 60 O+ OO O O O O OO O O O O O O O O O O |O O OO OOO O| 55 ++-+ will-it-scale.time.system_time 1010 ++---+ || 1005 ++ O O OO O O O O O| O OOO O O O O O O O O O O O O O O |OO | 1000 ++ | || 995 ++ | || 990 ++ | || || 985 ++ | *..*..*.*...*.*...*.*..*. .*...*.*..*. | 980 ++*--*-*-*---*-*-**-*-*--+ The voluntary_context_switches increase looks obvious, too, though not as consistent: will-it-scale.time.voluntary_context_switches 16 ++-+ | O | 15 ++ | | | 14 ++ O O O| |
Re: [lkp] [x86/platform/UV] 71854cb812: will-it-scale.per_thread_ops -2.3% regression
On Thu, 27 Oct 2016, Ye Xiaolong wrote: > Yes, this is weird, the per_thread_ops change is small and should be run > to run variation, the actual significant change is > will-it-scale.time.user_time > -27% decrease, but the patch seems not relevant, we can't interpret it. :( > > We've tried to queue the jobs (4 times) for > 71854cb812ec23bfe5f63d52217e6b9e6cb901f5 and v4.9-rc1 with new kconfig > (added CONFIG_DEBUG_INFO_REDUCED), and result shows user_time change is > quite stable. > v4.9-rc1 71854cb812ec23bfe5f63d5221 > -- > %stddev %change %stddev > \ |\ >1670068 ± 0% -3.8%1606650 ± 1% will-it-scale.per_thread_ops > 9749 ± 2% +1328.0% 139222 ±105% > will-it-scale.time.involuntary_context_switches ^^ This is massive I have no explanation for this either, but you really should try to figure out what's going on here. The only difference between plain rc1 and rc + this patch is the resulting text size and therefor some other unrelated stuff moving to different places in memory which has some yet to figure out side effects. Thanks, tglx
Re: [lkp] [x86/platform/UV] 71854cb812: will-it-scale.per_thread_ops -2.3% regression
On 10/25, Thomas Gleixner wrote:
>On Tue, 25 Oct 2016, kernel test robot wrote:
>> FYI, we noticed a -2.3% regression of will-it-scale.per_thread_ops due to
>> commit:
>>
>> commit 71854cb812ec23bfe5f63d52217e6b9e6cb901f5 ("x86/platform/UV: Fix
>> support for EFI_OLD_MEMMAP after BIOS callback updates")
>> https://github.com/0day-ci/linux
>> Alex-Thorlton/x86-platform-UV-Fix-support-for-EFI_OLD_MEMMAP-after-BIOS-callback-updates/20161020-095215
>>
>> in testcase: will-it-scale
>> on test machine: 12 threads Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz with 6G
>> memory
>
>This is completely bogus. That patch does not even affect anything outside
>of the SGI UV platform. And on your i7 system uv_bios_call() is definitely
>not invoked.
Yes, this is weird, the per_thread_ops change is small and should be run
to run variation, the actual significant change is will-it-scale.time.user_time
-27% decrease, but the patch seems not relevant, we can't interpret it. :(
We've tried to queue the jobs (4 times) for
71854cb812ec23bfe5f63d52217e6b9e6cb901f5 and v4.9-rc1
with new kconfig (added CONFIG_DEBUG_INFO_REDUCED), and result shows
user_time change is quite stable.
=
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase:
gcc-6/performance/x86_64-rhel-7.2+CONFIG_DEBUG_INFO_REDUCED/debian-x86_64-2016-08-31.cgz/wsm/read2/will-it-scale
commit:
v4.9-rc1
71854cb812ec23bfe5f63d52217e6b9e6cb901f5
v4.9-rc1 71854cb812ec23bfe5f63d5221
--
%stddev %change %stddev
\ |\
1670068 ± 0% -3.8%1606650 ± 1% will-it-scale.per_thread_ops
9749 ± 2% +1328.0% 139222 ±105%
will-it-scale.time.involuntary_context_switches
981.29 ± 0% +2.2% 1002 ± 0% will-it-scale.time.system_time
81.78 ± 0% -26.9% 59.74 ± 0% will-it-scale.time.user_time
32894 ± 0% -3.1% 31863 ± 2% vmstat.system.cs
9749 ± 2% +1328.0% 139222 ±105% time.involuntary_context_switches
380917 ± 2% -10.2% 341970 ± 3% sched_debug.cpu.avg_idle.avg
89166 ± 33% -73.4% 23731 ± 29% sched_debug.cpu.avg_idle.min
16.38 ± 10% -32.3% 11.08 ± 18%
sched_debug.cpu.nr_uninterruptible.max
0.29 ± 1% +32.6% 0.38 ± 1% perf-stat.branch-miss-rate%
2.897e+09 ± 1% +33.5% 3.867e+09 ± 2% perf-stat.branch-misses
10084878 ± 0% -3.2%9761852 ± 2% perf-stat.context-switches
0.00 ± 7% -9.3% 0.00 ± 1% perf-stat.dTLB-store-miss-rate%
33489012 ± 7% -9.2% 30416429 ± 1% perf-stat.dTLB-store-misses
Thanks,
Xiaolong
>
>I appreciate your effort, but posting such obviously bogus results does not
>make people more confident in your testing efforts.
>
>Thanks,
>
> tglx
Re: [lkp] [x86/platform/UV] 71854cb812: will-it-scale.per_thread_ops -2.3% regression
On Tue, 25 Oct 2016, kernel test robot wrote:
> FYI, we noticed a -2.3% regression of will-it-scale.per_thread_ops due to
> commit:
>
> commit 71854cb812ec23bfe5f63d52217e6b9e6cb901f5 ("x86/platform/UV: Fix
> support for EFI_OLD_MEMMAP after BIOS callback updates")
> https://github.com/0day-ci/linux
> Alex-Thorlton/x86-platform-UV-Fix-support-for-EFI_OLD_MEMMAP-after-BIOS-callback-updates/20161020-095215
>
> in testcase: will-it-scale
> on test machine: 12 threads Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz with 6G
> memory
This is completely bogus. That patch does not even affect anything outside
of the SGI UV platform. And on your i7 system uv_bios_call() is definitely
not invoked.
I appreciate your effort, but posting such obviously bogus results does not
make people more confident in your testing efforts.
Thanks,
tglx
