Bug#674153: [3.2.16 - 3.2.17 regression] High reported CPU load when idle
On Mon, 2012-07-23 at 08:56 -0700, Doug Smythies wrote: [...] With only a superficial few hours on each kernel, I verified The problem with kernel 3.2.23 and no problem with kernel 3.2.23 + patch. Note: the majority of my testing (one or two hundred hours) were Done with Ubuntu server edition 12.04, Kernel 3.2.0-24-generic #39, and all three of: 556061b00c9f2fd6a5524b6bde823ef12f299ecf 5aaa0b7a2ed5b12692c9ffb5222182bd558d3146 I have queued these up to be reviewed for inclusion in Linux 3.2.25. 5167e8d5417bf5c322a703d2927daec727ea40dd This was included in Linux 3.2.24. Ben. manually backedited. By the way this non-distro 3.2.23 kernel seems to run a lot faster than the Ubuntu distro 3.2.0-24 kernel. -- Ben Hutchings It is impossible to make anything foolproof because fools are so ingenious. signature.asc Description: This is a digitally signed message part
Bug#674153: [3.2.16 - 3.2.17 regression] High reported CPU load when idle
On 2012.07.17 16:26 -0700 Jonathan Nieder wrote: Please test the attached patch against a 3.2.y kernel, for example following the instructions below: 0. prerequisites: apt-get install git build-essential 1. get the kernel history, if you don't already have it: git clone \ git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 2. fetch point releases: cd linux git remote add stable \ git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git git fetch stable 3. configure, build, test: git checkout stable/linux-3.2.y cp /boot/config-$(uname -r) .config; # current configuration scripts/config --disable DEBUG_INFO make localmodconfig; # optional: minimize configuration make deb-pkg; # optionally with -jnum for parallel build dpkg -i ../name of package; # as root reboot ... test test test ... Hopefully that will reproduce the bug. So 4. try the patch: cd linux git am -3sc /path/to/the/patch dpkg -i ../name of package; # as root reboot ... test test test ... Hope that helps, Jonathan Hi Jonathan, Thanks for your instructions. I tried them, because I have never successfully got and compiled the kernel via GIT before. Your instructions worked fine. With only a superficial few hours on each kernel, I verified The problem with kernel 3.2.23 and no problem with kernel 3.2.23 + patch. Note: the majority of my testing (one or two hundred hours) were Done with Ubuntu server edition 12.04, Kernel 3.2.0-24-generic #39, and all three of: 556061b00c9f2fd6a5524b6bde823ef12f299ecf 5aaa0b7a2ed5b12692c9ffb5222182bd558d3146 5167e8d5417bf5c322a703d2927daec727ea40dd manually backedited. By the way this non-distro 3.2.23 kernel seems to run a lot faster than the Ubuntu distro 3.2.0-24 kernel. ... Doug Smythies -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/000f01cd68eb$c446a5e0$4cd3f1a0$@net
Bug#674153: [3.2.16 - 3.2.17 regression] High reported CPU load when idle
On 07/18/2012 01:25 AM, Jonathan Nieder wrote: Anders Boström wrote: Starting with 3.2.17-1, the CPU load accounting is broken when the computer is idle. The CPU load is reported as 0.50 when idle. 3.2.16-1 don't suffer from this problem. Suspected patch is the upstream patch sched: Fix nohz load accounting -- again! commit 5e2d50da11f0e6ec3ce8fe658d7c83b0b4346c68 to 3.2 and originating from c308b56b5398779cd3da0f62ab26b0453494c3d4 . Please test the attached patch against a 3.2.y kernel, for example following the instructions below: Good news everyone. I have tested kernel 3.2.21 and the attached patch (based on 5167e8d I presume) seems to be fixing all the load average oddities. I've compiled following kernels: * 3.2.21-hz (CONFIG_NO_HZ=n) * 3.2.21-no-hz (CONFIG_NO_HZ=y) * 3.2.21-no-hz-5167e8d (CONFIG_NO_HZ=y) + attached patch The load reported by 3.2.21-hz and 3.2.21-no-hz-5167e8d is exactly the same under different CPU usage. Without the patch the tickless kernel tends to show lower load values than what you would expect. I can't say much for the case when load is too high on an idle machine, because I haven't been able to reproduce the problem in the first place. To summarize: the bug is present in unpatched kernel and fixed by applying the attached patch. No nasty side effects noticed. -- Lesław Kopeć signature.asc Description: OpenPGP digital signature
Bug#674153: [3.2.16 - 3.2.17 regression] High reported CPU load when idle
On Thu, 2012-07-19 at 11:12 +0200, Lesław Kopeć wrote: On 07/18/2012 01:25 AM, Jonathan Nieder wrote: Anders Boström wrote: Starting with 3.2.17-1, the CPU load accounting is broken when the computer is idle. The CPU load is reported as 0.50 when idle. 3.2.16-1 don't suffer from this problem. Suspected patch is the upstream patch sched: Fix nohz load accounting -- again! commit 5e2d50da11f0e6ec3ce8fe658d7c83b0b4346c68 to 3.2 and originating from c308b56b5398779cd3da0f62ab26b0453494c3d4 . Please test the attached patch against a 3.2.y kernel, for example following the instructions below: Good news everyone. I have tested kernel 3.2.21 and the attached patch (based on 5167e8d I presume) seems to be fixing all the load average oddities. I've compiled following kernels: * 3.2.21-hz (CONFIG_NO_HZ=n) * 3.2.21-no-hz(CONFIG_NO_HZ=y) * 3.2.21-no-hz-5167e8d(CONFIG_NO_HZ=y) + attached patch The load reported by 3.2.21-hz and 3.2.21-no-hz-5167e8d is exactly the same under different CPU usage. Without the patch the tickless kernel tends to show lower load values than what you would expect. I can't say much for the case when load is too high on an idle machine, because I haven't been able to reproduce the problem in the first place. To summarize: the bug is present in unpatched kernel and fixed by applying the attached patch. No nasty side effects noticed. This is in the review queue for Linux 3.2.24. I'm hesistant to apply it until it's been through the stable review process (probably early next week). But, if there's no objection to it there, it will end up in Debian pretty soon. Ben. -- Ben Hutchings DNRC Motto: I can please only one person per day. Today is not your day. Tomorrow isn't looking good either. signature.asc Description: This is a digitally signed message part
Bug#674153: [3.2.16 - 3.2.17 regression] High reported CPU load when idle
tags 674153 = upstream patch moreinfo
quit
Hi again,
Anders Boström wrote:
Starting with 3.2.17-1, the CPU load accounting is broken when the
computer is idle. The CPU load is reported as 0.50 when
idle. 3.2.16-1 don't suffer from this problem.
Suspected patch is the upstream patch
sched: Fix nohz load accounting -- again!
commit 5e2d50da11f0e6ec3ce8fe658d7c83b0b4346c68 to 3.2 and
originating from c308b56b5398779cd3da0f62ab26b0453494c3d4 .
Please test the attached patch against a 3.2.y kernel, for example
following the instructions below:
0. prerequisites:
apt-get install git build-essential
1. get the kernel history, if you don't already have it:
git clone \
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
2. fetch point releases:
cd linux
git remote add stable \
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
git fetch stable
3. configure, build, test:
git checkout stable/linux-3.2.y
cp /boot/config-$(uname -r) .config; # current configuration
scripts/config --disable DEBUG_INFO
make localmodconfig; # optional: minimize configuration
make deb-pkg; # optionally with -jnum for parallel build
dpkg -i ../name of package; # as root
reboot
... test test test ...
Hopefully that will reproduce the bug. So
4. try the patch:
cd linux
git am -3sc /path/to/the/patch
dpkg -i ../name of package; # as root
reboot
... test test test ...
Hope that helps,
Jonathan
From: Peter Zijlstra a.p.zijls...@chello.nl
Date: Fri, 22 Jun 2012 15:52:09 +0200
Subject: sched/nohz: Rewrite and fix load-avg computation -- again
commit 5167e8d5417bf5c322a703d2927daec727ea40dd upstream.
Thanks to Charles Wang for spotting the defects in the current code:
- If we go idle during the sample window -- after sampling, we get a
negative bias because we can negate our own sample.
- If we wake up during the sample window we get a positive bias
because we push the sample to a known active period.
So rewrite the entire nohz load-avg muck once again, now adding
copious documentation to the code.
Reported-and-tested-by: Doug Smythies dsmyth...@telus.net
Reported-and-tested-by: Charles Wang muming...@gmail.com
Signed-off-by: Peter Zijlstra a.p.zijls...@chello.nl
Cc: Linus Torvalds torva...@linux-foundation.org
Cc: Andrew Morton a...@linux-foundation.org
Cc: sta...@kernel.org
Link: http://lkml.kernel.org/r/1340373782.18025.74.camel@twins
[ minor edits ]
Signed-off-by: Ingo Molnar mi...@kernel.org
Signed-off-by: Jonathan Nieder jrnie...@gmail.com
---
include/linux/sched.h|8 ++
kernel/sched.c | 276 ++
kernel/sched_idletask.c |1 -
kernel/time/tick-sched.c |2 +
4 files changed, 213 insertions(+), 74 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 1c4f3e9b9bc5..5afa2a345ab1 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1892,6 +1892,14 @@ static inline int set_cpus_allowed_ptr(struct
task_struct *p,
}
#endif
+#ifdef CONFIG_NO_HZ
+void calc_load_enter_idle(void);
+void calc_load_exit_idle(void);
+#else
+static inline void calc_load_enter_idle(void) { }
+static inline void calc_load_exit_idle(void) { }
+#endif /* CONFIG_NO_HZ */
+
#ifndef CONFIG_CPUMASK_OFFSTACK
static inline int set_cpus_allowed(struct task_struct *p, cpumask_t new_mask)
{
diff --git a/kernel/sched.c b/kernel/sched.c
index 576a27fa5efc..52ac69b6d4c7 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -1885,7 +1885,6 @@ static void double_rq_unlock(struct rq *rq1, struct rq
*rq2)
#endif
-static void calc_load_account_idle(struct rq *this_rq);
static void update_sysctl(void);
static int get_update_sysctl_factor(void);
static void update_cpu_load(struct rq *this_rq);
@@ -3401,11 +3400,73 @@ unsigned long this_cpu_load(void)
}
+/*
+ * Global load-average calculations
+ *
+ * We take a distributed and async approach to calculating the global load-avg
+ * in order to minimize overhead.
+ *
+ * The global load average is an exponentially decaying average of nr_running +
+ * nr_uninterruptible.
+ *
+ * Once every LOAD_FREQ:
+ *
+ * nr_active = 0;
+ * for_each_possible_cpu(cpu)
+ * nr_active += cpu_of(cpu)-nr_running + cpu_of(cpu)-nr_uninterruptible;
+ *
+ * avenrun[n] = avenrun[0] * exp_n + nr_active * (1 - exp_n)
+ *
+ * Due to a number of reasons the above turns in the mess below:
+ *
+ * - for_each_possible_cpu() is prohibitively expensive on machines with
+ *serious number of cpus, therefore we need to take a distributed approach
+ *to calculating nr_active.
+ *
+ *\Sum_i x_i(t) = \Sum_i x_i(t) - x_i(t_0) | x_i(t_0) := 0
+ * = \Sum_i { \Sum_j=1 x_i(t_j) - x_i(t_j-1) }
+ *
+ *So assuming nr_active := 0 when we start out -- true per definition, we
+ *
