Package: linux-image-6.6.13+bpo-amd64-unsigned
Severity: important
Dear Maintainer,
Upgrading from bookroms 6.1 to 6.6 causes a major performance degradation.
* What led up to the situation?
I semi-regularly stress-test systems with linpack-xtreme-1.1.5-amd64 to
see if there are thermal or stability problems. Typical output looks like
this. The GFlops performance varies somewhat due to thermal issues, but is
generally above 500 GFlops on this system:
Size LDA Align. Time(s) GFlops Residual Residual(norm) Check
22611 22611 4 15.171 508.0650 4.907015e-10 3.410840e-02 pass
22611 22611 4 14.935 516.0887 4.907015e-10 3.410840e-02 pass
22611 22611 4 14.978 514.6037 4.907015e-10 3.410840e-02 pass
22611 22611 4 15.260 505.0881 4.907015e-10 3.410840e-02 pass
22611 22611 4 14.669 525.4384 4.907015e-10 3.410840e-02 pass
After upgrading from 6.1 to 6.6, I noticed some programs being
surprisingly slow, so I run the stress test,a nd got output like this:
Size LDA Align. Time(s) GFlops Residual Residual(norm) Check
22611 22611 4 48.447 159.0972 5.357863e-10 3.724222e-02 pass
As you can see, the performance is seriously degraded. Moreso, it is all
over the place, sometimes it as 212 GFlops, sometimes only 44.
This is on an i7-14700k, but it happens with another system using a 13700k
in exactly the same way.
* What exactly did you do (or not do) that was effective (or
ineffective)?
After some investigation, I noticed that rapl shows a power usage of about
50W instead of the more expected 200+. Turned out thermald set a power
limit of 65W. Thinking this to be some bug in thermald, I disabled it and
restored the power limit(s) too 300W.
Unfortunately, while this increased the power usage to almost 200W, it did
not improve performance at all (normal cpu power suage for linpack is up
to 275W).
I then tried various other things, and found that booting the old 6.1
linux kernel fixed this problem completely. Not quite believing it, I
built a special initrd with linpack inside, and found it happens in
there too, that is, linux-6.6 is slow, erratic, and linux-6.1 performs
as expected, and this is independent of any configuration or installed
software.
I tried booting with no kernel arguments as well to exclude any command
line arguments being the culprit, and found the same performance
degradation, leaving essentially only the kernel. (my default arguments
include mitigations=off, so it's not caused by any mitigation either).
I also tried this on another system with a 13700K cpu, and got exactly the
same results - 6.1 works fine, 6.6 only reaches 10-50% of the performance,
very erratically.
I do notice that processes seem to jump around widely between cpus when
this happens, but that might or might not be related.
* What was the outcome of this action?
I did downgrade to 6.1 on all affected systems.
* What outcome did you expect instead?
Obviously, 6.6 should perform more or less the same as 6.1.
This might not be an issue with debians kernel, as I can find a few other,
similar reports affecting manjaro and arch linux, e.g.
https://forum.manjaro.org/t/linux-kernel-6-6-lts-cpu-regression-on-i7-alderlake/157474/30
*** End of the template - remove these template lines ***
-- System Information:
Debian Release: 12.5
APT prefers stable-updates
APT policy: (990, 'stable-updates'), (990, 'stable-security'), (990,
'stable'), (500, 'unstable-debug'), (500, 'testing-debug'), (500,
'oldstable-debug'), (500, 'unstable'), (500, 'testing'), (1,
'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386, x32
Kernel: Linux 6.1.0-18-amd64 (SMP w/28 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_USER, TAINT_OOT_MODULE,
TAINT_UNSIGNED_MODULE
Locale: LANG=en_IE.UTF-8, LC_CTYPE=en_IE.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages linux-image-6.6.13+bpo-amd64-unsigned depends on:
ii initramfs-tools [linux-initramfs-tool] 0.142
ii kmod 30+20221128-1
ii linux-base 4.9
Versions of packages linux-image-6.6.13+bpo-amd64-unsigned recommends:
ii apparmor 3.0.8-3
ii firmware-linux-free 20200122-1
Versions of packages linux-image-6.6.13+bpo-amd64-unsigned suggests:
pn debian-kernel-handbook <none>
ii extlinux 3:6.04~git20190206.bf6db5b4+dfsg1-3+b1
ii grub-efi-amd64 2.06-13+deb12u1
pn linux-doc-6.6 <none>
