Package: linux-image-6.6.13+bpo-amd64-unsigned Severity: important Dear Maintainer,
Upgrading from bookroms 6.1 to 6.6 causes a major performance degradation. * What led up to the situation? I semi-regularly stress-test systems with linpack-xtreme-1.1.5-amd64 to see if there are thermal or stability problems. Typical output looks like this. The GFlops performance varies somewhat due to thermal issues, but is generally above 500 GFlops on this system: Size LDA Align. Time(s) GFlops Residual Residual(norm) Check 22611 22611 4 15.171 508.0650 4.907015e-10 3.410840e-02 pass 22611 22611 4 14.935 516.0887 4.907015e-10 3.410840e-02 pass 22611 22611 4 14.978 514.6037 4.907015e-10 3.410840e-02 pass 22611 22611 4 15.260 505.0881 4.907015e-10 3.410840e-02 pass 22611 22611 4 14.669 525.4384 4.907015e-10 3.410840e-02 pass After upgrading from 6.1 to 6.6, I noticed some programs being surprisingly slow, so I run the stress test,a nd got output like this: Size LDA Align. Time(s) GFlops Residual Residual(norm) Check 22611 22611 4 48.447 159.0972 5.357863e-10 3.724222e-02 pass As you can see, the performance is seriously degraded. Moreso, it is all over the place, sometimes it as 212 GFlops, sometimes only 44. This is on an i7-14700k, but it happens with another system using a 13700k in exactly the same way. * What exactly did you do (or not do) that was effective (or ineffective)? After some investigation, I noticed that rapl shows a power usage of about 50W instead of the more expected 200+. Turned out thermald set a power limit of 65W. Thinking this to be some bug in thermald, I disabled it and restored the power limit(s) too 300W. Unfortunately, while this increased the power usage to almost 200W, it did not improve performance at all (normal cpu power suage for linpack is up to 275W). I then tried various other things, and found that booting the old 6.1 linux kernel fixed this problem completely. Not quite believing it, I built a special initrd with linpack inside, and found it happens in there too, that is, linux-6.6 is slow, erratic, and linux-6.1 performs as expected, and this is independent of any configuration or installed software. I tried booting with no kernel arguments as well to exclude any command line arguments being the culprit, and found the same performance degradation, leaving essentially only the kernel. (my default arguments include mitigations=off, so it's not caused by any mitigation either). I also tried this on another system with a 13700K cpu, and got exactly the same results - 6.1 works fine, 6.6 only reaches 10-50% of the performance, very erratically. I do notice that processes seem to jump around widely between cpus when this happens, but that might or might not be related. * What was the outcome of this action? I did downgrade to 6.1 on all affected systems. * What outcome did you expect instead? Obviously, 6.6 should perform more or less the same as 6.1. This might not be an issue with debians kernel, as I can find a few other, similar reports affecting manjaro and arch linux, e.g. https://forum.manjaro.org/t/linux-kernel-6-6-lts-cpu-regression-on-i7-alderlake/157474/30 *** End of the template - remove these template lines *** -- System Information: Debian Release: 12.5 APT prefers stable-updates APT policy: (990, 'stable-updates'), (990, 'stable-security'), (990, 'stable'), (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'oldstable-debug'), (500, 'unstable'), (500, 'testing'), (1, 'experimental-debug'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386, x32 Kernel: Linux 6.1.0-18-amd64 (SMP w/28 CPU threads; PREEMPT) Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_USER, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=en_IE.UTF-8, LC_CTYPE=en_IE.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages linux-image-6.6.13+bpo-amd64-unsigned depends on: ii initramfs-tools [linux-initramfs-tool] 0.142 ii kmod 30+20221128-1 ii linux-base 4.9 Versions of packages linux-image-6.6.13+bpo-amd64-unsigned recommends: ii apparmor 3.0.8-3 ii firmware-linux-free 20200122-1 Versions of packages linux-image-6.6.13+bpo-amd64-unsigned suggests: pn debian-kernel-handbook <none> ii extlinux 3:6.04~git20190206.bf6db5b4+dfsg1-3+b1 ii grub-efi-amd64 2.06-13+deb12u1 pn linux-doc-6.6 <none>