[Kernel-packages] [Bug 1829255] Re: Sever performance degradation after updating to 5.0.0-15 due to mds mitigation
Just I quick follow up. The impact was so severe in my code because I was using SDL_PollEvent() really often to read the keys status... which was basically blocking my code. That was one of the first things I wrote and forgot about it. So that was really a context switching overhead problem. Thanks again. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1829255 Title: Sever performance degradation after updating to 5.0.0-15 due to mds mitigation Status in linux package in Ubuntu: Won't Fix Bug description: After updating to the latest kernel 5.0.0-15 I'm experiencing a severe performance degradation on my machine with one of my applications (a gbc emulator developed by me). I saw about a 27% performance loss, going from 55fps to 40. Booting an old 5.0.0-13 kernel or disabling mds mitigation at boot with 'mds=off' solves the problem. I know there isn't much to do about (apart from disabling the mitigation) but I'm reporting this because I spent half a day bisecting with git cause I thought the problem was with my latest commits. Security updates are obviously important and a priority, but I think that at least with this kind of mitigations that cause big performance degradation, an message or a simple alert to the user during the update should be shown. Something like: "Plese be aware that this kernel update can result in performance loss. See this page for more info." and then link the specific SecurityTeam page (which in this case is https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/MDS ) ProblemType: Bug DistroRelease: Ubuntu 19.04 Package: linux-image-5.0.0-15-generic 5.0.0-15.16 ProcVersionSignature: Ubuntu 5.0.0-15.16-generic 5.0.6 Uname: Linux 5.0.0-15-generic x86_64 ApportVersion: 2.20.10-0ubuntu27 Architecture: amd64 AudioDevicesInUse: USERPID ACCESS COMMAND /dev/snd/controlC0: gabriele 2045 F pulseaudio /dev/snd/controlC1: gabriele 2045 F pulseaudio CurrentDesktop: ubuntu:GNOME Date: Wed May 15 16:36:36 2019 HibernationDevice: RESUME=UUID=e43b75a3-a278-4d4e-83e8-54a9e0c95276 InstallationDate: Installed on 2018-05-31 (349 days ago) InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426) MachineType: Dell Inc. Latitude E6540 ProcFB: 0 inteldrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.0.0-15-generic root=UUID=8a8a023d-2a12-4e14-b187-7beaf4d89763 ro quiet splash vt.handoff=1 RelatedPackageVersions: linux-restricted-modules-5.0.0-15-generic N/A linux-backports-modules-5.0.0-15-generic N/A linux-firmware1.178 SourcePackage: linux UpgradeStatus: Upgraded to disco on 2019-04-23 (21 days ago) dmi.bios.date: 10/09/2018 dmi.bios.vendor: Dell Inc. dmi.bios.version: A26 dmi.board.name: 0725FP dmi.board.vendor: Dell Inc. dmi.board.version: A00 dmi.chassis.type: 9 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvrA26:bd10/09/2018:svnDellInc.:pnLatitudeE6540:pvr00:rvnDellInc.:rn0725FP:rvrA00:cvnDellInc.:ct9:cvr: dmi.product.name: Latitude E6540 dmi.product.sku: 05BE dmi.product.version: 00 dmi.sys.vendor: Dell Inc. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1829255/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1829255] Re: Sever performance degradation after updating to 5.0.0-15 due to mds mitigation
Tyler, thanks a lot for taking the time to examine the issue and writing such a complete response. I agree with all your points. I'll take this as an opportunity to try to learn more about profiling and user/kernel space switching and optimization, because this made me curious: > Since the recommendation includes flushing the buffers before exiting from the kernel to userspace, > workloads which are syscall heavy are likely to see the largest performance hit. My emulator is basically a simple asm parser with SDL2 as graphic backend and I'm not doing (that I'm aware of..) any ioctl or syscall. So I need to understand what is happening and what I'm doing work. Back to study now :-) and thanks again -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1829255 Title: Sever performance degradation after updating to 5.0.0-15 due to mds mitigation Status in linux package in Ubuntu: Won't Fix Bug description: After updating to the latest kernel 5.0.0-15 I'm experiencing a severe performance degradation on my machine with one of my applications (a gbc emulator developed by me). I saw about a 27% performance loss, going from 55fps to 40. Booting an old 5.0.0-13 kernel or disabling mds mitigation at boot with 'mds=off' solves the problem. I know there isn't much to do about (apart from disabling the mitigation) but I'm reporting this because I spent half a day bisecting with git cause I thought the problem was with my latest commits. Security updates are obviously important and a priority, but I think that at least with this kind of mitigations that cause big performance degradation, an message or a simple alert to the user during the update should be shown. Something like: "Plese be aware that this kernel update can result in performance loss. See this page for more info." and then link the specific SecurityTeam page (which in this case is https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/MDS ) ProblemType: Bug DistroRelease: Ubuntu 19.04 Package: linux-image-5.0.0-15-generic 5.0.0-15.16 ProcVersionSignature: Ubuntu 5.0.0-15.16-generic 5.0.6 Uname: Linux 5.0.0-15-generic x86_64 ApportVersion: 2.20.10-0ubuntu27 Architecture: amd64 AudioDevicesInUse: USERPID ACCESS COMMAND /dev/snd/controlC0: gabriele 2045 F pulseaudio /dev/snd/controlC1: gabriele 2045 F pulseaudio CurrentDesktop: ubuntu:GNOME Date: Wed May 15 16:36:36 2019 HibernationDevice: RESUME=UUID=e43b75a3-a278-4d4e-83e8-54a9e0c95276 InstallationDate: Installed on 2018-05-31 (349 days ago) InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426) MachineType: Dell Inc. Latitude E6540 ProcFB: 0 inteldrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.0.0-15-generic root=UUID=8a8a023d-2a12-4e14-b187-7beaf4d89763 ro quiet splash vt.handoff=1 RelatedPackageVersions: linux-restricted-modules-5.0.0-15-generic N/A linux-backports-modules-5.0.0-15-generic N/A linux-firmware1.178 SourcePackage: linux UpgradeStatus: Upgraded to disco on 2019-04-23 (21 days ago) dmi.bios.date: 10/09/2018 dmi.bios.vendor: Dell Inc. dmi.bios.version: A26 dmi.board.name: 0725FP dmi.board.vendor: Dell Inc. dmi.board.version: A00 dmi.chassis.type: 9 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvrA26:bd10/09/2018:svnDellInc.:pnLatitudeE6540:pvr00:rvnDellInc.:rn0725FP:rvrA00:cvnDellInc.:ct9:cvr: dmi.product.name: Latitude E6540 dmi.product.sku: 05BE dmi.product.version: 00 dmi.sys.vendor: Dell Inc. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1829255/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1829255] Re: Sever performance degradation after updating to 5.0.0-15 due to mds mitigation
I'll point out that munbi is seeing this hit using the following CPU sig and microcode revision: sig=0x306c3, pf=0x10, revision=0x27 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1829255 Title: Sever performance degradation after updating to 5.0.0-15 due to mds mitigation Status in linux package in Ubuntu: Won't Fix Bug description: After updating to the latest kernel 5.0.0-15 I'm experiencing a severe performance degradation on my machine with one of my applications (a gbc emulator developed by me). I saw about a 27% performance loss, going from 55fps to 40. Booting an old 5.0.0-13 kernel or disabling mds mitigation at boot with 'mds=off' solves the problem. I know there isn't much to do about (apart from disabling the mitigation) but I'm reporting this because I spent half a day bisecting with git cause I thought the problem was with my latest commits. Security updates are obviously important and a priority, but I think that at least with this kind of mitigations that cause big performance degradation, an message or a simple alert to the user during the update should be shown. Something like: "Plese be aware that this kernel update can result in performance loss. See this page for more info." and then link the specific SecurityTeam page (which in this case is https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/MDS ) ProblemType: Bug DistroRelease: Ubuntu 19.04 Package: linux-image-5.0.0-15-generic 5.0.0-15.16 ProcVersionSignature: Ubuntu 5.0.0-15.16-generic 5.0.6 Uname: Linux 5.0.0-15-generic x86_64 ApportVersion: 2.20.10-0ubuntu27 Architecture: amd64 AudioDevicesInUse: USERPID ACCESS COMMAND /dev/snd/controlC0: gabriele 2045 F pulseaudio /dev/snd/controlC1: gabriele 2045 F pulseaudio CurrentDesktop: ubuntu:GNOME Date: Wed May 15 16:36:36 2019 HibernationDevice: RESUME=UUID=e43b75a3-a278-4d4e-83e8-54a9e0c95276 InstallationDate: Installed on 2018-05-31 (349 days ago) InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426) MachineType: Dell Inc. Latitude E6540 ProcFB: 0 inteldrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.0.0-15-generic root=UUID=8a8a023d-2a12-4e14-b187-7beaf4d89763 ro quiet splash vt.handoff=1 RelatedPackageVersions: linux-restricted-modules-5.0.0-15-generic N/A linux-backports-modules-5.0.0-15-generic N/A linux-firmware1.178 SourcePackage: linux UpgradeStatus: Upgraded to disco on 2019-04-23 (21 days ago) dmi.bios.date: 10/09/2018 dmi.bios.vendor: Dell Inc. dmi.bios.version: A26 dmi.board.name: 0725FP dmi.board.vendor: Dell Inc. dmi.board.version: A00 dmi.chassis.type: 9 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvrA26:bd10/09/2018:svnDellInc.:pnLatitudeE6540:pvr00:rvnDellInc.:rn0725FP:rvrA00:cvnDellInc.:ct9:cvr: dmi.product.name: Latitude E6540 dmi.product.sku: 05BE dmi.product.version: 00 dmi.sys.vendor: Dell Inc. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1829255/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1829255] Re: Sever performance degradation after updating to 5.0.0-15 due to mds mitigation
Hello and thanks for the bug report. We hate to hear that you're seeing such a performance hit on your application when the MDS mitigations are enabled. Unfortunately, we are simply following Intel's recommendations[1] for mitigating MDS attacks. The kernel changes are relatively simple and the overhead comes from the kernel calling into the CPU microcode to flush the internal CPU buffers as well as the inefficiencies involved with flushing such buffers. Since the recommendation includes flushing the buffers before exiting from the kernel to userspace, workloads which are syscall heavy are likely to see the largest performance hit. I like your idea of alerting the user of such a potential performance hit, on the surface. However, the vast majority of users won't know how to handle that information and, even worse, it could scare users out of taking the update even though the mitigations may not significantly impact their typical usage. Very few users will have the need to bisect kernel changes to identify a performance decrease that they've measured. Another problem is that there's not a consistent way to alert users with pertinent information. The updates are provided to desktop systems, to headless servers, packaged in pre-built cloud images, delivered automatically to IoT devices that don't support typical user logins, etc. Even across something like desktop systems, users apply the updates in a variety of ways (manually with apt, automatically with unattended- upgrades, with a GUI such as update-manager, etc.). This is why we provide out-of-band information like this in Ubuntu Security Announcements[2] and, in some cases, more verbose KnowledgeBase articles. What I can promise is that we'll continue to work with Intel and the upstream kernel community in the case that future improvements are identified for the existing MDS mitigations. Thanks again for opening this bug report and please don't take the "Won't Fix" bug status as your voice being ignored. At the very least, when writing up the next KnowledgeBase article, I now know that any time we spend describing performance impacts will be much appreciated by someone out there. :) [1] https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling [2] https://usn.ubuntu.com ** Changed in: linux (Ubuntu) Status: Confirmed => Won't Fix -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1829255 Title: Sever performance degradation after updating to 5.0.0-15 due to mds mitigation Status in linux package in Ubuntu: Won't Fix Bug description: After updating to the latest kernel 5.0.0-15 I'm experiencing a severe performance degradation on my machine with one of my applications (a gbc emulator developed by me). I saw about a 27% performance loss, going from 55fps to 40. Booting an old 5.0.0-13 kernel or disabling mds mitigation at boot with 'mds=off' solves the problem. I know there isn't much to do about (apart from disabling the mitigation) but I'm reporting this because I spent half a day bisecting with git cause I thought the problem was with my latest commits. Security updates are obviously important and a priority, but I think that at least with this kind of mitigations that cause big performance degradation, an message or a simple alert to the user during the update should be shown. Something like: "Plese be aware that this kernel update can result in performance loss. See this page for more info." and then link the specific SecurityTeam page (which in this case is https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/MDS ) ProblemType: Bug DistroRelease: Ubuntu 19.04 Package: linux-image-5.0.0-15-generic 5.0.0-15.16 ProcVersionSignature: Ubuntu 5.0.0-15.16-generic 5.0.6 Uname: Linux 5.0.0-15-generic x86_64 ApportVersion: 2.20.10-0ubuntu27 Architecture: amd64 AudioDevicesInUse: USERPID ACCESS COMMAND /dev/snd/controlC0: gabriele 2045 F pulseaudio /dev/snd/controlC1: gabriele 2045 F pulseaudio CurrentDesktop: ubuntu:GNOME Date: Wed May 15 16:36:36 2019 HibernationDevice: RESUME=UUID=e43b75a3-a278-4d4e-83e8-54a9e0c95276 InstallationDate: Installed on 2018-05-31 (349 days ago) InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426) MachineType: Dell Inc. Latitude E6540 ProcFB: 0 inteldrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.0.0-15-generic root=UUID=8a8a023d-2a12-4e14-b187-7beaf4d89763 ro quiet splash vt.handoff=1 RelatedPackageVersions: linux-restricted-modules-5.0.0-15-generic N/A linux-backports-modules-5.0.0-15-generic N/A linux-firmware1.178 SourcePackage: linux UpgradeStatus: Upgraded to disco on 2019-04-23 (21 days ago) dmi.bios.date: 10/09/2018 dmi.bios.vendor: Dell Inc.
