[Bug 1435363] Re: KSM causing performance and instability issues
** Changed in: linux (Ubuntu) Assignee: Chris J Arges (arges) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1435363 Title: KSM causing performance and instability issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1435363] Re: KSM causing performance and instability issues
Hi, I installed a new machine from the 14.04.2 media which gave me the HWE stack with kernel 3.16.0-33-generic and it's running with no problems. The machine is now loaded up to 125GB worth of VMs and with the following memory stats: # free -m total used free sharedbuffers cached Mem:257599 253046 4553 3218 161499 -/+ buffers/cache: 91328 166271 Swap:0 0 0 So, the server is quite loaded and I haven't seen any hiccups. We'll see from there on. Thanks, Mohammed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1435363 Title: KSM causing performance and instability issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1435363] Re: KSM causing performance and instability issues
Hi Chris, Thanks for the help so far. I'm deploying a new machine right now and I'll be trying to replicate it on -48. The way I detected it was that i'd see messaging in dmesg on guest similar to this: hrtimer: interrupt took 4352551231 ns In addition, when pinging the machine, you'd have a few seconds of stable pings, then unresponsive for 2-3s, and it starts responding again (with a huge delay, latency of 3s to 4s because of the delay). I will be running this machine and monitoring it closely and report on the output, however, I'd like to note that these machines have heavy KSM usage, before turning it off, one had almost ~45-50GB of deduplicated memory on a 256GB node, so I'm not sure if that plays in as a factor.. I'll report back on -48 and see what I can check Thank you, Mohammed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1435363 Title: KSM causing performance and instability issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1435363] Re: KSM causing performance and instability issues
apport information ** Tags added: apport-collected trusty ** Description changed: This seems to be a bug that has regressed, I have encountered the same issue as 2 other reports: LP: #1349897 Running kernel: 3.13.0-46-generic This is replicated over many compute nodes (KVM) running OpenStack. The workaround is to disable KSM: echo 2 /sys/kernel/mm/ksm/run This fixes the issue temporarily. + --- + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 Mar 15 02:22 seq + crw-rw 1 root audio 116, 33 Mar 15 02:22 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.14.1-0ubuntu3.7 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + CRDA: Error: [Errno 2] No such file or directory + DistroRelease: Ubuntu 14.04 + InstallationDate: Installed on 2015-03-15 (8 days ago) + InstallationMedia: Ubuntu-Server 14.04.1 LTS Trusty Tahr - Release amd64 (20140722.3) + Lsusb: + Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub + Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub + Bus 001 Device 003: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon + Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub + Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub + MachineType: Supermicro X9DRT-HF+ + Package: linux (not installed) + PciMultimedia: + + ProcFB: 0 VESA VGA + ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-32-generic root=UUID=b1b3ee88-e8fc-4a36-9fd3-1b0e3da6ca5a ro nomdmonddf nomdmonisw + ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4 + RelatedPackageVersions: + linux-restricted-modules-3.13.0-32-generic N/A + linux-backports-modules-3.13.0-32-generic N/A + linux-firmware 1.127.11 + RfKill: Error: [Errno 2] No such file or directory + Tags: trusty + Uname: Linux 3.13.0-32-generic x86_64 + UpgradeStatus: No upgrade log present (probably fresh install) + UserGroups: + + _MarkForUpload: True + dmi.bios.date: 01/21/2014 + dmi.bios.vendor: American Megatrends Inc. + dmi.bios.version: 3.0b + dmi.board.asset.tag: To be filled by O.E.M. + dmi.board.name: X9DRT-HF+ + dmi.board.vendor: Supermicro + dmi.board.version: 0123456789 + dmi.chassis.asset.tag: To Be Filled By O.E.M. + dmi.chassis.type: 17 + dmi.chassis.vendor: Supermicro + dmi.chassis.version: 0123456789 + dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3.0b:bd01/21/2014:svnSupermicro:pnX9DRT-HF+:pvr0123456789:rvnSupermicro:rnX9DRT-HF+:rvr0123456789:cvnSupermicro:ct17:cvr0123456789: + dmi.product.name: X9DRT-HF+ + dmi.product.version: 0123456789 + dmi.sys.vendor: Supermicro ** Attachment added: BootDmesg.txt https://bugs.launchpad.net/bugs/1435363/+attachment/4353426/+files/BootDmesg.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1435363 Title: KSM causing performance and instability issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1435363] Re: KSM causing performance and instability issues
I'll be setting up a new server in the next few days, I'll attempt to use -48 and see if that issue is present or not. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1435363 Title: KSM causing performance and instability issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1435363] Re: KSM causing performance and instability issues
apport information ** Attachment removed: BootDmesg.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+attachment/4353426/+files/BootDmesg.txt ** Attachment removed: CurrentDmesg.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+attachment/4353427/+files/CurrentDmesg.txt ** Attachment removed: IwConfig.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+attachment/4353428/+files/IwConfig.txt ** Attachment removed: Lspci.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+attachment/4353429/+files/Lspci.txt ** Attachment removed: ProcCpuinfo.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+attachment/4353430/+files/ProcCpuinfo.txt ** Attachment removed: ProcEnviron.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+attachment/4353431/+files/ProcEnviron.txt ** Attachment removed: ProcInterrupts.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+attachment/4353432/+files/ProcInterrupts.txt ** Attachment removed: ProcModules.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+attachment/4353433/+files/ProcModules.txt ** Attachment removed: UdevDb.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+attachment/4353434/+files/UdevDb.txt ** Attachment removed: UdevLog.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+attachment/4353435/+files/UdevLog.txt ** Attachment removed: WifiSyslog.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+attachment/4353436/+files/WifiSyslog.txt ** Description changed: This seems to be a bug that has regressed, I have encountered the same issue as 2 other reports: LP: #1349897 Running kernel: 3.13.0-46-generic This is replicated over many compute nodes (KVM) running OpenStack. The workaround is to disable KSM: echo 2 /sys/kernel/mm/ksm/run This fixes the issue temporarily. - --- - AlsaDevices: - total 0 - crw-rw 1 root audio 116, 1 Mar 15 02:22 seq - crw-rw 1 root audio 116, 33 Mar 15 02:22 timer - AplayDevices: Error: [Errno 2] No such file or directory - ApportVersion: 2.14.1-0ubuntu3.7 - Architecture: amd64 - ArecordDevices: Error: [Errno 2] No such file or directory - AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: - CRDA: Error: [Errno 2] No such file or directory - DistroRelease: Ubuntu 14.04 - InstallationDate: Installed on 2015-03-15 (8 days ago) - InstallationMedia: Ubuntu-Server 14.04.1 LTS Trusty Tahr - Release amd64 (20140722.3) - Lsusb: - Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub - Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub - Bus 001 Device 003: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon - Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub - Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub - MachineType: Supermicro X9DRT-HF+ - Package: linux (not installed) - PciMultimedia: - - ProcFB: 0 VESA VGA - ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-32-generic root=UUID=b1b3ee88-e8fc-4a36-9fd3-1b0e3da6ca5a ro nomdmonddf nomdmonisw - ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4 - RelatedPackageVersions: - linux-restricted-modules-3.13.0-32-generic N/A - linux-backports-modules-3.13.0-32-generic N/A - linux-firmware 1.127.11 - RfKill: Error: [Errno 2] No such file or directory - Tags: trusty - Uname: Linux 3.13.0-32-generic x86_64 - UpgradeStatus: No upgrade log present (probably fresh install) - UserGroups: - - _MarkForUpload: True - dmi.bios.date: 01/21/2014 - dmi.bios.vendor: American Megatrends Inc. - dmi.bios.version: 3.0b - dmi.board.asset.tag: To be filled by O.E.M. - dmi.board.name: X9DRT-HF+ - dmi.board.vendor: Supermicro - dmi.board.version: 0123456789 - dmi.chassis.asset.tag: To Be Filled By O.E.M. - dmi.chassis.type: 17 - dmi.chassis.vendor: Supermicro - dmi.chassis.version: 0123456789 - dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3.0b:bd01/21/2014:svnSupermicro:pnX9DRT-HF+:pvr0123456789:rvnSupermicro:rnX9DRT-HF+:rvr0123456789:cvnSupermicro:ct17:cvr0123456789: - dmi.product.name: X9DRT-HF+ - dmi.product.version: 0123456789 - dmi.sys.vendor: Supermicro ** Description changed: This seems to be a bug that has regressed, I have encountered the same issue as 2 other reports: LP: #1349897 Running kernel: 3.13.0-46-generic This is replicated over many compute nodes (KVM) running OpenStack. The workaround is to disable KSM: echo 2 /sys/kernel/mm/ksm/run This fixes the issue temporarily. + --- + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 Mar 22 14:30 seq + crw-rw 1 root audio 116, 33 Mar 22 14:30 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.14.1-0ubuntu3.7 + Architecture: amd64 + ArecordDevices:
[Bug 1435363] Re: KSM causing performance and instability issues
** Description changed: This seems to be a bug that has regressed, I have encountered the same issue as 2 other reports: + LP: #1346917 LP: #1349897 Running kernel: 3.13.0-46-generic This is replicated over many compute nodes (KVM) running OpenStack. The workaround is to disable KSM: echo 2 /sys/kernel/mm/ksm/run This fixes the issue temporarily. - --- + --- AlsaDevices: - total 0 - crw-rw 1 root audio 116, 1 Mar 22 14:30 seq - crw-rw 1 root audio 116, 33 Mar 22 14:30 timer + total 0 + crw-rw 1 root audio 116, 1 Mar 22 14:30 seq + crw-rw 1 root audio 116, 33 Mar 22 14:30 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.7 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 InstallationDate: Installed on 2014-12-14 (99 days ago) InstallationMedia: Ubuntu-Server 14.04.1 LTS Trusty Tahr - Release amd64 (20140722.3) MachineType: Dell Inc. PowerEdge R620 Package: linux (not installed) PciMultimedia: - + ProcFB: 0 EFI VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-46-generic.efi.signed root=UUID=68d30a86-3c67-4691-b142-d27a459986e8 ro ProcVersionSignature: Ubuntu 3.13.0-46.79-generic 3.13.11-ckt15 RelatedPackageVersions: - linux-restricted-modules-3.13.0-46-generic N/A - linux-backports-modules-3.13.0-46-generic N/A - linux-firmware 1.127.11 + linux-restricted-modules-3.13.0-46-generic N/A + linux-backports-modules-3.13.0-46-generic N/A + linux-firmware 1.127.11 RfKill: Error: [Errno 2] No such file or directory Tags: trusty Uname: Linux 3.13.0-46-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: - + _MarkForUpload: True dmi.bios.date: 01/16/2014 dmi.bios.vendor: Dell Inc. dmi.bios.version: 2.2.2 dmi.board.name: 01W23F dmi.board.vendor: Dell Inc. dmi.board.version: A05 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr2.2.2:bd01/16/2014:svnDellInc.:pnPowerEdgeR620:pvr:rvnDellInc.:rn01W23F:rvrA05:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R620 dmi.sys.vendor: Dell Inc. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1435363 Title: KSM causing performance and instability issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1435363] Re: KSM causing performance and instability issues
I've attempted to reproduce bug 1346917 again on a NUMA machine and was unable to do so with the latest 3.13 kernel. Perhaps I could have more details on how your reproducing this issue to assist with debugging? Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1435363 Title: KSM causing performance and instability issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1435363] Re: KSM causing performance and instability issues
** Changed in: linux (Ubuntu) Importance: Undecided = High ** Tags added: kernel-da-key -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1435363 Title: KSM causing performance and instability issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1435363] Re: KSM causing performance and instability issues
Can you also test with /sys/kernel/mm/ksm/merge_across_nodes set to 0 (and KSM enabled normally) and confirm this is a KSM+NUMA issue? Another way to potentially cause this issue to occur faster would be to set: /sys/kernel/mm/ksm/sleep_millisecs to a value lower than 200 or /sys/kernel/mm/ksm/pages_to_scan to a much larger number than 100 Thanks, --chris j arges ** Changed in: linux (Ubuntu) Assignee: (unassigned) = Chris J Arges (arges) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1435363 Title: KSM causing performance and instability issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1435363] Re: KSM causing performance and instability issues
In addition could you explain more in depth about the performance an instability issues you've observed? How do you detect them or test for them? Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1435363 Title: KSM causing performance and instability issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1435363/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
