[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
https://bugs.launchpad.net/cloud-images/+bug/2038894 is a related bug to track specifically the introduction of listening port 5353 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
There is a related bug @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2036968 which might have affected boot speed. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
@paelzer agreed. Good plan. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
@phil As I said above for me it mostly was "I'm sure some change was expected, but so much?". It has various benefits as gladly outlined by Dimitri, fixing many issues, but coming at a price tag. Seeing how big the price tag is for small size, high density cases I consider it potentially too much for some. As I said in my report "This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected." so a way forward IMHO would be: 1. We leave mantic as is 2. We certainly need a release notes entry that explains the pros/cons that led to the decision 3. kernel can continue to optimize this case (see all the discussions/suggestions above already) 4. towards 24.04 4a) if this is received well, keep it 4b) if our decision was good, but not accounting for too many use-cases that we can't fix otherwise reconsider having a reduced kernel in 24.04 That would mostly an ask to the kernel team (but needs CPC to ensure it is correct e.g. in making it clear which images change due to that and which won't as they were already based on -generic). Phil/Xnox - what do you think about that approach? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
@paelzer given the above findings and discussion, I would like to mark this as Invalid for cloud-images project and continue the conversation in the context of kernel only. +1 / -1 ? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
Ah. I see. KSM_ENABLED=AUTO from /etc/default/qemu-kvm is used through /usr/share/qemu/init/qemu-kvm-init via the qemu-kvm.service systemd service. thanks for the hint! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
> it only gets activated when you install ksmtuned (which is not installed by default). No, installing qemu-system-... will also enable it. So Seth gladly filed bug 2033565 to discuss and change this now or at least towards 24.04. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
** Description changed: - The Mantic (Ubuntu 23.10) images are undergoing some big changes prior - to 23.10 release in October. + The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ + are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages + * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
@Seth, ksm is disabled by default so it's still opt-in: # cat /sys/kernel/mm/ksm/run 0 it only gets activated when you install ksmtuned (which is not installed by default). So I think that's fine. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
> @paelzer Are you happy to adjust your regression testing/metrics gathering to > increase the > memory required knowing that it was a conscious decision to switch kernel and > incur the > performance hit for the benefit of using a kernel with more support and less > reported bugs? I am. In fact I already have. To be clear, I knew that the change from -kvm to -virtual was a conscious decision (I do not even know how long back I heard xnox talking about it, many years for sure) fixing the many issues xnox mentioned. I just never heard anyone saying or suggesting - and personally didn't expect it to be - that very intense on low-memory / density scenarios. I'd be happy if everyone here could continue the thought of checking if any further package/module could be omitted (or at least not auto- loaded) for the minimal images. Which should tune us back down a bit, even if that is not all the way to where we were before. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
I have uploaded further data now to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/server-metrics/ with kernelmodules, kernelconfig, services, timers etc. for each of the three images being inspected. This additional data was gathered with a modified fork of the `server-test- scripts` repo @ https://github.com/philroche/server-test- scripts/blob/feature/local-lxc-image-execution-additional-data- gathering/metric-server-simple/metric-server-simple.sh. It seems that most of the mem and process increase is attributed to the kernel change and we know that this was a conscious decision with the following reported bugs supporting that decision. * https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2006488 * https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1931841 * https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1685291 Given the above I feel what we can work on is whether any of the process/modules introduced by the switch to the generic kernel should be omitted for the minimal images. The best, easiest source of this information is the data gathered from the latest image with both the generic kernel and the switch to the new minimal seed - https://people.canonical.com/~philroche/20230824-manticl- minimal-LP2032933/server-metrics/20230821.1-after-kernel-change-after- seed-change-mantic-minimal-cloudimg-amd64-data-f93870221eb8/ @seth-arnold You highlighted `ksmd`. Are there any others that concern you. @paelzer Are you happy to adjust your regression testing/metrics gathering to increase the memory required knowing that it was a conscious decision to switch kernel and incur the performance hit for the benefit of using a kernel with more support and less reported bugs? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
Is lxd / zfs now installed by default in the minimal images? It seems like some zfs things are loaded by default. Also counting processes alone, may or may not increase memory usage. Example: having N agetty, doesn't actually have memory cost of each individual agetty, does it? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
Additionally if something is built into the kernel that is optional at runtime, and resource intensive, we can make it modularization. But not sure that will win us anything if it gets autoloaded by default. Thus maybe some of the things listed should also be blacklisted to prevent auto-loading on boot. Or even filtered / excluded from the minimal image (i.e. demotion to modules-extra, or dpkg filter-package skipped for minimal images). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
Separately, please do note that switch from kvm kernel to generic is resolving the hundreds of bug reports of "works with generic / every other cloud, doesn't work with kvm". Also note that probably the baseline comparison shouldn't be the kvm kernel => as we know kvm kernel has never provided complete Ubuntu experience the way cloud kernels do (aws,azure,gcp,oracle) nor the generic kernel (most baremetal installs). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
> +ksmd I'm concerned about this change. Historically, the page-merging code has allowed cross-VM snooping, including even recovery of GnuPG private keys: https://eprint.iacr.org/2013/448.pdf Unless something has changed to mitigate the cross-domain privacy leaks in ksmd, it ought to be opt-in for administrators to select if all their VMs are in the same security domain. Thanks -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
@paelzer > The change of the image build sadly combined it all See the description noting https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ which should help in determining where the changes were introduced as I have provided three images across the various stages of changes - no change -> new kernel -> new kernel + new seed For example https://pastebin.ubuntu.com/p/sJ5wGk4G7h/ show the process diff between previous images and images with new kernel only -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
The diff in process count from kernel change image -> kernel change + seed change image is actually a reduction in processes - see diff @ https://pastebin.ubuntu.com/p/PXtQM9gB2K/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
In the above tests from Christian, it's interesting to note the MemTotal shrink by ~22MiB. Is this due to the higher NR_CPUS alone? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
IMHO: If it turns out unfixable, this at least needs to be a release notes entry -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
One thing that came up when discussing with SMB as an obvious "uses mem more in a kernel" are the structures needed per POSSIBLE cpu. Comparing the system that I downgraded I found from [1]: With the -kvm kernel before: kernel: setup_percpu: NR_CPUS:64 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1 kernel: rcu: RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1. With the -generic kernel: kernel: setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1 kernel: rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=1. So let us try the new 6.3 generic kernel, but with kernel parameter maxcpus=1 possible_cpus=1 nr_cpus=1 ^^ this might be too hard, but ok for testing That gives me: kernel: setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1 But sadly, the memory does not look much better: root@m-new:~# cat /proc/meminfo MemTotal: 956176 kB MemFree: 640960 kB MemAvailable: 664816 kB Was a good idea, but sadly that wasn't it. And the initial NR_CPUS is a compile time parameter which we can not tune down :-/ [1]: https://github.com/torvalds/linux/blob/master/arch/x86/kernel/setup_percpu.c#L122 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
Checking kernels: 1. This already uses linux-image-virtual, it is not the even bigger linux-image-generic. 2. The change of the image build sadly combined it all a) new image build/seeding by CPC b) different kernel type -kvm -> -virtual that uses generic c) switch of kernel versions We did not yet think about 2c here afaics. This is not just -kvm -> -generic. But instead 6.2.0-1003-kvm -> 6.3.0-7-generic 6.3.0-7-kvm does not exist (not built anymore), so to check which of the changes broke this we might need to look at. The closest comparison I'd find is - https://launchpad.net/ubuntu/+source/linux/6.2.0-21.21 - https://launchpad.net/ubuntu/+source/linux-signed/6.2.0-21.21 Installing that on my system The new system with the high memory consumption has: linux-image-6.3.0-7-generic 6.3.0-7.7+1 linux-modules-6.3.0-7-generic 6.3.0-7.7 I'm comparing for: https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/bootstrap/+build/25980736/+files/linux-modules-6.2.0-21-generic_6.2.0-21.21_amd64.deb https://launchpad.net/~canonical-signing/+archive/ubuntu/primary-2022v1/+build/25984194/+files/linux-image-6.2.0-21-generic_6.2.0-21.21_amd64.deb root@m-old:~# cat /proc/meminfo MemTotal: 955692 kB MemFree: 630128 kB MemAvailable: 653536 kB So that lets us assume, that the move 6.2->6.3 was not what broke it (actually 6.3 is even a little bit better). The cause really seems to be more the move -kvm -> -generic. I'm sure some change was expected, but so much? Adding Linux bug tasks for them to chime in. ** Also affects: linux (Ubuntu) Importance: Undecided Status: New ** Changed in: linux (Ubuntu) Status: New => Confirmed ** Changed in: cloud-images Status: New => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
