[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
@xnox markig procps as wontfix since the kernel fix seems to be sufficient. ** Changed in: procps (Ubuntu) Status: New => Won't Fix ** Changed in: procps (Ubuntu Xenial) Status: New => Won't Fix ** Changed in: procps (Ubuntu Artful) Status: New => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
** Changed in: ubuntu-z-systems Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Commit 2a8a98673c13 has been in Xenial and Artful for some time now. Can you confirm this bug is now resolved with the latest updates. ** Changed in: linux (Ubuntu Artful) Status: In Progress => Fix Released ** Changed in: linux (Ubuntu Xenial) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
** No longer affects: linux (Ubuntu Zesty) ** No longer affects: procps (Ubuntu Zesty) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
** Changed in: linux (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Zesty is end of life. ** Changed in: procps (Ubuntu Zesty) Status: New => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
zesty is End Of Life ** Changed in: linux (Ubuntu Zesty) Status: In Progress => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
** Changed in: ubuntu-z-systems Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Commit 2a8a98673c13 has been in Xenial since Ubuntu-4.4.0-98.121. If you can confirm the bug is now fixed since this kernel version, I'll mark the bug as Fix Released. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Hi Heinz-Werner, thanks for the ping. I think we agreed back then that the config itself is a Ubuntu wide thing - but I haven't seen any discussions on procps or in general to lift it. But then as I outlined in comment #7 any limit can be too low. I'll set the KVM task to won't fix to clearly mark it as "not a kvm task". The fix that you need thou is the kernel change to account correctly - so that part of the question goes to the Kernel Team if the change that was proposed back then was actually released. @Joseph - we clearly passed the number, but was the change released with it? ** Changed in: kvm (Ubuntu) Status: Confirmed => Won't Fix ** No longer affects: kvm (Ubuntu Xenial) ** No longer affects: kvm (Ubuntu Zesty) ** No longer affects: kvm (Ubuntu Artful) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Hi Scott, the howto is mixed for Desktop users, Server users and selective upgrades. For your case you only need the most simple case which would be: Essentially you want to: # Check - all other updates done (to clear the view) $ apt list --upgradable Listing... Done # Enable proposed for z on Server $ echo "deb http://ports.ubuntu.com/ubuntu-ports/ xenial-proposed main restricted universe multiverse" | sudo tee /etc/apt/sources.list.d/enable-proposed.list $ sudo apt update $ apt list --upgradable [...] linux-headers-generic/xenial-proposed 4.4.0.100.105 s390x [upgradable from: 4.4.0.98.103] linux-headers-virtual/xenial-proposed 4.4.0.100.105 s390x [upgradable from: 4.4.0.98.103] linux-image-virtual/xenial-proposed 4.4.0.100.105 s390x [upgradable from: 4.4.0.98.103] # Install just the kernels from proposed $ sudo apt install linux-generic No need to set apt prefs if you only do a selective install. If you'd do a global "sudo apt upgrade" you'd get all, but that is likely not what you want in your case. After you have done so you can just enable/disable the line in /etc/apt/sources.list.d/enable-proposed.list as needed. Hope that helps -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Commit 2a8a98673c13 is in Xenial -proposed, kernel version: Ubuntu-4.4.0-98.121. It looks like you tested with version 4.4.0-62-generic. Can you try testing with the -proposed Xenial kernel: See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Another option is to test with the current -updates Artful kernel. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
** Changed in: ubuntu-z-systems Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Actually, Artful already has commit 2a8a98673c13. It was pulled in for bug 1718397. It looks like Xenial and Zesty will also get that commit via that bug. Can you test Artful and/or X/Y -proposed to see if this bug is resolved with that commit? ** Also affects: procps (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: kvm (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: procps (Ubuntu Zesty) Importance: Undecided Status: New ** Also affects: kvm (Ubuntu Zesty) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Zesty) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Zesty) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Zesty) Status: New => In Progress ** Changed in: linux (Ubuntu Zesty) Assignee: (unassigned) => Joseph Salisbury (jsalisbury) ** Changed in: linux (Ubuntu Xenial) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Xenial) Status: New => In Progress ** Changed in: linux (Ubuntu Xenial) Assignee: (unassigned) => Joseph Salisbury (jsalisbury) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
I'll build an Artful test kernel with commit 2a8a98673c13 and post it shortly. ** Changed in: linux (Ubuntu) Status: New => In Progress ** Changed in: linux (Ubuntu) Importance: Undecided => Medium ** Changed in: linux (Ubuntu) Assignee: (unassigned) => Joseph Salisbury (jsalisbury) ** Also affects: procps (Ubuntu Artful) Importance: Undecided Status: New ** Also affects: kvm (Ubuntu Artful) Importance: Undecided Assignee: Skipper Bug Screeners (skipper-screen-team) Status: Confirmed ** Also affects: linux (Ubuntu Artful) Importance: Medium Assignee: Joseph Salisbury (jsalisbury) Status: In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Thanks for checking cborntra! The referred patch is in 4.14-rc1. I'm not so sure on backports of this, but at least for the Artful and HWE kernel of 4.13 this would be good to have. @Kernel Team - could you take a look and consider 2a8a9867 for our 4.13 kernels? @Xnox - changing the global default is a Foundations thing, would you support the 128k change and would you ask in the Team for opinions? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Yeah, such a change needs some reasoning and since (at least to me) it is not yet clear how such few guests consume so much aio resources we need to check that. If you can provide: 1. the increase per guest spawned 2. are all your guests having more or less the same xml or do some have more disks or more iothreads? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Any updates? Have you been able to monitor fs.aio-nr exhaustion? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Hi, I think this is more a "is it time to rethink the default fs.aio-max-nr than a qemu problem. In general pick any limit and you will be able to create a testcase breaking it. Are all these bugs ... do we need to set all limits to unlimited - I'm sure you agree not to. Anyway I agree to Viktor that "I would also suggest that the system default setting is increased" as this could be hit by many other aio workload. But that also means it has to be a broader discussion. Kernel default: fs/aio.c:194:unsigned long aio_max_nr = 0x1; /* system wide maximum number of aio requests */ Options: 1. minor kernel patch as ubuntu delta 2. a file in /etc/sysctl.d/ in package procps 3. a file in /etc/sysctl.d/ in another package 4. deny As it is a kernel limit and a kernel default value I'll add a kernel and a procps task for discussion with kernel and foundations Team. They also might have experience on those "limit raising" cases and how they were handled. Related docs: https://www.kernel.org/doc/Documentation/sysctl/fs.txt http://man7.org/linux/man-pages/man2/io_setup.2.html ** Also affects: procps (Ubuntu) Importance: Undecided Status: New ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
OTOH I wonder to some extend how you exceded the 65k with "just" 18 guests. Not strictly required, but it might be very interesting before your system becomes unavailable to track: $ sysctl fs.aio-nr while starting the guests. How much get added per guest, would the settle down after the guest is done with initial rampup. ** Changed in: kvm (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
** Changed in: ubuntu-z-systems Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
** Also affects: ubuntu-z-systems Importance: Undecided Status: New ** Changed in: ubuntu-z-systems Assignee: (unassigned) => Canonical Server Team (canonical-server) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
** Package changed: linux (Ubuntu) => kvm (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/kvm/+bug/1717224/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
