[Group.of.nepali.translators] [Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
@xnox markig procps as wontfix since the kernel fix seems to be sufficient. ** Changed in: procps (Ubuntu) Status: New => Won't Fix ** Changed in: procps (Ubuntu Xenial) Status: New => Won't Fix ** Changed in: procps (Ubuntu Artful) Status: New => Won't Fix -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value Status in Ubuntu on IBM z Systems: Fix Released Status in kvm package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Fix Released Status in procps package in Ubuntu: Won't Fix Status in linux source package in Xenial: Fix Released Status in procps source package in Xenial: Won't Fix Status in linux source package in Artful: Fix Released Status in procps source package in Artful: Won't Fix Bug description: Starting virtual guests via on Ubuntu 16.04.2 LTS installed with its KVM hypervisor on an IBM Z14 system LPAR fails on the 18th guest with the following error: root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70038 error: Failed to start domain zs93kag70038 error: internal error: process exited while connecting to monitor: 2017-07-26T01:48:26.352534Z qemu-kvm: -drive file=/guestimages/data1/zs93kag70038.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=native: Could not open backing file: Could not set AIO state: Inappropriate ioctl for device The previous 17 guests started fine: root@zm93k8# virsh start zs93kag70020 Domain zs93kag70020 started root@zm93k8# virsh start zs93kag70021 Domain zs93kag70021 started . . root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70036 Domain zs93kag70036 started We ended up fixing the issue by adding the following line to /etc/sysctl.conf : fs.aio-max-nr = 4194304 ... then, reload the sysctl config file: root@zm93k8:/etc# sysctl -p /etc/sysctl.conf fs.aio-max-nr = 4194304 Now, we're able to start more guests... root@zm93k8:/etc# virsh start zs93kag70036 Domain zs93kag70036 started The default value was originally set to 65535: root@zm93k8:/rawimages/ubu1604qcow2# cat /proc/sys/fs/aio-max-nr 65536 Note, we chose the 4194304 value, because this is what our KVM on System Z hypervisor ships as its default value. Eg. on our zKVM system: [root@zs93ka ~]# cat /proc/sys/fs/aio-max-nr 4194304 ubuntu@zm93k8:/etc$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 16.04.2 LTS Release:16.04 Codename: xenial ubuntu@zm93k8:/etc$ ubuntu@zm93k8:/etc$ dpkg -s qemu-kvm |grep Version Version: 1:2.5+dfsg-5ubuntu10.8 Is something already documented for Ubuntu KVM users warning them about the low default value, and some guidance as to how to select an appropriate value? Also, would you consider increasing the default aio-max-nr value to something much higher, to accommodate significantly more virtual guests? Thanks! ---uname output--- ubuntu@zm93k8:/etc$ uname -a Linux zm93k8 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:12:54 UTC 2017 s390x s390x s390x GNU/Linux Machine Type = z14 ---Debugger--- A debugger is not configured ---Steps to Reproduce--- See Problem Description. The problem was happening a week ago, so this may not reflect that activity. This file was collected on Aug 7, one week after we were hitting the problem. If I need to reproduce the problem and get fresh data, please let me know. /var/log/messages doesn't exist on this system, so I provided syslog output instead. All data have been collected too late after the problem was observed over a week ago. If you need me to reproduce the problem and get new data, please let me know. That's not a problem. Also, we would have to make special arrangements for login access to these systems. I'm happy to run traces and data collection for you as needed. If that's not sufficient, then we'll explore log in access for you. Thanks... - Scott G. I was able to successfully recreate the problem and captured / attached new debug docs. Recreate procedure: # Started out with no virtual guests running. ubuntu@zm93k8:/home/scottg$ virsh list IdName State # Set fs.aio-max-nr back to original Ubuntu "out of the box" value in /etc/sysctl.conf ubuntu@zm93k8:~$ tail -1 /etc/sysctl.conf fs.aio-max-nr = 65536 ## sysctl -a shows: fs.aio-max-nr = 4194304 ## Reload sysctl. ubuntu@zm93k8:~$ sudo sysctl -p /etc/sysctl.conf fs.aio-max-nr = 65536 ubuntu@zm93k8:~$ ubuntu@zm93k8:~$ sudo sysctl -a |grep
[Group.of.nepali.translators] [Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
** Changed in: ubuntu-z-systems Status: In Progress => Fix Released -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value Status in Ubuntu on IBM z Systems: Fix Released Status in kvm package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Fix Released Status in procps package in Ubuntu: New Status in linux source package in Xenial: Fix Released Status in procps source package in Xenial: New Status in linux source package in Artful: Fix Released Status in procps source package in Artful: New Bug description: Starting virtual guests via on Ubuntu 16.04.2 LTS installed with its KVM hypervisor on an IBM Z14 system LPAR fails on the 18th guest with the following error: root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70038 error: Failed to start domain zs93kag70038 error: internal error: process exited while connecting to monitor: 2017-07-26T01:48:26.352534Z qemu-kvm: -drive file=/guestimages/data1/zs93kag70038.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=native: Could not open backing file: Could not set AIO state: Inappropriate ioctl for device The previous 17 guests started fine: root@zm93k8# virsh start zs93kag70020 Domain zs93kag70020 started root@zm93k8# virsh start zs93kag70021 Domain zs93kag70021 started . . root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70036 Domain zs93kag70036 started We ended up fixing the issue by adding the following line to /etc/sysctl.conf : fs.aio-max-nr = 4194304 ... then, reload the sysctl config file: root@zm93k8:/etc# sysctl -p /etc/sysctl.conf fs.aio-max-nr = 4194304 Now, we're able to start more guests... root@zm93k8:/etc# virsh start zs93kag70036 Domain zs93kag70036 started The default value was originally set to 65535: root@zm93k8:/rawimages/ubu1604qcow2# cat /proc/sys/fs/aio-max-nr 65536 Note, we chose the 4194304 value, because this is what our KVM on System Z hypervisor ships as its default value. Eg. on our zKVM system: [root@zs93ka ~]# cat /proc/sys/fs/aio-max-nr 4194304 ubuntu@zm93k8:/etc$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 16.04.2 LTS Release:16.04 Codename: xenial ubuntu@zm93k8:/etc$ ubuntu@zm93k8:/etc$ dpkg -s qemu-kvm |grep Version Version: 1:2.5+dfsg-5ubuntu10.8 Is something already documented for Ubuntu KVM users warning them about the low default value, and some guidance as to how to select an appropriate value? Also, would you consider increasing the default aio-max-nr value to something much higher, to accommodate significantly more virtual guests? Thanks! ---uname output--- ubuntu@zm93k8:/etc$ uname -a Linux zm93k8 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:12:54 UTC 2017 s390x s390x s390x GNU/Linux Machine Type = z14 ---Debugger--- A debugger is not configured ---Steps to Reproduce--- See Problem Description. The problem was happening a week ago, so this may not reflect that activity. This file was collected on Aug 7, one week after we were hitting the problem. If I need to reproduce the problem and get fresh data, please let me know. /var/log/messages doesn't exist on this system, so I provided syslog output instead. All data have been collected too late after the problem was observed over a week ago. If you need me to reproduce the problem and get new data, please let me know. That's not a problem. Also, we would have to make special arrangements for login access to these systems. I'm happy to run traces and data collection for you as needed. If that's not sufficient, then we'll explore log in access for you. Thanks... - Scott G. I was able to successfully recreate the problem and captured / attached new debug docs. Recreate procedure: # Started out with no virtual guests running. ubuntu@zm93k8:/home/scottg$ virsh list IdName State # Set fs.aio-max-nr back to original Ubuntu "out of the box" value in /etc/sysctl.conf ubuntu@zm93k8:~$ tail -1 /etc/sysctl.conf fs.aio-max-nr = 65536 ## sysctl -a shows: fs.aio-max-nr = 4194304 ## Reload sysctl. ubuntu@zm93k8:~$ sudo sysctl -p /etc/sysctl.conf fs.aio-max-nr = 65536 ubuntu@zm93k8:~$ ubuntu@zm93k8:~$ sudo sysctl -a |grep fs.aio-max-nr fs.aio-max-nr = 65536 ubuntu@zm93k8:~$ cat /proc/sys/fs/aio-max-nr 65536 # Attempt to start more than 17 qcow2 virtual guests on the Ubuntu host. Fails on the 18th XML. Script used to start guests..
[Group.of.nepali.translators] [Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Commit 2a8a98673c13 has been in Xenial and Artful for some time now. Can you confirm this bug is now resolved with the latest updates. ** Changed in: linux (Ubuntu Artful) Status: In Progress => Fix Released ** Changed in: linux (Ubuntu Xenial) Status: In Progress => Fix Released -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value Status in Ubuntu on IBM z Systems: In Progress Status in kvm package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Fix Released Status in procps package in Ubuntu: New Status in linux source package in Xenial: Fix Released Status in procps source package in Xenial: New Status in linux source package in Artful: Fix Released Status in procps source package in Artful: New Bug description: Starting virtual guests via on Ubuntu 16.04.2 LTS installed with its KVM hypervisor on an IBM Z14 system LPAR fails on the 18th guest with the following error: root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70038 error: Failed to start domain zs93kag70038 error: internal error: process exited while connecting to monitor: 2017-07-26T01:48:26.352534Z qemu-kvm: -drive file=/guestimages/data1/zs93kag70038.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=native: Could not open backing file: Could not set AIO state: Inappropriate ioctl for device The previous 17 guests started fine: root@zm93k8# virsh start zs93kag70020 Domain zs93kag70020 started root@zm93k8# virsh start zs93kag70021 Domain zs93kag70021 started . . root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70036 Domain zs93kag70036 started We ended up fixing the issue by adding the following line to /etc/sysctl.conf : fs.aio-max-nr = 4194304 ... then, reload the sysctl config file: root@zm93k8:/etc# sysctl -p /etc/sysctl.conf fs.aio-max-nr = 4194304 Now, we're able to start more guests... root@zm93k8:/etc# virsh start zs93kag70036 Domain zs93kag70036 started The default value was originally set to 65535: root@zm93k8:/rawimages/ubu1604qcow2# cat /proc/sys/fs/aio-max-nr 65536 Note, we chose the 4194304 value, because this is what our KVM on System Z hypervisor ships as its default value. Eg. on our zKVM system: [root@zs93ka ~]# cat /proc/sys/fs/aio-max-nr 4194304 ubuntu@zm93k8:/etc$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 16.04.2 LTS Release:16.04 Codename: xenial ubuntu@zm93k8:/etc$ ubuntu@zm93k8:/etc$ dpkg -s qemu-kvm |grep Version Version: 1:2.5+dfsg-5ubuntu10.8 Is something already documented for Ubuntu KVM users warning them about the low default value, and some guidance as to how to select an appropriate value? Also, would you consider increasing the default aio-max-nr value to something much higher, to accommodate significantly more virtual guests? Thanks! ---uname output--- ubuntu@zm93k8:/etc$ uname -a Linux zm93k8 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:12:54 UTC 2017 s390x s390x s390x GNU/Linux Machine Type = z14 ---Debugger--- A debugger is not configured ---Steps to Reproduce--- See Problem Description. The problem was happening a week ago, so this may not reflect that activity. This file was collected on Aug 7, one week after we were hitting the problem. If I need to reproduce the problem and get fresh data, please let me know. /var/log/messages doesn't exist on this system, so I provided syslog output instead. All data have been collected too late after the problem was observed over a week ago. If you need me to reproduce the problem and get new data, please let me know. That's not a problem. Also, we would have to make special arrangements for login access to these systems. I'm happy to run traces and data collection for you as needed. If that's not sufficient, then we'll explore log in access for you. Thanks... - Scott G. I was able to successfully recreate the problem and captured / attached new debug docs. Recreate procedure: # Started out with no virtual guests running. ubuntu@zm93k8:/home/scottg$ virsh list IdName State # Set fs.aio-max-nr back to original Ubuntu "out of the box" value in /etc/sysctl.conf ubuntu@zm93k8:~$ tail -1 /etc/sysctl.conf fs.aio-max-nr = 65536 ## sysctl -a shows: fs.aio-max-nr = 4194304 ## Reload sysctl. ubuntu@zm93k8:~$ sudo sysctl -p /etc/sysctl.conf fs.aio-max-nr = 65536 ubuntu@zm93k8:~$ ubuntu@zm93k8:~$ sudo sysctl -a |grep fs.aio-max-nr
[Group.of.nepali.translators] [Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
** No longer affects: linux (Ubuntu Zesty) ** No longer affects: procps (Ubuntu Zesty) -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value Status in Ubuntu on IBM z Systems: In Progress Status in kvm package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Fix Released Status in procps package in Ubuntu: New Status in linux source package in Xenial: In Progress Status in procps source package in Xenial: New Status in linux source package in Artful: In Progress Status in procps source package in Artful: New Bug description: Starting virtual guests via on Ubuntu 16.04.2 LTS installed with its KVM hypervisor on an IBM Z14 system LPAR fails on the 18th guest with the following error: root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70038 error: Failed to start domain zs93kag70038 error: internal error: process exited while connecting to monitor: 2017-07-26T01:48:26.352534Z qemu-kvm: -drive file=/guestimages/data1/zs93kag70038.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=native: Could not open backing file: Could not set AIO state: Inappropriate ioctl for device The previous 17 guests started fine: root@zm93k8# virsh start zs93kag70020 Domain zs93kag70020 started root@zm93k8# virsh start zs93kag70021 Domain zs93kag70021 started . . root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70036 Domain zs93kag70036 started We ended up fixing the issue by adding the following line to /etc/sysctl.conf : fs.aio-max-nr = 4194304 ... then, reload the sysctl config file: root@zm93k8:/etc# sysctl -p /etc/sysctl.conf fs.aio-max-nr = 4194304 Now, we're able to start more guests... root@zm93k8:/etc# virsh start zs93kag70036 Domain zs93kag70036 started The default value was originally set to 65535: root@zm93k8:/rawimages/ubu1604qcow2# cat /proc/sys/fs/aio-max-nr 65536 Note, we chose the 4194304 value, because this is what our KVM on System Z hypervisor ships as its default value. Eg. on our zKVM system: [root@zs93ka ~]# cat /proc/sys/fs/aio-max-nr 4194304 ubuntu@zm93k8:/etc$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 16.04.2 LTS Release:16.04 Codename: xenial ubuntu@zm93k8:/etc$ ubuntu@zm93k8:/etc$ dpkg -s qemu-kvm |grep Version Version: 1:2.5+dfsg-5ubuntu10.8 Is something already documented for Ubuntu KVM users warning them about the low default value, and some guidance as to how to select an appropriate value? Also, would you consider increasing the default aio-max-nr value to something much higher, to accommodate significantly more virtual guests? Thanks! ---uname output--- ubuntu@zm93k8:/etc$ uname -a Linux zm93k8 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:12:54 UTC 2017 s390x s390x s390x GNU/Linux Machine Type = z14 ---Debugger--- A debugger is not configured ---Steps to Reproduce--- See Problem Description. The problem was happening a week ago, so this may not reflect that activity. This file was collected on Aug 7, one week after we were hitting the problem. If I need to reproduce the problem and get fresh data, please let me know. /var/log/messages doesn't exist on this system, so I provided syslog output instead. All data have been collected too late after the problem was observed over a week ago. If you need me to reproduce the problem and get new data, please let me know. That's not a problem. Also, we would have to make special arrangements for login access to these systems. I'm happy to run traces and data collection for you as needed. If that's not sufficient, then we'll explore log in access for you. Thanks... - Scott G. I was able to successfully recreate the problem and captured / attached new debug docs. Recreate procedure: # Started out with no virtual guests running. ubuntu@zm93k8:/home/scottg$ virsh list IdName State # Set fs.aio-max-nr back to original Ubuntu "out of the box" value in /etc/sysctl.conf ubuntu@zm93k8:~$ tail -1 /etc/sysctl.conf fs.aio-max-nr = 65536 ## sysctl -a shows: fs.aio-max-nr = 4194304 ## Reload sysctl. ubuntu@zm93k8:~$ sudo sysctl -p /etc/sysctl.conf fs.aio-max-nr = 65536 ubuntu@zm93k8:~$ ubuntu@zm93k8:~$ sudo sysctl -a |grep fs.aio-max-nr fs.aio-max-nr = 65536 ubuntu@zm93k8:~$ cat /proc/sys/fs/aio-max-nr 65536 # Attempt to start more than 17 qcow2 virtual guests on the Ubuntu host. Fails on the 18th XML. Script used to start
[Group.of.nepali.translators] [Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
** Changed in: linux (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value Status in Ubuntu on IBM z Systems: In Progress Status in kvm package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Fix Released Status in procps package in Ubuntu: New Status in linux source package in Xenial: In Progress Status in procps source package in Xenial: New Status in linux source package in Zesty: Won't Fix Status in procps source package in Zesty: Won't Fix Status in linux source package in Artful: In Progress Status in procps source package in Artful: New Bug description: Starting virtual guests via on Ubuntu 16.04.2 LTS installed with its KVM hypervisor on an IBM Z14 system LPAR fails on the 18th guest with the following error: root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70038 error: Failed to start domain zs93kag70038 error: internal error: process exited while connecting to monitor: 2017-07-26T01:48:26.352534Z qemu-kvm: -drive file=/guestimages/data1/zs93kag70038.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=native: Could not open backing file: Could not set AIO state: Inappropriate ioctl for device The previous 17 guests started fine: root@zm93k8# virsh start zs93kag70020 Domain zs93kag70020 started root@zm93k8# virsh start zs93kag70021 Domain zs93kag70021 started . . root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70036 Domain zs93kag70036 started We ended up fixing the issue by adding the following line to /etc/sysctl.conf : fs.aio-max-nr = 4194304 ... then, reload the sysctl config file: root@zm93k8:/etc# sysctl -p /etc/sysctl.conf fs.aio-max-nr = 4194304 Now, we're able to start more guests... root@zm93k8:/etc# virsh start zs93kag70036 Domain zs93kag70036 started The default value was originally set to 65535: root@zm93k8:/rawimages/ubu1604qcow2# cat /proc/sys/fs/aio-max-nr 65536 Note, we chose the 4194304 value, because this is what our KVM on System Z hypervisor ships as its default value. Eg. on our zKVM system: [root@zs93ka ~]# cat /proc/sys/fs/aio-max-nr 4194304 ubuntu@zm93k8:/etc$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 16.04.2 LTS Release:16.04 Codename: xenial ubuntu@zm93k8:/etc$ ubuntu@zm93k8:/etc$ dpkg -s qemu-kvm |grep Version Version: 1:2.5+dfsg-5ubuntu10.8 Is something already documented for Ubuntu KVM users warning them about the low default value, and some guidance as to how to select an appropriate value? Also, would you consider increasing the default aio-max-nr value to something much higher, to accommodate significantly more virtual guests? Thanks! ---uname output--- ubuntu@zm93k8:/etc$ uname -a Linux zm93k8 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:12:54 UTC 2017 s390x s390x s390x GNU/Linux Machine Type = z14 ---Debugger--- A debugger is not configured ---Steps to Reproduce--- See Problem Description. The problem was happening a week ago, so this may not reflect that activity. This file was collected on Aug 7, one week after we were hitting the problem. If I need to reproduce the problem and get fresh data, please let me know. /var/log/messages doesn't exist on this system, so I provided syslog output instead. All data have been collected too late after the problem was observed over a week ago. If you need me to reproduce the problem and get new data, please let me know. That's not a problem. Also, we would have to make special arrangements for login access to these systems. I'm happy to run traces and data collection for you as needed. If that's not sufficient, then we'll explore log in access for you. Thanks... - Scott G. I was able to successfully recreate the problem and captured / attached new debug docs. Recreate procedure: # Started out with no virtual guests running. ubuntu@zm93k8:/home/scottg$ virsh list IdName State # Set fs.aio-max-nr back to original Ubuntu "out of the box" value in /etc/sysctl.conf ubuntu@zm93k8:~$ tail -1 /etc/sysctl.conf fs.aio-max-nr = 65536 ## sysctl -a shows: fs.aio-max-nr = 4194304 ## Reload sysctl. ubuntu@zm93k8:~$ sudo sysctl -p /etc/sysctl.conf fs.aio-max-nr = 65536 ubuntu@zm93k8:~$ ubuntu@zm93k8:~$ sudo sysctl -a |grep fs.aio-max-nr fs.aio-max-nr = 65536 ubuntu@zm93k8:~$ cat /proc/sys/fs/aio-max-nr 65536 # Attempt to start more than 17
[Group.of.nepali.translators] [Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Zesty is end of life. ** Changed in: procps (Ubuntu Zesty) Status: New => Won't Fix -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value Status in Ubuntu on IBM z Systems: In Progress Status in kvm package in Ubuntu: Won't Fix Status in linux package in Ubuntu: In Progress Status in procps package in Ubuntu: New Status in linux source package in Xenial: In Progress Status in procps source package in Xenial: New Status in linux source package in Zesty: Won't Fix Status in procps source package in Zesty: Won't Fix Status in linux source package in Artful: In Progress Status in procps source package in Artful: New Bug description: Starting virtual guests via on Ubuntu 16.04.2 LTS installed with its KVM hypervisor on an IBM Z14 system LPAR fails on the 18th guest with the following error: root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70038 error: Failed to start domain zs93kag70038 error: internal error: process exited while connecting to monitor: 2017-07-26T01:48:26.352534Z qemu-kvm: -drive file=/guestimages/data1/zs93kag70038.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=native: Could not open backing file: Could not set AIO state: Inappropriate ioctl for device The previous 17 guests started fine: root@zm93k8# virsh start zs93kag70020 Domain zs93kag70020 started root@zm93k8# virsh start zs93kag70021 Domain zs93kag70021 started . . root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70036 Domain zs93kag70036 started We ended up fixing the issue by adding the following line to /etc/sysctl.conf : fs.aio-max-nr = 4194304 ... then, reload the sysctl config file: root@zm93k8:/etc# sysctl -p /etc/sysctl.conf fs.aio-max-nr = 4194304 Now, we're able to start more guests... root@zm93k8:/etc# virsh start zs93kag70036 Domain zs93kag70036 started The default value was originally set to 65535: root@zm93k8:/rawimages/ubu1604qcow2# cat /proc/sys/fs/aio-max-nr 65536 Note, we chose the 4194304 value, because this is what our KVM on System Z hypervisor ships as its default value. Eg. on our zKVM system: [root@zs93ka ~]# cat /proc/sys/fs/aio-max-nr 4194304 ubuntu@zm93k8:/etc$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 16.04.2 LTS Release:16.04 Codename: xenial ubuntu@zm93k8:/etc$ ubuntu@zm93k8:/etc$ dpkg -s qemu-kvm |grep Version Version: 1:2.5+dfsg-5ubuntu10.8 Is something already documented for Ubuntu KVM users warning them about the low default value, and some guidance as to how to select an appropriate value? Also, would you consider increasing the default aio-max-nr value to something much higher, to accommodate significantly more virtual guests? Thanks! ---uname output--- ubuntu@zm93k8:/etc$ uname -a Linux zm93k8 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:12:54 UTC 2017 s390x s390x s390x GNU/Linux Machine Type = z14 ---Debugger--- A debugger is not configured ---Steps to Reproduce--- See Problem Description. The problem was happening a week ago, so this may not reflect that activity. This file was collected on Aug 7, one week after we were hitting the problem. If I need to reproduce the problem and get fresh data, please let me know. /var/log/messages doesn't exist on this system, so I provided syslog output instead. All data have been collected too late after the problem was observed over a week ago. If you need me to reproduce the problem and get new data, please let me know. That's not a problem. Also, we would have to make special arrangements for login access to these systems. I'm happy to run traces and data collection for you as needed. If that's not sufficient, then we'll explore log in access for you. Thanks... - Scott G. I was able to successfully recreate the problem and captured / attached new debug docs. Recreate procedure: # Started out with no virtual guests running. ubuntu@zm93k8:/home/scottg$ virsh list IdName State # Set fs.aio-max-nr back to original Ubuntu "out of the box" value in /etc/sysctl.conf ubuntu@zm93k8:~$ tail -1 /etc/sysctl.conf fs.aio-max-nr = 65536 ## sysctl -a shows: fs.aio-max-nr = 4194304 ## Reload sysctl. ubuntu@zm93k8:~$ sudo sysctl -p /etc/sysctl.conf fs.aio-max-nr = 65536 ubuntu@zm93k8:~$ ubuntu@zm93k8:~$ sudo sysctl -a |grep fs.aio-max-nr fs.aio-max-nr = 65536 ubuntu@zm93k8:~$ cat /proc/sys/fs/aio-max-nr 65536 # Attempt to start
[Group.of.nepali.translators] [Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
zesty is End Of Life ** Changed in: linux (Ubuntu Zesty) Status: In Progress => Won't Fix -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value Status in Ubuntu on IBM z Systems: In Progress Status in kvm package in Ubuntu: Won't Fix Status in linux package in Ubuntu: In Progress Status in procps package in Ubuntu: New Status in linux source package in Xenial: In Progress Status in procps source package in Xenial: New Status in linux source package in Zesty: Won't Fix Status in procps source package in Zesty: Won't Fix Status in linux source package in Artful: In Progress Status in procps source package in Artful: New Bug description: Starting virtual guests via on Ubuntu 16.04.2 LTS installed with its KVM hypervisor on an IBM Z14 system LPAR fails on the 18th guest with the following error: root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70038 error: Failed to start domain zs93kag70038 error: internal error: process exited while connecting to monitor: 2017-07-26T01:48:26.352534Z qemu-kvm: -drive file=/guestimages/data1/zs93kag70038.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=native: Could not open backing file: Could not set AIO state: Inappropriate ioctl for device The previous 17 guests started fine: root@zm93k8# virsh start zs93kag70020 Domain zs93kag70020 started root@zm93k8# virsh start zs93kag70021 Domain zs93kag70021 started . . root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70036 Domain zs93kag70036 started We ended up fixing the issue by adding the following line to /etc/sysctl.conf : fs.aio-max-nr = 4194304 ... then, reload the sysctl config file: root@zm93k8:/etc# sysctl -p /etc/sysctl.conf fs.aio-max-nr = 4194304 Now, we're able to start more guests... root@zm93k8:/etc# virsh start zs93kag70036 Domain zs93kag70036 started The default value was originally set to 65535: root@zm93k8:/rawimages/ubu1604qcow2# cat /proc/sys/fs/aio-max-nr 65536 Note, we chose the 4194304 value, because this is what our KVM on System Z hypervisor ships as its default value. Eg. on our zKVM system: [root@zs93ka ~]# cat /proc/sys/fs/aio-max-nr 4194304 ubuntu@zm93k8:/etc$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 16.04.2 LTS Release:16.04 Codename: xenial ubuntu@zm93k8:/etc$ ubuntu@zm93k8:/etc$ dpkg -s qemu-kvm |grep Version Version: 1:2.5+dfsg-5ubuntu10.8 Is something already documented for Ubuntu KVM users warning them about the low default value, and some guidance as to how to select an appropriate value? Also, would you consider increasing the default aio-max-nr value to something much higher, to accommodate significantly more virtual guests? Thanks! ---uname output--- ubuntu@zm93k8:/etc$ uname -a Linux zm93k8 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:12:54 UTC 2017 s390x s390x s390x GNU/Linux Machine Type = z14 ---Debugger--- A debugger is not configured ---Steps to Reproduce--- See Problem Description. The problem was happening a week ago, so this may not reflect that activity. This file was collected on Aug 7, one week after we were hitting the problem. If I need to reproduce the problem and get fresh data, please let me know. /var/log/messages doesn't exist on this system, so I provided syslog output instead. All data have been collected too late after the problem was observed over a week ago. If you need me to reproduce the problem and get new data, please let me know. That's not a problem. Also, we would have to make special arrangements for login access to these systems. I'm happy to run traces and data collection for you as needed. If that's not sufficient, then we'll explore log in access for you. Thanks... - Scott G. I was able to successfully recreate the problem and captured / attached new debug docs. Recreate procedure: # Started out with no virtual guests running. ubuntu@zm93k8:/home/scottg$ virsh list IdName State # Set fs.aio-max-nr back to original Ubuntu "out of the box" value in /etc/sysctl.conf ubuntu@zm93k8:~$ tail -1 /etc/sysctl.conf fs.aio-max-nr = 65536 ## sysctl -a shows: fs.aio-max-nr = 4194304 ## Reload sysctl. ubuntu@zm93k8:~$ sudo sysctl -p /etc/sysctl.conf fs.aio-max-nr = 65536 ubuntu@zm93k8:~$ ubuntu@zm93k8:~$ sudo sysctl -a |grep fs.aio-max-nr fs.aio-max-nr = 65536 ubuntu@zm93k8:~$ cat /proc/sys/fs/aio-max-nr 65536 # Attempt to
[Group.of.nepali.translators] [Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Hi Heinz-Werner, thanks for the ping. I think we agreed back then that the config itself is a Ubuntu wide thing - but I haven't seen any discussions on procps or in general to lift it. But then as I outlined in comment #7 any limit can be too low. I'll set the KVM task to won't fix to clearly mark it as "not a kvm task". The fix that you need thou is the kernel change to account correctly - so that part of the question goes to the Kernel Team if the change that was proposed back then was actually released. @Joseph - we clearly passed the number, but was the change released with it? ** Changed in: kvm (Ubuntu) Status: Confirmed => Won't Fix ** No longer affects: kvm (Ubuntu Xenial) ** No longer affects: kvm (Ubuntu Zesty) ** No longer affects: kvm (Ubuntu Artful) -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value Status in Ubuntu on IBM z Systems: In Progress Status in kvm package in Ubuntu: Won't Fix Status in linux package in Ubuntu: In Progress Status in procps package in Ubuntu: New Status in linux source package in Xenial: In Progress Status in procps source package in Xenial: New Status in linux source package in Zesty: In Progress Status in procps source package in Zesty: New Status in linux source package in Artful: In Progress Status in procps source package in Artful: New Bug description: Starting virtual guests via on Ubuntu 16.04.2 LTS installed with its KVM hypervisor on an IBM Z14 system LPAR fails on the 18th guest with the following error: root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70038 error: Failed to start domain zs93kag70038 error: internal error: process exited while connecting to monitor: 2017-07-26T01:48:26.352534Z qemu-kvm: -drive file=/guestimages/data1/zs93kag70038.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=native: Could not open backing file: Could not set AIO state: Inappropriate ioctl for device The previous 17 guests started fine: root@zm93k8# virsh start zs93kag70020 Domain zs93kag70020 started root@zm93k8# virsh start zs93kag70021 Domain zs93kag70021 started . . root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70036 Domain zs93kag70036 started We ended up fixing the issue by adding the following line to /etc/sysctl.conf : fs.aio-max-nr = 4194304 ... then, reload the sysctl config file: root@zm93k8:/etc# sysctl -p /etc/sysctl.conf fs.aio-max-nr = 4194304 Now, we're able to start more guests... root@zm93k8:/etc# virsh start zs93kag70036 Domain zs93kag70036 started The default value was originally set to 65535: root@zm93k8:/rawimages/ubu1604qcow2# cat /proc/sys/fs/aio-max-nr 65536 Note, we chose the 4194304 value, because this is what our KVM on System Z hypervisor ships as its default value. Eg. on our zKVM system: [root@zs93ka ~]# cat /proc/sys/fs/aio-max-nr 4194304 ubuntu@zm93k8:/etc$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 16.04.2 LTS Release:16.04 Codename: xenial ubuntu@zm93k8:/etc$ ubuntu@zm93k8:/etc$ dpkg -s qemu-kvm |grep Version Version: 1:2.5+dfsg-5ubuntu10.8 Is something already documented for Ubuntu KVM users warning them about the low default value, and some guidance as to how to select an appropriate value? Also, would you consider increasing the default aio-max-nr value to something much higher, to accommodate significantly more virtual guests? Thanks! ---uname output--- ubuntu@zm93k8:/etc$ uname -a Linux zm93k8 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:12:54 UTC 2017 s390x s390x s390x GNU/Linux Machine Type = z14 ---Debugger--- A debugger is not configured ---Steps to Reproduce--- See Problem Description. The problem was happening a week ago, so this may not reflect that activity. This file was collected on Aug 7, one week after we were hitting the problem. If I need to reproduce the problem and get fresh data, please let me know. /var/log/messages doesn't exist on this system, so I provided syslog output instead. All data have been collected too late after the problem was observed over a week ago. If you need me to reproduce the problem and get new data, please let me know. That's not a problem. Also, we would have to make special arrangements for login access to these systems. I'm happy to run traces and data collection for you as needed. If that's not sufficient, then we'll explore log in access for you. Thanks... - Scott G. I was able to successfully recreate the problem and captured / attached new debug docs. Recreate
[Group.of.nepali.translators] [Bug 1717224] Re: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value
Actually, Artful already has commit 2a8a98673c13. It was pulled in for bug 1718397. It looks like Xenial and Zesty will also get that commit via that bug. Can you test Artful and/or X/Y -proposed to see if this bug is resolved with that commit? ** Also affects: procps (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: kvm (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: procps (Ubuntu Zesty) Importance: Undecided Status: New ** Also affects: kvm (Ubuntu Zesty) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Zesty) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Zesty) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Zesty) Status: New => In Progress ** Changed in: linux (Ubuntu Zesty) Assignee: (unassigned) => Joseph Salisbury (jsalisbury) ** Changed in: linux (Ubuntu Xenial) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Xenial) Status: New => In Progress ** Changed in: linux (Ubuntu Xenial) Assignee: (unassigned) => Joseph Salisbury (jsalisbury) -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1717224 Title: virsh start of virtual guest domain fails with internal error due to low default aio-max-nr sysctl value Status in Ubuntu on IBM z Systems: Confirmed Status in kvm package in Ubuntu: Confirmed Status in linux package in Ubuntu: In Progress Status in procps package in Ubuntu: New Status in kvm source package in Xenial: New Status in linux source package in Xenial: In Progress Status in procps source package in Xenial: New Status in kvm source package in Zesty: New Status in linux source package in Zesty: In Progress Status in procps source package in Zesty: New Status in kvm source package in Artful: Confirmed Status in linux source package in Artful: In Progress Status in procps source package in Artful: New Bug description: Starting virtual guests via on Ubuntu 16.04.2 LTS installed with its KVM hypervisor on an IBM Z14 system LPAR fails on the 18th guest with the following error: root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70038 error: Failed to start domain zs93kag70038 error: internal error: process exited while connecting to monitor: 2017-07-26T01:48:26.352534Z qemu-kvm: -drive file=/guestimages/data1/zs93kag70038.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=native: Could not open backing file: Could not set AIO state: Inappropriate ioctl for device The previous 17 guests started fine: root@zm93k8# virsh start zs93kag70020 Domain zs93kag70020 started root@zm93k8# virsh start zs93kag70021 Domain zs93kag70021 started . . root@zm93k8:/rawimages/ubu1604qcow2# virsh start zs93kag70036 Domain zs93kag70036 started We ended up fixing the issue by adding the following line to /etc/sysctl.conf : fs.aio-max-nr = 4194304 ... then, reload the sysctl config file: root@zm93k8:/etc# sysctl -p /etc/sysctl.conf fs.aio-max-nr = 4194304 Now, we're able to start more guests... root@zm93k8:/etc# virsh start zs93kag70036 Domain zs93kag70036 started The default value was originally set to 65535: root@zm93k8:/rawimages/ubu1604qcow2# cat /proc/sys/fs/aio-max-nr 65536 Note, we chose the 4194304 value, because this is what our KVM on System Z hypervisor ships as its default value. Eg. on our zKVM system: [root@zs93ka ~]# cat /proc/sys/fs/aio-max-nr 4194304 ubuntu@zm93k8:/etc$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 16.04.2 LTS Release:16.04 Codename: xenial ubuntu@zm93k8:/etc$ ubuntu@zm93k8:/etc$ dpkg -s qemu-kvm |grep Version Version: 1:2.5+dfsg-5ubuntu10.8 Is something already documented for Ubuntu KVM users warning them about the low default value, and some guidance as to how to select an appropriate value? Also, would you consider increasing the default aio-max-nr value to something much higher, to accommodate significantly more virtual guests? Thanks! ---uname output--- ubuntu@zm93k8:/etc$ uname -a Linux zm93k8 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:12:54 UTC 2017 s390x s390x s390x GNU/Linux Machine Type = z14 ---Debugger--- A debugger is not configured ---Steps to Reproduce--- See Problem Description. The problem was happening a week ago, so this may not reflect that activity. This file was collected on Aug 7, one week after we were hitting the problem. If I need to reproduce the problem and get fresh data, please let me know. /var/log/messages doesn't exist on this system, so I provided
