For the Ubuntu SRU, the stable/queens upstream backport is failing
tests. Additionally the fix for
https://bugs.launchpad.net/nova/+bug/1810977 would need backporting to
stable/queens. These would both need to land upstream in stable/queens
first and we would need capability to test both fixes and ensure no
regressions. For now, marking as won't fix. Suggestion is to upgrade to
rocky+ where this is already fixed.
** Changed in: cloud-archive/queens
Status: In Progress => Won't Fix
** Changed in: nova (Ubuntu Bionic)
Status: Triaged => Won't Fix
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1734204
Title:
Insufficient free host memory pages available to allocate guest RAM
with Open vSwitch DPDK in Newton
Status in Ubuntu Cloud Archive:
Invalid
Status in Ubuntu Cloud Archive queens series:
Won't Fix
Status in OpenStack Compute (nova):
Fix Released
Status in nova package in Ubuntu:
Invalid
Status in nova source package in Bionic:
Won't Fix
Bug description:
When spawning an instance and scheduling it onto a compute node which still
has sufficient pCPUs for the instance and also sufficient free huge pages for
the instance memory, nova returns:
Raw
[stack@undercloud-4 ~]$ nova show 1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc
(...)
| fault | {"message": "Exceeded maximum number
of retries. Exceeded max scheduling attempts 3 for instance
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc. Last exception: internal error: process
exited while connecting to monitor: 2017-11-23T19:53:20.311446Z qemu-kvm:
-chardev pty,id=cha", "code": 500, "details": " File
\"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 492, in
build_instances |
| | filter_properties,
instances[0].uuid)
|
| | File
\"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 184, in
populate_retry
|
| | raise
exception.MaxRetriesExceeded(reason=msg)
|
| | ", "created":
"2017-11-23T19:53:22Z"}
(...)
And /var/log/nova/nova-compute.log on the compute node gives the following
ERROR message:
Raw
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager
[req-2ad59cdf-4901-4df1-8bd7-ebaea20b9361 5d1785ee87294a6fad5e2bdddd91cc20
8c307c08d2234b339c504bfdd896c13e - - -] [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] Instance failed
to spawn
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] Traceback (most recent call last):
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2087, in
_build_resources
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] yield resources
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1928, in
_build_and_run_instance
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] block_device_info=block_device_info)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2674, in
spawn
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] destroy_disks_on_failure=True)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5013, in
_create_domain_and_network
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] destroy_disks_on_failure)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] self.force_reraise()
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] six.reraise(self.type_, self.value,
self.tb)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4985, in
_create_domain_and_network
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] post_xml_callback=post_xml_callback)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4903, in
_create_domain
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] guest.launch(pause=pause)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in
launch
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] self._encoded_xml, errors='ignore')
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] self.force_reraise()
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] six.reraise(self.type_, self.value,
self.tb)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in
launch
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] return
self._domain.createWithFlags(flags)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] result = proxy_call(self._autowrap,
f, *args, **kwargs)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] rv = execute(f, *args, **kwargs)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] six.reraise(c, e, tb)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] rv = meth(*args, **kwargs)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in createWithFlags
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] if ret == -1: raise libvirtError
('virDomainCreateWithFlags() failed', dom=self)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] libvirtError: internal error: process
exited while connecting to monitor: 2017-11-23T19:53:20.311446Z qemu-kvm:
-chardev pty,id=charserial1: char device redirected to /dev/pts/3 (label
charserial1)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] 2017-11-23T19:53:20.477183Z qemu-kvm:
-object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/7-instance-00000006,share=yes,size=536870912,host-nodes=0,policy=bind:
os_mem_prealloc: Insufficient free host memory pages available to allocate
guest RAM
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc]
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc]
Additionally, libvirt creates the following log file:
Raw
[root@overcloud-compute-1 qemu]# cat instance-00000006.log
2017-11-23 19:53:02.145+0000: starting up libvirt version: 3.2.0, package:
14.el7_4.3 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>,
2017-08-22-08:54:01, x86-039.build.eng.bos.redhat.com), qemu version:
2.9.0(qemu-kvm-rhev-2.9.0-10.el7), hostname: overcloud-compute-1.localdomain
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name
guest=instance-00000006,debug-threads=on -S -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-instance-00000006/master-key.aes
-machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu
SandyBridge,vme=on,hypervisor=on,arat=on,tsc_adjust=on,xsaveopt=on -m 512
-realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/5-instance-00000006,share=yes,size=536870912,host-nodes=0,policy=bind
-numa node,nodeid=0,cpus=0,memdev=ram-node0 -uuid
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc -smbios 'type=1,manufacturer=Red
Hat,product=OpenStack
Compute,version=14.0.8-5.el7ost,serial=4f88fcca-0cd3-4e19-8dc4-4436a54daff8,uuid=1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc,family=Virtual
Machine' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-5-instance-00000006/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
-global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/var/lib/nova/instances/1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-chardev socket,id=charnet0,path=/var/run/openvswitch/vhu9758ef15-d2 -netdev
vhost-user,chardev=charnet0,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:d6:89:65,bus=pci.0,addr=0x3
-add-fd set=0,fd=29 -chardev file,id=charserial0,path=/dev/fdset/0,append=on
-device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1
-device isa-serial,chardev=charserial1,id=serial1 -device
usb-tablet,id=input0,bus=usb.0,port=1 -vnc 172.16.2.8:2 -k en-us -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
2017-11-23T19:53:03.217386Z qemu-kvm: -chardev pty,id=charserial1: char
device redirected to /dev/pts/3 (label charserial1)
2017-11-23T19:53:03.359799Z qemu-kvm: -object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/5-instance-00000006,share=yes,size=536870912,host-nodes=0,policy=bind:
os_mem_prealloc: Insufficient free host memory pages available to allocate
guest RAM
2017-11-23 19:53:03.630+0000: shutting down, reason=failed
2017-11-23 19:53:10.052+0000: starting up libvirt version: 3.2.0, package:
14.el7_4.3 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>,
2017-08-22-08:54:01, x86-039.build.eng.bos.redhat.com), qemu version:
2.9.0(qemu-kvm-rhev-2.9.0-10.el7), hostname: overcloud-compute-1.localdomain
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name
guest=instance-00000006,debug-threads=on -S -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-instance-00000006/master-key.aes
-machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu
SandyBridge,vme=on,hypervisor=on,arat=on,tsc_adjust=on,xsaveopt=on -m 512
-realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/6-instance-00000006,share=yes,size=536870912,host-nodes=0,policy=bind
-numa node,nodeid=0,cpus=0,memdev=ram-node0 -uuid
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc -smbios 'type=1,manufacturer=Red
Hat,product=OpenStack
Compute,version=14.0.8-5.el7ost,serial=4f88fcca-0cd3-4e19-8dc4-4436a54daff8,uuid=1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc,family=Virtual
Machine' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-instance-00000006/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
-global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/var/lib/nova/instances/1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-chardev socket,id=charnet0,path=/var/run/openvswitch/vhu9758ef15-d2 -netdev
vhost-user,chardev=charnet0,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:d6:89:65,bus=pci.0,addr=0x3
-add-fd set=0,fd=29 -chardev file,id=charserial0,path=/dev/fdset/0,append=on
-device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1
-device isa-serial,chardev=charserial1,id=serial1 -device
usb-tablet,id=input0,bus=usb.0,port=1 -vnc 172.16.2.8:2 -k en-us -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
2017-11-23T19:53:11.466399Z qemu-kvm: -chardev pty,id=charserial1: char
device redirected to /dev/pts/3 (label charserial1)
2017-11-23T19:53:11.729226Z qemu-kvm: -object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/6-instance-00000006,share=yes,size=536870912,host-nodes=0,policy=bind:
os_mem_prealloc: Insufficient free host memory pages available to allocate
guest RAM
2017-11-23 19:53:12.159+0000: shutting down, reason=failed
2017-11-23 19:53:19.370+0000: starting up libvirt version: 3.2.0, package:
14.el7_4.3 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>,
2017-08-22-08:54:01, x86-039.build.eng.bos.redhat.com), qemu version:
2.9.0(qemu-kvm-rhev-2.9.0-10.el7), hostname: overcloud-compute-1.localdomain
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name
guest=instance-00000006,debug-threads=on -S -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-7-instance-00000006/master-key.aes
-machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu
SandyBridge,vme=on,hypervisor=on,arat=on,tsc_adjust=on,xsaveopt=on -m 512
-realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/7-instance-00000006,share=yes,size=536870912,host-nodes=0,policy=bind
-numa node,nodeid=0,cpus=0,memdev=ram-node0 -uuid
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc -smbios 'type=1,manufacturer=Red
Hat,product=OpenStack
Compute,version=14.0.8-5.el7ost,serial=4f88fcca-0cd3-4e19-8dc4-4436a54daff8,uuid=1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc,family=Virtual
Machine' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-7-instance-00000006/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
-global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/var/lib/nova/instances/1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-chardev socket,id=charnet0,path=/var/run/openvswitch/vhu9758ef15-d2 -netdev
vhost-user,chardev=charnet0,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:d6:89:65,bus=pci.0,addr=0x3
-add-fd set=0,fd=29 -chardev file,id=charserial0,path=/dev/fdset/0,append=on
-device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1
-device isa-serial,chardev=charserial1,id=serial1 -device
usb-tablet,id=input0,bus=usb.0,port=1 -vnc 172.16.2.8:2 -k en-us -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
2017-11-23T19:53:20.311446Z qemu-kvm: -chardev pty,id=charserial1: char
device redirected to /dev/pts/3 (label charserial1)
2017-11-23T19:53:20.477183Z qemu-kvm: -object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/7-instance-00000006,share=yes,size=536870912,host-nodes=0,policy=bind:
os_mem_prealloc: Insufficient free host memory pages available to allocate
guest RAM
2017-11-23 19:53:20.724+0000: shutting down, reason=failed
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1734204/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
