Public bug reported:
Description
===========
Hi guys, i ran into a problem in our OpenStack Ocata/Rocky clusters:
When i was trying to use `set-password` subcommand of nova CLI to reset
root password for my VM, it failed and my VM became error.
I searched launchpad for similar issues, but got nothing. I believe the
problem may also exist in latest OpenStack distro.
Steps to reproduce
==================
* Upload any image(without QGA inside), e.g: cirros
* Update the image with property: hw_qemu_guest_agent=yes
$ glance image-update --property hw_qemu_guest_agent=yes <UUID of cirros>
* Boot new instance (e.g: QGA) with image cirros and ensure instance is
active/running.
* Use cli `nova set-password <UUID of QGA>` to reset password for the instance.
Expected result
===============
Error Messages like 'QGA not running' occur.
Instance becomes active/ruuning again from task_state `updating_password`.
Actual result
=============
CLI returns with: Failed to set admin password on XXXXXX became error setting
admin password
(HTTP 409)(Request-ID: req-XXXXX)
And instance became error.
Environment
===========
1. version: OpenStack Ocata/Rocky + centOS7
2. hypervisor: Libvirt + KVM
3. storage: Ceph
4. networking Neutron with OpenVSwitch
Logs & Configs
==============
#### Nova CLI error #####
[root@node159 ~]# nova set-password f355e4d0-574c-4792-bbbd-04ad03ce6066
New password:
Again:
ERROR (Conflict): Failed to set admin password on
f355e4d0-574c-4792-bbbd-04ad03ce6066 because error setting
admin password (HTTP 409) (Request-ID: req-34715791-f42a-4235-98d5-f69680440fc8)
##### Grep nova-compute errors by Instance UUID #############
23698 2019-05-21 14:53:50.355 7 INFO nova.compute.manager
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Enter manager build_and_run_instance
23699 2019-05-21 14:53:50.521 7 INFO nova.compute.manager
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Enter manager _build_and_run_instance
23700 2019-05-21 14:53:50.546 7 INFO nova.compute.claims
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Attempting claim: memory 2048 MB, disk 1
GB, vcpus 2 CPU
23701 2019-05-21 14:53:50.547 7 INFO nova.compute.claims
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Total memory: 65417 MB, used: 37568.00 MB
23702 2019-05-21 14:53:50.548 7 INFO nova.compute.claims
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] memory limit: 52333.60 MB, free: 14765.60
MB
23703 2019-05-21 14:53:50.548 7 INFO nova.compute.claims
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Total disk: 3719 GB, used: 285.00 GB
23704 2019-05-21 14:53:50.549 7 INFO nova.compute.claims
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] disk limit: 3719.00 GB, free: 3434.00 GB
23705 2019-05-21 14:53:50.550 7 INFO nova.compute.claims
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Total vcpu: 16 VCPU, used: 41.00 VCPU
23706 2019-05-21 14:53:50.550 7 INFO nova.compute.claims
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] vcpu limit not specified, defaulting to
unlimited
23707 2019-05-21 14:53:50.552 7 INFO nova.compute.claims
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Claim successful
23708 2019-05-21 14:53:50.762 7 INFO nova.scheduler.client.report
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Submitted allocation for instance
23709 2019-05-21 14:53:50.913 7 INFO nova.compute.manager
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Enter manager _build_resources
23713 2019-05-21 14:53:51.430 7 INFO nova.virt.libvirt.driver
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Creating image
23727 2019-05-21 14:54:01.087 7 INFO nova.compute.manager
[req-a476ad61-30e0-4b25-98b2-e25a5028f4e2 - - - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] VM Started (Lifecycle Event)
23728 2019-05-21 14:54:01.137 7 INFO nova.compute.manager
[req-a476ad61-30e0-4b25-98b2-e25a5028f4e2 - - - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] VM Paused (Lifecycle Event)
23729 2019-05-21 14:54:01.232 7 INFO nova.compute.manager
[req-a476ad61-30e0-4b25-98b2-e25a5028f4e2 - - - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] During sync_power_state the instance has
a pending task (spawning). Skip.
23730 2019-05-21 14:54:03.872 7 INFO nova.compute.manager
[req-a476ad61-30e0-4b25-98b2-e25a5028f4e2 - - - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] VM Resumed (Lifecycle Event)
23731 2019-05-21 14:54:03.877 7 INFO nova.virt.driver [-] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Instance spawned successfully.
23732 2019-05-21 14:54:03.878 7 INFO nova.compute.manager
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Took 12.45 seconds to spawn the instance
on the hypervisor.
23733 2019-05-21 14:54:03.971 7 INFO nova.compute.manager
[req-a476ad61-30e0-4b25-98b2-e25a5028f4e2 - - - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] During sync_power_state the instance has
a pending task (spawning). Skip.
23734 2019-05-21 14:54:03.972 7 INFO nova.compute.manager
[req-a476ad61-30e0-4b25-98b2-e25a5028f4e2 - - - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] VM Resumed (Lifecycle Event)
23735 2019-05-21 14:54:04.029 7 INFO nova.compute.manager
[req-6ec684e7-ee6e-47a4-8f75-53cccc235d86 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Took 13.54 seconds to build instance.
23736 2019-05-21 14:54:49.985 7 ERROR nova.compute.manager
[req-34715791-f42a-4235-98d5-f69680440fc8 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] set_admin_password failed
23737 2019-05-21 14:54:49.985 7 ERROR nova.compute.manager [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Traceback (most recent call last):
23738 2019-05-21 14:54:49.985 7 ERROR nova.compute.manager [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] File
"/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line
3266, in set_admin_password
23739 2019-05-21 14:54:49.985 7 ERROR nova.compute.manager [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066]
self.driver.set_admin_password(instance, new_pass)
23740 2019-05-21 14:54:49.985 7 ERROR nova.compute.manager [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] File
"/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
line 1719, in set_admin_password
23741 2019-05-21 14:54:49.985 7 ERROR nova.compute.manager [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] raise exception.InternalError(msg)
23742 2019-05-21 14:54:49.985 7 ERROR nova.compute.manager [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] InternalError: Error from libvirt while
set password for username "root": [Error Code 86] Guest agent is not
responding: QEMU guest agent is not connected
23743 2019-05-21 14:54:49.985 7 ERROR nova.compute.manager [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066]
23744 2019-05-21 14:54:50.493 7 INFO nova.compute.manager
[req-34715791-f42a-4235-98d5-f69680440fc8 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Successfully reverted task state from
updating_password on failure for instance.
23781 2019-05-21 14:54:50.498 7 ERROR oslo_messaging.rpc.server
InstancePasswordSetFailed: Failed to set admin password on
f355e4d0-574c-4792-bbbd-04ad03ce6066 because error setting admin password
###### Grep nova-compute errors by requrest UUID ####
23736 2019-05-21 14:54:49.985 7 ERROR nova.compute.manager
[req-34715791-f42a-4235-98d5-f69680440fc8 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] set_admin_password failed
23744 2019-05-21 14:54:50.493 7 INFO nova.compute.manager
[req-34715791-f42a-4235-98d5-f69680440fc8 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] [instance:
f355e4d0-574c-4792-bbbd-04ad03ce6066] Successfully reverted task state from
updating_password on failure for instance.
23745 2019-05-21 14:54:50.498 7 ERROR oslo_messaging.rpc.server
[req-34715791-f42a-4235-98d5-f69680440fc8 9fef2099c3254226a96e48311d124131
380f701f5575430195526229dc143a1f - - -] Exception during message handling
** Affects: ubuntu
Importance: Undecided
Assignee: pandatt (pandatt)
Status: Confirmed
** Tags: libvirt nova-compute qga
** Changed in: nova
Status: New => Confirmed
** Changed in: nova
Assignee: (unassigned) => pandatt (pandatt)
** Tags added: libvirt
** Tags added: nova-compute
** Tags added: qga
** Project changed: nova => ubuntu
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1829828
Title:
instance failed to revert after a set-password failure
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/1829828/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
