Looks like you have a misconfiguration problem or an issue with your
message queue. Could you please verify your nova.conf parameters for
accessing your Rabbit instance ?
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1716234
Title:
Live migrate fails on Ocata
Status in OpenStack Compute (nova):
Invalid
Bug description:
Instructed to report this bug by Nova.
Instance was running on Host compute3O
root@controllerO:~# openstack server show eb335c1a-836e-4e65-b900-d7fbd1b78227
+-------------------------------------+----------------------------------------------------------+
| Field | Value
|
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO
|
| OS-EXT-AZ:availability_zone | nova
|
| OS-EXT-SRV-ATTR:host | compute3O
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute3O
|
| OS-EXT-SRV-ATTR:instance_name | instance-0000002a
|
| OS-EXT-STS:power_state | Running
|
| OS-EXT-STS:task_state | None
|
| OS-EXT-STS:vm_state | active
|
| OS-SRV-USG:launched_at | 2017-09-10T11:55:41.000000
|
| OS-SRV-USG:terminated_at | None
|
| accessIPv4 |
|
| accessIPv6 |
|
| addresses | selfservice=172.16.1.6,
192.168.179.104 |
| config_drive |
|
| created | 2017-09-10T11:54:17Z
|
| flavor | C1R1D10 (1)
|
| hostId |
c1f359b371ca8c010ed1f0d992903906455ec5668656fb2b588b7db4 |
| id | eb335c1a-836e-4e65-b900-d7fbd1b78227
|
| image |
|
| key_name | marcus-key
|
| name | d1
|
| progress | 0
|
| project_id | 83a0f55c38b14931bf1fed6758af8691
|
| properties |
|
| security_groups | name='default'
|
| status | ACTIVE
|
| updated | 2017-09-10T14:36:41Z
|
| user_id | bf2bd08f16a34c629c7d211b71b4f0b8
|
| volumes_attached |
id='3771d614-11f1-4757-8b00-1181995d3fde' |
+-------------------------------------+----------------------------------------------------------+
root@controllerO:~# openstack hypervisor show compute3O
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value
|
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| aggregates | []
|
| cpu_info | {"vendor": "AMD", "model": "Opteron_G3", "arch":
"x86_64", "features": ["pge", "clflush", "sep", "syscall", "vme", "msr",
"nodeid_msr", "cmov", "fpu", "pat", |
| | "monitor", "lm", "tsc", "3dnowprefetch", "nx",
"fxsr", "pae", "wdt", "cmp_legacy", "skinit", "sse4a", "mmx", "cx8", "mce",
"de", "cr8legacy", "ht", "pse", |
| | "3dnowext", "abm", "fxsr_opt", "popcnt", "mca",
"pdpe1gb", "apic", "sse", "3dnow", "invtsc", "pni", "rdtscp", "svm", "mmxext",
"osvw", "sse2", "lahf_lm", |
| | "misalignsse", "cx16", "extapic", "pse36", "mtrr",
"ibs"], "topology": {"cores": 6, "cells": 8, "threads": 1, "sockets": 1}}
|
| current_workload | 0
|
| disk_available_least | 186
|
| free_disk_gb | 186
|
| free_ram_mb | 256412
|
| host_ip | 192.168.178.237
|
| host_time | 16:46:00
|
| hypervisor_hostname | compute3O
|
| hypervisor_type | QEMU
|
| hypervisor_version | 2008000
|
| id | 2
|
| load_average | 0.11, 0.03, 0.01
|
| local_gb | 196
|
| local_gb_used | 0
|
| memory_mb | 257948
|
| memory_mb_used | 1312
|
| running_vms | 1
|
| service_host | compute3O
|
| service_id | 8
|
| state | up
|
| status | enabled
|
| uptime | 2:56
|
| users | 0
|
| vcpus | 48
|
| vcpus_used | 1
|
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
and should be migrated to compute1O:
root@controllerO:~# openstack hypervisor show compute1O
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value
|
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| aggregates | []
|
| cpu_info | {"vendor": "AMD", "model": "Opteron_G3", "arch":
"x86_64", "features": ["pge", "clflush", "sep", "syscall", "vme", "msr",
"cmov", "fpu", "pat", "monitor", "lm", |
| | "tsc", "3dnowprefetch", "nx", "fxsr", "pae", "wdt",
"cmp_legacy", "skinit", "sse4a", "mmx", "cx8", "mce", "de", "cr8legacy", "ht",
"pse", "3dnowext", "abm", |
| | "fxsr_opt", "popcnt", "mca", "pdpe1gb", "apic",
"sse", "3dnow", "invtsc", "pni", "rdtscp", "svm", "mmxext", "osvw", "sse2",
"lahf_lm", "misalignsse", "cx16", |
| | "extapic", "pse36", "mtrr", "ibs"], "topology":
{"cores": 4, "cells": 4, "threads": 1, "sockets": 1}}
|
| current_workload | 0
|
| disk_available_least | 114
|
| free_disk_gb | 115
|
| free_ram_mb | 128427
|
| host_ip | 192.168.178.235
|
| host_time | 16:47:45
|
| hypervisor_hostname | compute1O
|
| hypervisor_type | QEMU
|
| hypervisor_version | 2008000
|
| id | 3
|
| load_average | 1.61, 1.71, 1.78
|
| local_gb | 115
|
| local_gb_used | 0
|
| memory_mb | 128939
|
| memory_mb_used | 734
|
| running_vms | 0
|
| service_host | compute1O
|
| service_id | 9
|
| state | up
|
| status | enabled
|
| uptime | 3:14
|
| users | 1
|
| vcpus | 16
|
| vcpus_used | 0
|
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Command was:
root@controllerO:~# openstack server migrate --live compute1o
eb335c1a-836e-4e65-b900-d7fbd1b78227
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/
and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID:
req-a84a7163-98f6-496e-8965-656db5773a10)
nova-api.log excerpt follows:
:
:
2017-09-10 16:36:36.902 3260 INFO nova.osapi_compute.wsgi.server
[req-bd9d463c-fcc1-4d09-ad81-b3b19d9880ba bf2bd08f16a34c629c7d211b71b4f0b8
83a0f55c38b14931bf1fed6758af8691 - default default] 192.168.178.212 "GET
/v2.1/servers/eb335c1a-836e-4e65-b900-d7fbd1b78227 HTTP/1.1" status: 200 len:
1763 time: 0.2645531
2017-09-10 16:36:36.930 3260 INFO nova.osapi_compute.wsgi.server
[req-135317cd-f1e6-4e49-bb63-520db5b5567a bf2bd08f16a34c629c7d211b71b4f0b8
83a0f55c38b14931bf1fed6758af8691 - default default] 192.168.178.212 "GET
/v2.1/flavors/1 HTTP/1.1" status: 200 len: 691 time: 0.0239990
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
[req-a84a7163-98f6-496e-8965-656db5773a10 ca81492f80b3453ab0b27db819c0396e
a504f787d71245109b33822cfb9771cf - default default] Unexpected exception in API
method
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions Traceback
(most recent call last):
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 338,
in wrapped
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return
f(*args, **kwargs)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108,
in wrapper
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return
func(*args, **kwargs)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108,
in wrapper
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return
func(*args, **kwargs)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108,
in wrapper
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return
func(*args, **kwargs)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/migrate_server.py",
line 100, in _migrate_live
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
disk_over_commit, host, force, async)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 170, in inner
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return
function(self, context, instance, *args, **kwargs)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 178, in _wrapped
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return
fn(self, context, instance, *args, **kwargs)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 151, in inner
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return
f(self, context, instance, *args, **kw)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3910, in
live_migrate
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
messaging_timeout)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
self.force_reraise()
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
six.reraise(self.type_, self.value, self.tb)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3902, in
live_migrate
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
request_spec=request_spec, async=async)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 112, in
live_migrate_instance
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
request_spec=request_spec)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 322, in
migrate_server
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return
cctxt.call(context, 'migrate_server', **kw)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in
call
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
retry=self.retry)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in
_send
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
timeout=timeout, retry=retry)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line
458, in send
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
retry=retry)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line
447, in _send
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions result =
self._waiter.wait(msg_id, timeout)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line
339, in wait
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions message
= self.waiters.get(msg_id, timeout=timeout)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line
238, in get
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions 'to
message ID %s' % msg_id)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
MessagingTimeout: Timed out waiting for a reply to message ID
77a90c5ad6ca4097a1407619e37f1890
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
2017-09-10 16:36:41.178 3258 INFO nova.api.openstack.wsgi
[req-a84a7163-98f6-496e-8965-656db5773a10 ca81492f80b3453ab0b27db819c0396e
a504f787d71245109b33822cfb9771cf - default default] HTTP exception thrown:
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and
attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2017-09-10 16:36:41.180 3258 INFO nova.osapi_compute.wsgi.server
[req-a84a7163-98f6-496e-8965-656db5773a10 ca81492f80b3453ab0b27db819c0396e
a504f787d71245109b33822cfb9771cf - default default] 192.168.178.212 "POST
/v2.1/servers/eb335c1a-836e-4e65-b900-d7fbd1b78227/action HTTP/1.1" status: 500
len: 583 time: 60.1477718
2017-09-10 16:36:41.345 3258 INFO oslo_messaging._drivers.amqpdriver [-] No
calling threads waiting for msg_id : 77a90c5ad6ca4097a1407619e37f1890
2017-09-10 16:36:50.448 3250 INFO nova.osapi_compute.wsgi.server
[req-997edc7b-fc4d-4d96-8d12-2bb0810e4af5 bf2bd08f16a34c629c7d211b71b4f0b8
83a0f55c38b14931bf1fed6758af8691 - default default] 192.168.178.212 "GET
/v2.1/servers/eb335c1a-836e-4e65-b900-d7fbd1b78227 HTTP/1.1" status: 200 len:
1753 time: 0.3524611
2017-09-10 16:36:50.475 3250 INFO nova.osapi_compute.wsgi.server
[req-9126da6b-10bb-4ed0-b432-3b80c6f7b80f bf2bd08f16a34c629c7d211b71b4f0b8
83a0f55c38b14931bf1fed6758af8691 - default default] 192.168.178.212 "GET
/v2.1/flavors/1 HTTP/1.1" status: 200 len: 691 time: 0.0227520
:
:
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1716234/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
