Public bug reported:
Description
===========
When trying to live-migrate an instance, the instance start on the second node
but then nova throw a "service catalog is empty".
Steps to reproduce
==================
* openstack server migrate 2a58b49e-08df-471b-9560-a9281621b2d1 --live
ruby.makz.me
Expected result
===============
The instance is migrated on the ruby.makz.me node
Actual result
=============
Nova start the instance on the second node for the live migration, nova
controller throw a "Migration error: The service catalog is empty"
Environment
===========
Controller
openstack-nova-scheduler-18.0.0-1.el7.noarch
python-nova-18.0.0-1.el7.noarch
openstack-nova-console-18.0.0-1.el7.noarch
openstack-nova-common-18.0.0-1.el7.noarch
openstack-nova-conductor-18.0.0-1.el7.noarch
openstack-nova-spicehtml5proxy-18.0.0-1.el7.noarch
python2-novaclient-11.0.0-1.el7.noarch
openstack-nova-api-18.0.0-1.el7.noarch
openstack-nova-placement-api-18.0.0-1.el7.noarch
openstack-nova-novncproxy-18.0.0-1.el7.noarch
Compute node
openstack-nova-compute-18.0.0-1.el7.noarch
openstack-nova-common-18.0.0-1.el7.noarch
python-nova-18.0.0-1.el7.noarch
python2-novaclient-11.0.0-1.el7.noarch
2. Which hypervisor did you use?
Libvirt + KVM
libvirt-daemon-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-secret-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-mpath-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-network-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-logical-3.9.0-14.el7_5.7.x86_64
libvirt-libs-3.9.0-14.el7_5.7.x86_64
libvirt-client-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-core-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-iscsi-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-nwfilter-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-rbd-3.9.0-14.el7_5.7.x86_64
libvirt-python-3.9.0-1.el7.x86_64
libvirt-daemon-driver-storage-disk-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-interface-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-scsi-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-kvm-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-nodedev-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.7.x86_64
qemu-kvm-ev-2.10.0-21.el7_5.4.1.x86_64
qemu-img-ev-2.10.0-21.el7_5.4.1.x86_64
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.7.x86_64
qemu-kvm-common-ev-2.10.0-21.el7_5.4.1.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
centos-release-qemu-ev-1.0-3.el7.centos.noarch
2. Which storage type did you use?
Cinder + LVM
python-cinder-13.0.0-1.el7.noarch
openstack-cinder-13.0.0-1.el7.noarch
python2-cinderclient-4.0.1-1.el7.noarch
3. Which networking type did you use?
openstack-neutron-13.0.1-1.el7.noarch
openstack-neutron-common-13.0.1-1.el7.noarch
openstack-neutron-ml2-13.0.1-1.el7.noarch
python2-neutron-lib-1.18.0-1.el7.noarch
python2-neutronclient-6.9.0-1.el7.noarch
openstack-neutron-linuxbridge-13.0.1-1.el7.noarch
python-neutron-13.0.1-1.el7.noarch
python2-openvswitch-2.9.0-3.el7.noarch
Logs & Configs
==============
Controller
==========
==> /var/log/neutron/server.log <==
2018-09-25 15:48:40.734 6677 INFO neutron.wsgi
[req-b5ffce01-5f45-4099-9f4e-dca8d66bd04f 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET
/v2.0/ports?device_id=2a58b49e-08df-471b-9560-a9281621b2d1 HTTP/1.1" status:
200 len: 1189 time: 0.5954452
2018-09-25 15:48:40.830 6677 INFO neutron.wsgi
[req-d524e8ea-7201-4f31-a774-dd9b88e3cd1f 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET
/v2.0/security-groups?id=2b2618f8-8c86-4ff0-8beb-52a9e8e5edad&id=1f47e4f6-62d5-4e91-bb91-f27c525bd0be&id=4e0d94e2-e915-41ea-8103-636af0d5c82f&id=9cfc779e-dd64-44f1-85b2-a9c6c852cfba
HTTP/1.1" status: 200 len: 11665 time: 0.0915859
==> /var/log/nova/nova-api.log <==
2018-09-25 15:48:40.837 24668 INFO nova.osapi_compute.wsgi.server
[req-c2c432cc-4911-4f8e-a8fc-9b2a732ed93c 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET
/v2.1/servers/2a58b49e-08df-471b-9560-a9281621b2d1 HTTP/1.1" status: 200 len:
2108 time: 1.3573151
==> /var/log/nova/nova-placement-api.log <==
2018-09-25 15:48:42.552 8830 INFO nova.api.openstack.placement.requestlog
[req-a2384cab-5e52-4c99-a4db-56c96475f4ee 0412e7701b674098b81afc2424781afb
a03be5acfab64945a9259b798130d1eb - default default] 192.168.58.1 "GET
/allocations/2a58b49e-08df-471b-9560-a9281621b2d1" status: 200 len: 245
microversion: 1.28\x1b[00m
2018-09-25 15:48:42.746 8831 INFO nova.api.openstack.placement.requestlog
[req-95abd683-8e2f-4955-8c2d-25b6c88db49b 0412e7701b674098b81afc2424781afb
a03be5acfab64945a9259b798130d1eb - default default] 192.168.58.1 "POST
/allocations" status: 204 len: 0 microversion: 1.13\x1b[00m
==> /var/log/nova/nova-conductor.log <==
2018-09-25 15:48:43.132 24594 WARNING oslo_config.cfg
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Deprecated: Option "url"
from group "neutron" is deprecated for removal (Endpoint lookup uses the
service catalog via common keystoneauth1 Adapter configuration options. In the
current release, "url" will override this behavior, but will be ignored and/or
removed in a future release. To achieve the same result, use the
endpoint_override option instead.). Its value may be silently ignored in the
future.
==> /var/log/neutron/server.log <==
2018-09-25 15:48:43.151 6674 INFO neutron.wsgi
[req-0751549f-ce04-4fd5-a590-e0fcbf4fc39d 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET
/v2.0/extensions HTTP/1.1" status: 200 len: 9702 time: 0.0112290
==> /var/log/nova/nova-placement-api.log <==
2018-09-25 15:48:45.234 8829 INFO nova.api.openstack.placement.requestlog
[req-b61f462c-1f34-4855-aa4c-83042f503897 0412e7701b674098b81afc2424781afb
a03be5acfab64945a9259b798130d1eb - default default] 192.168.58.1 "POST
/allocations" status: 204 len: 0 microversion: 1.13\x1b[00m
==> /var/log/nova/nova-conductor.log <==
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Migration of instance
2a58b49e-08df-471b-9560-a9281621b2d1 to host ruby.makz.me unexpectedly failed.:
EmptyCatalog: The service catalog is empty.
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager Traceback (most
recent call last):
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 424, in
_live_migrate
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager task.execute()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 27, in
wrap
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager self.rollback()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
self.force_reraise()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
six.reraise(self.type_, self.value, self.tb)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 24, in
wrap
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager return
original(self)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 42, in
execute
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager return
self._execute()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line
97, in _execute
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager source_node,
dest_node = self._check_requested_destination()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line
171, in _check_requested_destination
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
self._call_livem_checks_on_host(self.destination)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line
252, in _call_livem_checks_on_host
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
self._bind_ports_on_destination(destination))
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line
263, in _bind_ports_on_destination
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager self.context,
self.instance, destination)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1277, in
bind_ports_to_host
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager json=data,
raise_exc=False)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager return
self.request(url, 'POST', **kwargs)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 213, in
request
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager return
self.session.request(url, method, **kwargs)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 706, in
request
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
**endpoint_filter)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1113, in
get_endpoint
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager return
auth.get_endpoint(self, **kwargs)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in
get_endpoint
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
allow_version_hack=allow_version_hack, **kwargs)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 279, in
get_endpoint_data
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
service_name=service_name)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py",
line 425, in endpoint_data_for
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager raise
exceptions.EmptyCatalog('The service catalog is empty.')
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager EmptyCatalog: The
service catalog is empty.
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
2018-09-25 15:48:45.269 24594 WARNING nova.scheduler.utils
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Failed to
compute_task_migrate_server: The service catalog is empty.: EmptyCatalog: The
service catalog is empty.
2018-09-25 15:48:45.270 24594 WARNING nova.scheduler.utils
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] [instance:
2a58b49e-08df-471b-9560-a9281621b2d1] Setting instance to ERROR state.:
EmptyCatalog: The service catalog is empty.
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Exception during message
handling: MigrationError: Migration error: The service catalog is empty.
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in
_process_incoming
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265,
in dispatch
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194,
in _do_dispatch
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in
inner
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server return
func(*args, **kwargs)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 86, in
wrapper
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server return
fn(self, context, *args, **kwargs)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in
decorated_function
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 281, in
migrate_server
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server
block_migration, disk_over_commit, request_spec)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 454, in
_live_migrate
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server raise
exception.MigrationError(reason=six.text_type(ex))
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server MigrationError:
Migration error: The service catalog is empty.
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server
==> /var/log/nova/nova-api.log <==
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Unexpected exception in API
method: MigrationError_Remote: Migration error: The service catalog is empty.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
163, in _process_incoming
res = self.dispatcher.dispatch(message)
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch
result = func(ctxt, **new_args)
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
226, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 86,
in wrapper
return fn(self, context, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in
decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 281,
in migrate_server
block_migration, disk_over_commit, request_spec)
File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 454,
in _live_migrate
raise exception.MigrationError(reason=six.text_type(ex))
MigrationError: Migration error: The service catalog is empty.
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi Traceback (most
recent call last):
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in
wrapped
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return f(*args,
**kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110,
in wrapper
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110,
in wrapper
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110,
in wrapper
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/migrate_server.py",
line 107, in _migrate_live
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi async_)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 205, in inner
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
function(self, context, instance, *args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 213, in _wrapped
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return fn(self,
context, instance, *args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 153, in inner
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return f(self,
context, instance, *args, **kw)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 4396, in
live_migrate
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
request_spec=request_spec, async_=async_)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 112, in
live_migrate_instance
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
request_spec=request_spec)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 338, in
migrate_server
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
cctxt.call(context, 'migrate_server', **kw)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in
call
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
retry=self.retry)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in
_send
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi retry=retry)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
584, in send
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
call_monitor_timeout, retry=retry)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
575, in _send
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi raise result
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
MigrationError_Remote: Migration error: The service catalog is empty.
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi Traceback (most
recent call last):
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in
_process_incoming
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi res =
self.dispatcher.dispatch(message)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265,
in dispatch
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
self._do_dispatch(endpoint, method, ctxt, args)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194,
in _do_dispatch
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi result =
func(ctxt, **new_args)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in
inner
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 86, in
wrapper
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return fn(self,
context, *args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in
decorated_function
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
function(self, context, *args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 281, in
migrate_server
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
block_migration, disk_over_commit, request_spec)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 454, in
_live_migrate
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi raise
exception.MigrationError(reason=six.text_type(ex))
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi MigrationError:
Migration error: The service catalog is empty.
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.481 24668 INFO nova.api.openstack.wsgi
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] HTTP exception thrown:
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and
attach the Nova API log if possible.
<class 'MigrationError_Remote'>
2018-09-25 15:48:45.483 24668 INFO nova.osapi_compute.wsgi.server
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "POST
/v2.1/servers/2a58b49e-08df-471b-9560-a9281621b2d1/action HTTP/1.1" status: 500
len: 628 time: 4.6382320
Compute node
============
2018-09-25 15:48:42.961 1741 INFO nova.virt.libvirt.driver [req-
d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Instance launched
has CPU info: {"vendor": "Intel", "model": "Westmere-IBRS", "arch":
"x86_64", "features": ["pge", "clflush", "sep", "syscall", "vme",
"dtes64", "stibp", "msr", "vmx", "xtpr", "cmov", "ssse3", "est", "pat",
"monitor", "smx", "pbe", "lm", "tsc", "nx", "fxsr", "tm", "sse4.1",
"pae", "sse4.2", "pclmuldq", "cx16", "pcid", "mmx", "arat", "cx8",
"mce", "de", "rdtscp", "ht", "dca", "lahf_lm", "pdcm", "mca", "pdpe1gb",
"apic", "sse", "pse", "ds", "invtsc", "pni", "tm2", "aes", "sse2", "ss",
"ds_cpl", "acpi", "spec-ctrl", "fpu", "ssbd", "pse36", "mtrr",
"popcnt"], "topology": {"cores": 6, "cells": 2, "threads": 2, "sockets":
1}}
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1794302
Title:
Live migration : service catalog is empty
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
When trying to live-migrate an instance, the instance start on the second
node but then nova throw a "service catalog is empty".
Steps to reproduce
==================
* openstack server migrate 2a58b49e-08df-471b-9560-a9281621b2d1 --live
ruby.makz.me
Expected result
===============
The instance is migrated on the ruby.makz.me node
Actual result
=============
Nova start the instance on the second node for the live migration, nova
controller throw a "Migration error: The service catalog is empty"
Environment
===========
Controller
openstack-nova-scheduler-18.0.0-1.el7.noarch
python-nova-18.0.0-1.el7.noarch
openstack-nova-console-18.0.0-1.el7.noarch
openstack-nova-common-18.0.0-1.el7.noarch
openstack-nova-conductor-18.0.0-1.el7.noarch
openstack-nova-spicehtml5proxy-18.0.0-1.el7.noarch
python2-novaclient-11.0.0-1.el7.noarch
openstack-nova-api-18.0.0-1.el7.noarch
openstack-nova-placement-api-18.0.0-1.el7.noarch
openstack-nova-novncproxy-18.0.0-1.el7.noarch
Compute node
openstack-nova-compute-18.0.0-1.el7.noarch
openstack-nova-common-18.0.0-1.el7.noarch
python-nova-18.0.0-1.el7.noarch
python2-novaclient-11.0.0-1.el7.noarch
2. Which hypervisor did you use?
Libvirt + KVM
libvirt-daemon-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-secret-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-mpath-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-network-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-logical-3.9.0-14.el7_5.7.x86_64
libvirt-libs-3.9.0-14.el7_5.7.x86_64
libvirt-client-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-core-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-iscsi-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-nwfilter-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-rbd-3.9.0-14.el7_5.7.x86_64
libvirt-python-3.9.0-1.el7.x86_64
libvirt-daemon-driver-storage-disk-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-interface-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-scsi-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-kvm-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-nodedev-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.7.x86_64
qemu-kvm-ev-2.10.0-21.el7_5.4.1.x86_64
qemu-img-ev-2.10.0-21.el7_5.4.1.x86_64
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.7.x86_64
qemu-kvm-common-ev-2.10.0-21.el7_5.4.1.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
centos-release-qemu-ev-1.0-3.el7.centos.noarch
2. Which storage type did you use?
Cinder + LVM
python-cinder-13.0.0-1.el7.noarch
openstack-cinder-13.0.0-1.el7.noarch
python2-cinderclient-4.0.1-1.el7.noarch
3. Which networking type did you use?
openstack-neutron-13.0.1-1.el7.noarch
openstack-neutron-common-13.0.1-1.el7.noarch
openstack-neutron-ml2-13.0.1-1.el7.noarch
python2-neutron-lib-1.18.0-1.el7.noarch
python2-neutronclient-6.9.0-1.el7.noarch
openstack-neutron-linuxbridge-13.0.1-1.el7.noarch
python-neutron-13.0.1-1.el7.noarch
python2-openvswitch-2.9.0-3.el7.noarch
Logs & Configs
==============
Controller
==========
==> /var/log/neutron/server.log <==
2018-09-25 15:48:40.734 6677 INFO neutron.wsgi
[req-b5ffce01-5f45-4099-9f4e-dca8d66bd04f 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET
/v2.0/ports?device_id=2a58b49e-08df-471b-9560-a9281621b2d1 HTTP/1.1" status:
200 len: 1189 time: 0.5954452
2018-09-25 15:48:40.830 6677 INFO neutron.wsgi
[req-d524e8ea-7201-4f31-a774-dd9b88e3cd1f 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET
/v2.0/security-groups?id=2b2618f8-8c86-4ff0-8beb-52a9e8e5edad&id=1f47e4f6-62d5-4e91-bb91-f27c525bd0be&id=4e0d94e2-e915-41ea-8103-636af0d5c82f&id=9cfc779e-dd64-44f1-85b2-a9c6c852cfba
HTTP/1.1" status: 200 len: 11665 time: 0.0915859
==> /var/log/nova/nova-api.log <==
2018-09-25 15:48:40.837 24668 INFO nova.osapi_compute.wsgi.server
[req-c2c432cc-4911-4f8e-a8fc-9b2a732ed93c 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET
/v2.1/servers/2a58b49e-08df-471b-9560-a9281621b2d1 HTTP/1.1" status: 200 len:
2108 time: 1.3573151
==> /var/log/nova/nova-placement-api.log <==
2018-09-25 15:48:42.552 8830 INFO nova.api.openstack.placement.requestlog
[req-a2384cab-5e52-4c99-a4db-56c96475f4ee 0412e7701b674098b81afc2424781afb
a03be5acfab64945a9259b798130d1eb - default default] 192.168.58.1 "GET
/allocations/2a58b49e-08df-471b-9560-a9281621b2d1" status: 200 len: 245
microversion: 1.28\x1b[00m
2018-09-25 15:48:42.746 8831 INFO nova.api.openstack.placement.requestlog
[req-95abd683-8e2f-4955-8c2d-25b6c88db49b 0412e7701b674098b81afc2424781afb
a03be5acfab64945a9259b798130d1eb - default default] 192.168.58.1 "POST
/allocations" status: 204 len: 0 microversion: 1.13\x1b[00m
==> /var/log/nova/nova-conductor.log <==
2018-09-25 15:48:43.132 24594 WARNING oslo_config.cfg
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Deprecated: Option "url"
from group "neutron" is deprecated for removal (Endpoint lookup uses the
service catalog via common keystoneauth1 Adapter configuration options. In the
current release, "url" will override this behavior, but will be ignored and/or
removed in a future release. To achieve the same result, use the
endpoint_override option instead.). Its value may be silently ignored in the
future.
==> /var/log/neutron/server.log <==
2018-09-25 15:48:43.151 6674 INFO neutron.wsgi
[req-0751549f-ce04-4fd5-a590-e0fcbf4fc39d 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET
/v2.0/extensions HTTP/1.1" status: 200 len: 9702 time: 0.0112290
==> /var/log/nova/nova-placement-api.log <==
2018-09-25 15:48:45.234 8829 INFO nova.api.openstack.placement.requestlog
[req-b61f462c-1f34-4855-aa4c-83042f503897 0412e7701b674098b81afc2424781afb
a03be5acfab64945a9259b798130d1eb - default default] 192.168.58.1 "POST
/allocations" status: 204 len: 0 microversion: 1.13\x1b[00m
==> /var/log/nova/nova-conductor.log <==
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Migration of instance
2a58b49e-08df-471b-9560-a9281621b2d1 to host ruby.makz.me unexpectedly failed.:
EmptyCatalog: The service catalog is empty.
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager Traceback (most
recent call last):
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 424, in
_live_migrate
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager task.execute()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 27, in
wrap
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager self.rollback()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
self.force_reraise()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
six.reraise(self.type_, self.value, self.tb)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 24, in
wrap
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager return
original(self)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 42, in
execute
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager return
self._execute()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line
97, in _execute
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager source_node,
dest_node = self._check_requested_destination()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line
171, in _check_requested_destination
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
self._call_livem_checks_on_host(self.destination)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line
252, in _call_livem_checks_on_host
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
self._bind_ports_on_destination(destination))
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line
263, in _bind_ports_on_destination
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager self.context,
self.instance, destination)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1277, in
bind_ports_to_host
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager json=data,
raise_exc=False)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager return
self.request(url, 'POST', **kwargs)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 213, in
request
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager return
self.session.request(url, method, **kwargs)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 706, in
request
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
**endpoint_filter)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1113, in
get_endpoint
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager return
auth.get_endpoint(self, **kwargs)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in
get_endpoint
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
allow_version_hack=allow_version_hack, **kwargs)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 279, in
get_endpoint_data
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
service_name=service_name)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py",
line 425, in endpoint_data_for
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager raise
exceptions.EmptyCatalog('The service catalog is empty.')
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager EmptyCatalog: The
service catalog is empty.
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager
2018-09-25 15:48:45.269 24594 WARNING nova.scheduler.utils
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Failed to
compute_task_migrate_server: The service catalog is empty.: EmptyCatalog: The
service catalog is empty.
2018-09-25 15:48:45.270 24594 WARNING nova.scheduler.utils
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] [instance:
2a58b49e-08df-471b-9560-a9281621b2d1] Setting instance to ERROR state.:
EmptyCatalog: The service catalog is empty.
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Exception during message
handling: MigrationError: Migration error: The service catalog is empty.
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in
_process_incoming
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265,
in dispatch
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194,
in _do_dispatch
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in
inner
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server return
func(*args, **kwargs)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 86, in
wrapper
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server return
fn(self, context, *args, **kwargs)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in
decorated_function
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 281, in
migrate_server
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server
block_migration, disk_over_commit, request_spec)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 454, in
_live_migrate
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server raise
exception.MigrationError(reason=six.text_type(ex))
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server MigrationError:
Migration error: The service catalog is empty.
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server
==> /var/log/nova/nova-api.log <==
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Unexpected exception in API
method: MigrationError_Remote: Migration error: The service catalog is empty.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
163, in _process_incoming
res = self.dispatcher.dispatch(message)
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch
result = func(ctxt, **new_args)
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
226, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 86,
in wrapper
return fn(self, context, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141,
in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line
281, in migrate_server
block_migration, disk_over_commit, request_spec)
File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line
454, in _live_migrate
raise exception.MigrationError(reason=six.text_type(ex))
MigrationError: Migration error: The service catalog is empty.
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi Traceback (most
recent call last):
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in
wrapped
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
f(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110,
in wrapper
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110,
in wrapper
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110,
in wrapper
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/migrate_server.py",
line 107, in _migrate_live
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi async_)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 205, in inner
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
function(self, context, instance, *args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 213, in _wrapped
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
fn(self, context, instance, *args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 153, in inner
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
f(self, context, instance, *args, **kw)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 4396, in
live_migrate
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
request_spec=request_spec, async_=async_)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 112, in
live_migrate_instance
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
request_spec=request_spec)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 338, in
migrate_server
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
cctxt.call(context, 'migrate_server', **kw)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in
call
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
retry=self.retry)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in
_send
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi retry=retry)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
584, in send
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
call_monitor_timeout, retry=retry)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
575, in _send
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi raise result
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
MigrationError_Remote: Migration error: The service catalog is empty.
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi Traceback (most
recent call last):
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in
_process_incoming
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi res =
self.dispatcher.dispatch(message)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265,
in dispatch
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
self._do_dispatch(endpoint, method, ctxt, args)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194,
in _do_dispatch
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi result =
func(ctxt, **new_args)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in
inner
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 86, in
wrapper
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
fn(self, context, *args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in
decorated_function
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi return
function(self, context, *args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 281, in
migrate_server
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
block_migration, disk_over_commit, request_spec)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 454, in
_live_migrate
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi raise
exception.MigrationError(reason=six.text_type(ex))
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi MigrationError:
Migration error: The service catalog is empty.
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi
2018-09-25 15:48:45.481 24668 INFO nova.api.openstack.wsgi
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] HTTP exception thrown:
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and
attach the Nova API log if possible.
<class 'MigrationError_Remote'>
2018-09-25 15:48:45.483 24668 INFO nova.osapi_compute.wsgi.server
[req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "POST
/v2.1/servers/2a58b49e-08df-471b-9560-a9281621b2d1/action HTTP/1.1" status: 500
len: 628 time: 4.6382320
Compute node
============
2018-09-25 15:48:42.961 1741 INFO nova.virt.libvirt.driver [req-
d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Instance launched
has CPU info: {"vendor": "Intel", "model": "Westmere-IBRS", "arch":
"x86_64", "features": ["pge", "clflush", "sep", "syscall", "vme",
"dtes64", "stibp", "msr", "vmx", "xtpr", "cmov", "ssse3", "est",
"pat", "monitor", "smx", "pbe", "lm", "tsc", "nx", "fxsr", "tm",
"sse4.1", "pae", "sse4.2", "pclmuldq", "cx16", "pcid", "mmx", "arat",
"cx8", "mce", "de", "rdtscp", "ht", "dca", "lahf_lm", "pdcm", "mca",
"pdpe1gb", "apic", "sse", "pse", "ds", "invtsc", "pni", "tm2", "aes",
"sse2", "ss", "ds_cpl", "acpi", "spec-ctrl", "fpu", "ssbd", "pse36",
"mtrr", "popcnt"], "topology": {"cores": 6, "cells": 2, "threads": 2,
"sockets": 1}}
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1794302/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
