** Changed in: nova
Status: New => Invalid
** Changed in: nova/queens
Status: Fix Committed => Fix Released
** Changed in: nova/queens
Importance: Undecided => High
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1775075
Title:
EndpointNotFound raised by Pike n-cpu when running alongside Queens
n-api
Status in OpenStack Compute (nova):
Invalid
Status in OpenStack Compute (nova) queens series:
Fix Released
Bug description:
Description
===========
During a P to Q upgrade n-cpu processes still running P will be unable to
find the volumev2 endpoint when running alongside Q n-api processes due to the
following change:
Update cinder in RequestContext service catalog
https://review.openstack.org/#/c/510947/
This results in failures anytime the P n-cpu process attempts to
interact with the volume service, for example during LM from the node:
2018-06-02 00:19:17.683 1 WARNING nova.virt.libvirt.driver
[req-3712be3d-b883-4fe1-bab0-83ee44bd5bb5 e16a043a84b14e2b8afbdd1b8677259f
cb92ed750eac463faf8935cb137f1e60 - default default] [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] Error monitoring migration: internalURL
endpoint for volumev2 service named cinderv2 not found: EndpointNotFound:
internalURL endpoint for volumev2 service named cinderv2 not found
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] Traceback (most recent call last):
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6817, in
_live_migration
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] finish_event, disk_paths)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6728, in
_live_migration_monitor
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] migrate_data)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in
wrapped
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] function_name, call_dict, binary)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] self.force_reraise()
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] six.reraise(self.type_, self.value,
self.tb)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in
wrapped
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] return f(self, context, *args, **kw)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 218, in
decorated_function
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] kwargs['instance'], e, sys.exc_info())
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] self.force_reraise()
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] six.reraise(self.type_, self.value,
self.tb)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 206, in
decorated_function
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] return function(self, context, *args,
**kwargs)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5684, in
_post_live_migration
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] migrate_data)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7169, in
post_live_migration
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] connector)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 235, in wrapper
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] res = method(self, ctx, *args,
**kwargs)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 257, in wrapper
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] res = method(self, ctx, volume_id,
*args, **kwargs)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 410, in
initialize_connection
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b]
context).volumes.initialize_connection(volume_id, connector)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 116, in
cinderclient
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] url = _SESSION.get_endpoint(auth,
**service_parameters)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 947, in
get_endpoint
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] return auth.get_endpoint(self,
**kwargs)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/nova/context.py", line 78, in get_endpoint
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] region_name=region_name)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] return wrapped(*args, **kwargs)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py",
line 344, in url_for
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] endpoint_id=endpoint_id).url
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] return wrapped(*args, **kwargs)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] File
"/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py",
line 407, in endpoint_data_for
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] raise exceptions.EndpointNotFound(msg)
2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance:
1a3800b3-9f75-4999-b726-a1afb0ebdd9b] EndpointNotFound: internalURL endpoint
for volumev2 service named cinderv2 not found
Steps to reproduce
==================
* Starting with a P environment
* Upgrade n-api/n-schd/n-cond etc to Q
* Attempt to attach a volume to an instance on a P n-cpu host
Expected result
===============
volumev2 endpoint found and used.
Actual result
=============
EndpointNotFound
Environment
===========
1. Exact version of OpenStack you are running. See the following
list for all releases: http://docs.openstack.org/releases/
Queens control plane, Pike computes.
2. Which hypervisor did you use?
(For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
What's the version of that?
Libvirt + KVM
2. Which storage type did you use?
(For example: Ceph, LVM, GPFS, ...)
What's the version of that?
N/A
3. Which networking type did you use?
(For example: nova-network, Neutron with OpenVSwitch, ...)
N/A
Logs & Configs
==============
LM between OSP 12 and OSP 13 computes fail during major upgrade
https://bugzilla.redhat.com/show_bug.cgi?id=1585656
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1775075/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
