Dear all,
we are experiencing some problems with our openstack infrastructure after the
upgrade from havana to icehouse (on Ubuntu 12.04).
We have successfully upgraded the controller and network nodes (the neutron
server is running on the network node) with quite no downtime using the upgrade
compatibility flag set to icehouse-compat in nova.conf.
All the existing VMs running on the havana compute nodes were not affected by
the upgrade and worked normally. Note that we are using neutron + ml2 plugin
with ovs configured for both flat and gre networks. Now, we are encountering
problems with the gre network starting new VM on the havana compute nodes
whereas the public flat network works fine.
Moreover we are also experiencing some problems as we started to upgrade the
compute nodes: we have selected one compute node and executed dist-upgrade
using the icehouse repo. Unfortunately, something goes wrong in the VM
instatiation on this icehouse compute node and it seems to be related to the
external event notification between the neutron server and the nova service
/os-server-external-events.
In more details, we get the following:
- on the compute node, a timeout is notified during the instance creation:
WARNING nova.virt.libvirt.driver [req-a1e6e66e-9b18-458a-b31e-0d64eb098e81
769b7fafd6c145ccb8680e398702f14f 57f427029f1a45cc862a0c1321fd5ff6] Timeout
waiting for vif plugging callback for instance
65ffc677-0bd4-4ef6-84ef-852fad012fd1
We had to add the following two parameters in the nova.conf file to carry out
successfully the VM instantiation:
vif_plugging_is_fatal = False
vif_plugging_timeout = 2
- on the network node we get (/var/log/neutron/server.log):
2014-10-02 15:38:50.232 4559 DEBUG urllib3.connectionpool [-] "POST
/v2/15ee65175a6a4548b5422bb276732580/os-server-external-events HTTP/1.1" 500
128 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415
2014-10-02 15:38:50.234 4559 ERROR neutron.notifiers.nova [-] Failed to notify
nova on events: [{'status': 'completed', 'tag':
u'3d06fab7-5b82-4122-b519-6bed39b78ed9', 'name': 'network-vif-plugged',
'server_uuid': u'65ffc677-0bd4-4ef6-84ef-852fad012fd1'}]
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova Traceback (most
recent call last):
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File
"/usr/lib/python2.7/dist-packages/neutron/notifiers/nova.py", line 222, in
send_events
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova batched_events)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File
"/usr/lib/python2.7/dist-packages/novaclient/v1_1/contrib/server_external_events.py",
line 39, in create
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova return_raw=True)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File
"/usr/lib/python2.7/dist-packages/novaclient/base.py", line 152, in _create
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova _resp, body =
self.api.client.post(url, body=body)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File
"/usr/lib/python2.7/dist-packages/novaclient/client.py", line 286, in post
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova return
self._cs_request(url, 'POST', **kwargs)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File
"/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in
_cs_request
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova **kwargs)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File
"/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in
_time_request
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova resp, body =
self.request(url, method, **kwargs)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File
"/usr/lib/python2.7/dist-packages/novaclient/client.py", line 236, in request
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova raise
exceptions.from_response(resp, body, url, method)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova ClientException: The
server has either erred or is incapable of performing the requested operation.
(HTTP 500) (Request-ID: req-3d7bf2e7-b7fc-4fd8-bbd3-658072525d05)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova
- on the controller node, we get the following error:
2014-10-02 15:38:50.223 9091 AUDIT
nova.api.openstack.compute.contrib.server_external_events
[req-3d7bf2e7-b7fc-4fd8-bbd3-658072525d05 5a85cf9e9b814c73828cbaff1810e0d9
15ee65175a6a4548b5422bb276732580] Creating event
network-vif-plugged:3d06fab7-5b82-4122-b519-6bed39b78ed9 for instance
65ffc677-0bd4-4ef6-84ef-852fad012fd1
2014-10-02 15:38:50.224 9091 ERROR nova.api.openstack
[req-3d7bf2e7-b7fc-4fd8-bbd3-658072525d05 5a85cf9e9b814c73828cbaff1810e0d9
15ee65175a6a4548b5422bb276732580] Caught error: Specified RPC version cap, 3.0,
is too low. Needs to be higher than 3.23.
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack Traceback (most recent
call last):
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in
__call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return
req.get_response(self.application)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack application,
catch_exc_info=False)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in
call_application
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack app_iter =
application(self.environ, start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return resp(environ,
start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 582, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return self.app(env,
start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return resp(environ,
start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return resp(environ,
start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack response =
self.app(environ, start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return resp(environ,
start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack resp =
self.call_func(req, *args, **self.kwargs)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return self.func(req,
*args, **kwargs)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 917, in
__call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack content_type, body,
accept)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 983, in
_process_stack
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack action_result =
self.dispatch(meth, request, action_args)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1070, in
dispatch
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return
method(req=request, **action_args)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/server_external_events.py",
line 146, in create
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack context,
accepted_instances, accepted_events)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3112, in
external_instance_event
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack context,
instances_by_host[host], events_by_host[host])
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 969, in
external_instance_event
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack events=events)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 130, in
cast
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack
self._check_version_cap(msg.get('version'))
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 115, in
_check_version_cap
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack
version_cap=self.version_cap)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack RPCVersionCapError:
Specified RPC version cap, 3.0, is too low. Needs to be higher than 3.23.
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack
2014-10-02 15:38:50.227 9091 INFO nova.api.openstack
[req-3d7bf2e7-b7fc-4fd8-bbd3-658072525d05 5a85cf9e9b814c73828cbaff1810e0d9
15ee65175a6a4548b5422bb276732580]
http://prisma-cloud.ba.infn.it:8774/v2/15ee65175a6a4548b5422bb276732580/os-server-external-events
returned with HTTP 500
What is causing the rpc version mismatch? Here is the list of neutron and nova
packages installed on the controller node:
# dpkg -l | grep -E "nova|neutron"
ii nova-ajax-console-proxy 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-api 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-cert 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-common 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-conductor 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-consoleauth 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-doc 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-novncproxy 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-scheduler 1:2014.1.2-0ubuntu1.1~cloud0
ii python-neutronclient 1:2.3.4-0ubuntu1~cloud0
ii python-nova 1:2014.1.2-0ubuntu1.1~cloud0
ii python-novaclient 1:2.17.0-0ubuntu1~cloud0
and on the network node:
# dpkg -l | grep -E "nova|neutron"
ii neutron-common 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-dhcp-agent 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-l3-agent 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-metadata-agent 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-plugin-ml2 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-plugin-openvswitch 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-plugin-openvswitch-agent 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-server 1:2014.1.2-0ubuntu1.1~cloud0
ii python-neutron 1:2014.1.2-0ubuntu1.1~cloud0
ii python-neutronclient 1:2.3.4-0ubuntu1~cloud0
ii python-novaclient 1:2.17.0-0ubuntu1~cloud0
Any help is appreciated.
Thank you in advance._______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : [email protected]
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
