Reviewed:  https://review.openstack.org/549737
Committed: 
https://git.openstack.org/cgit/openstack/nova/commit/?id=b81a66b3b0b1ce9f56280dda40ac871132cb29f2
Submitter: Zuul
Branch:    master

commit b81a66b3b0b1ce9f56280dda40ac871132cb29f2
Author: git-harry <git-ha...@live.co.uk>
Date:   Mon Mar 5 10:48:28 2018 +0000

    Fix version cap when no nova-compute started
    
    When a zero service version is returned, it means that we have no
    services running for the requested binary. In that case, we should
    assume the latest version available until told otherwise. This usually
    happens in first-start cases, where everything is likely to be up to
    date anyway.
    
    This change addresses an issue where the version returned had been
    hard-coded to 4.11 (mitaka).
    
    Change-Id: I696a8ea8adbe9481e11407ecafd5e47b2bd29804
    Closes-bug: 1753443


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1753443

Title:
  os_nova: upgrade_levels/compute=auto failure on master

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) queens series:
  Confirmed
Status in openstack-ansible:
  Fix Released

Bug description:
  It looks like a recent change [1] in nova, to remove RPC 4.x support,
  has exposed a bug when using upgrade_levels/compute=auto on a new
  deployment. This is blocking the openstack-ansible-os_nova master
  gate.

  Tempest tests are failing, the following in nova-conductor.log shows the 
failure:
  ```
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server 
[req-9c3f2dd3-81dd-4275-9a61-a3a859dde29d 3639ea84ebcf4c858de98eeede6789a9 
3b9624e03ed740f483c64301d0d11372 - default default] Exception during message 
handling: RPCVersionCapError: Requested message version, 5.0 is incompatible.  
It needs to be equal in major version and less than or equal in minor version 
as the specified version cap 4.11.
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server Traceback (most 
recent call last):
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
 line 163, in _process_incoming
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server     res = 
self.dispatcher.dispatch(message)
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
 line 220, in dispatch
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server     return 
self._do_dispatch(endpoint, method, ctxt, args)
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
 line 190, in _do_dispatch
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server     result = 
func(ctxt, **new_args)
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/nova/conductor/manager.py",
 line 1265, in schedule_and_build_instances
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server     
limits=host.limits, host_list=host_list)
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/nova/compute/rpcapi.py",
 line 1030, in build_and_run_instance
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server     
cctxt.cast(ctxt, 'build_and_run_instance', **kwargs)
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
 line 149, in cast
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server     
self._check_version_cap(msg.get('version'))
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
 line 126, in _check_version_cap
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server     
version_cap=self.version_cap)
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server 
RPCVersionCapError: Requested message version, 5.0 is incompatible.  It needs 
to be equal in major version and less than or equal in minor version as the 
specified version cap 4.11.
  2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server 
  ```

  When openstack-ansible-os_nova is used for a new deployment, the following 
appears in the logs:
  ```
  2018-03-02 17:25:55.954 19495 DEBUG nova.compute.rpcapi 
[req-97c173ed-052e-4ce7-8314-d220dfdab8e7 - - - - -] Not caching compute RPC 
version_cap, because min service_version is 0. Please ensure a nova-compute 
service has been started. Defaulting to Mitaka RPC. _determine_version_cap 
/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/compute/rpcapi.py:408
  ```

  The reference to Mitaka is caused by [2], it looks to be intended to
  set the version cap to be as permissive as possible (N to N+1
  upgrades) but it appears it hasn't been updated since it was first
  added for newton.

  Restarting the services addresses the issue, observed by:
  ```
  2018-03-04 21:42:14.367 21270 INFO nova.compute.rpcapi 
[req-95678c1e-8465-4059-8095-70479085b179 - - - - -] Automatically selected 
compute RPC version 5.0 from minimum service version 30
  ```

  It seems like there may be two issues exposed here, one is the bug in
  nova setting the minimum version and the other is how OSA handles the
  deployment. With the default OSA deployment the minimum RPC version
  will change with a restart, it would seem that has the potential to
  cause failures if the order of the restarts is not controlled given
  those restarts are not triggered by the deployment process.

  [1] 
https://github.com/openstack/nova/commit/a761e57368280b6d3e931831ecd393fd5787b3ef
  [2] 
https://github.com/openstack/nova/blob/a761e57368280b6d3e931831ecd393fd5787b3ef/nova/compute/rpcapi.py#L384-L392

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1753443/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

Reply via email to