Public bug reported:
Environment is neutron 14.0.2 with DVR and HA (OVS).
Upgraded a single compute or deployed new with 14.0.3.
Expected outcome:
Minor versions should be fully compatible and neutron should work with
the same major version.
Actual outcome:
Can't schedule instances on computes holding this version and neutron
services spew out errors.
neutron-server on controller/network node:
Exception during message handling: InvalidTargetVersion: Invalid target version
1.5
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 166, in _process_incoming
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 229, in inner
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server return
func(*args, **kwargs)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 148, in bulk_pull
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server **filter_kwargs)]
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
line 551, in obj_to_primitive
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server raise
exception.InvalidTargetVersion(version=target_version)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server
InvalidTargetVersion: Invalid target version 1.5
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server
neutron-openvswitch-agent on compute node:
Error while processing VIF ports: RemoteError: Remote error:
InvalidTargetVersion Invalid target version 1.5
[u'Traceback (most recent call last):\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 166, in _process_incoming\n res =
self.dispatcher.dispatch(message)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 229, in inner\n return func(*args, **kwargs)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 148, in bulk_pull\n **filter_kwargs)]\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
line 551, in obj_to_primitive\n raise
exception.InvalidTargetVersion(version=target_version)\n',
u'InvalidTargetVersion: Invalid target version 1.5\n'].
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2278, in rpc_loop
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info,
provisioning_needed)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py",
line 160, in wrapper
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result =
f(*args, **kwargs)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1845, in process_network_ports
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
port_info.get('updated', set()))
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py",
line 258, in setup_port_filters
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self.prepare_devices_filter(new_devices)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py",
line 125, in decorated_function
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args,
**kwargs)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py",
line 133, in prepare_devices_filter
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self._apply_port_filter(device_ids)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py",
line 143, in _apply_port_filter
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self.context, list(device_ids)[i:i + step])
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/securitygroups_rpc.py",
line 232, in security_group_info_for_devices
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ports =
self._get_devices_info(context, devices)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/securitygroups_rpc.py",
line 300, in _get_devices_info
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ovo =
self.rcache.get_resource_by_id('Port', device)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py",
line 61, in get_resource_by_id
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
agent_restarted=agent_restarted)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py",
line 79, in _flood_cache_for_query
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
filter_kwargs=filter_kwargs)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_log/helpers.py",
line 67, in wrapper
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
method(*args, **kwargs)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 114, in bulk_pull
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron_lib/rpc.py",
line 157, in call
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
self._original_context.call(ctxt, method, **kwargs)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
line 178, in call
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
retry=self.retry)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
line 128, in _send
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 645, in send
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout, retry=retry)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 636, in _send
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise result
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RemoteError:
Remote error: InvalidTargetVersion Invalid target version 1.5
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [u'Traceback
(most recent call last):\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 166, in _process_incoming\n res =
self.dispatcher.dispatch(message)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 229, in inner\n return func(*args, **kwargs)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 148, in bulk_pull\n **filter_kwargs)]\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
line 551, in obj_to_primitive\n raise
exception.InvalidTargetVersion(version=target_version)\n',
u'InvalidTargetVersion: Invalid target version 1.5\n'].
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
AND
Failed to get details for device 227d33c9-a8c5-45aa-bd98-8bc3ff2f6029:
RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5
[u'Traceback (most recent call last):\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 166, in _process_incoming\n res =
self.dispatcher.dispatch(message)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 229, in inner\n return func(*args, **kwargs)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 148, in bulk_pull\n **filter_kwargs)]\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
line 551, in obj_to_primitive\n raise
exception.InvalidTargetVersion(version=target_version)\n',
u'InvalidTargetVersion: Invalid target version 1.5\n'].
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc Traceback (most recent call
last):
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/rpc.py",
line 303, in get_devices_details_list_and_failed_devices
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc agent_restarted))
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/rpc.py",
line 312, in get_device_details
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc resources.PORT, device,
agent_restarted)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py",
line 61, in get_resource_by_id
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc
agent_restarted=agent_restarted)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py",
line 79, in _flood_cache_for_query
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc
filter_kwargs=filter_kwargs)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_log/helpers.py",
line 67, in wrapper
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc return method(*args,
**kwargs)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 114, in bulk_pull
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc
version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron_lib/rpc.py",
line 157, in call
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc return
self._original_context.call(ctxt, method, **kwargs)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
line 178, in call
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc retry=self.retry)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
line 128, in _send
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc retry=retry)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 645, in send
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc call_monitor_timeout,
retry=retry)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 636, in _send
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc raise result
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc RemoteError: Remote error:
InvalidTargetVersion Invalid target version 1.5
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc [u'Traceback (most recent
call last):\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 166, in _process_incoming\n res =
self.dispatcher.dispatch(message)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 229, in inner\n return func(*args, **kwargs)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 148, in bulk_pull\n **filter_kwargs)]\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
line 551, in obj_to_primitive\n raise
exception.InvalidTargetVersion(version=target_version)\n',
u'InvalidTargetVersion: Invalid target version 1.5\n'].
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc
AND
Clearing cache of registered ports, retries to resync were > 5:
RemoteError: Remote error: InvalidTargetVersion Invalid target version
1.5
I have switched back to 14.0.2 (5ish weeks old) and things are back on
track in the environment.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1854050
Title:
minor versions 14.0.2 & 14.0.3 are not compatible in dvr-ha
Status in neutron:
New
Bug description:
Environment is neutron 14.0.2 with DVR and HA (OVS).
Upgraded a single compute or deployed new with 14.0.3.
Expected outcome:
Minor versions should be fully compatible and neutron should work with
the same major version.
Actual outcome:
Can't schedule instances on computes holding this version and neutron
services spew out errors.
neutron-server on controller/network node:
Exception during message handling: InvalidTargetVersion: Invalid target
version 1.5
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 166, in _process_incoming
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 229, in inner
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server return
func(*args, **kwargs)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 148, in bulk_pull
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server
**filter_kwargs)]
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
line 551, in obj_to_primitive
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server raise
exception.InvalidTargetVersion(version=target_version)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server
InvalidTargetVersion: Invalid target version 1.5
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server
neutron-openvswitch-agent on compute node:
Error while processing VIF ports: RemoteError: Remote error:
InvalidTargetVersion Invalid target version 1.5
[u'Traceback (most recent call last):\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 166, in _process_incoming\n res =
self.dispatcher.dispatch(message)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 229, in inner\n return func(*args, **kwargs)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 148, in bulk_pull\n **filter_kwargs)]\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
line 551, in obj_to_primitive\n raise
exception.InvalidTargetVersion(version=target_version)\n',
u'InvalidTargetVersion: Invalid target version 1.5\n'].
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2278, in rpc_loop
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info,
provisioning_needed)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py",
line 160, in wrapper
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result =
f(*args, **kwargs)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1845, in process_network_ports
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
port_info.get('updated', set()))
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py",
line 258, in setup_port_filters
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self.prepare_devices_filter(new_devices)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py",
line 125, in decorated_function
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args,
**kwargs)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py",
line 133, in prepare_devices_filter
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self._apply_port_filter(device_ids)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py",
line 143, in _apply_port_filter
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self.context, list(device_ids)[i:i + step])
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/securitygroups_rpc.py",
line 232, in security_group_info_for_devices
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ports =
self._get_devices_info(context, devices)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/securitygroups_rpc.py",
line 300, in _get_devices_info
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ovo =
self.rcache.get_resource_by_id('Port', device)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py",
line 61, in get_resource_by_id
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
agent_restarted=agent_restarted)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py",
line 79, in _flood_cache_for_query
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
filter_kwargs=filter_kwargs)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_log/helpers.py",
line 67, in wrapper
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
method(*args, **kwargs)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 114, in bulk_pull
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron_lib/rpc.py",
line 157, in call
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
self._original_context.call(ctxt, method, **kwargs)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
line 178, in call
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
retry=self.retry)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
line 128, in _send
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 645, in send
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout, retry=retry)
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 636, in _send
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise result
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RemoteError:
Remote error: InvalidTargetVersion Invalid target version 1.5
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [u'Traceback
(most recent call last):\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 166, in _process_incoming\n res =
self.dispatcher.dispatch(message)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 229, in inner\n return func(*args, **kwargs)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 148, in bulk_pull\n **filter_kwargs)]\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
line 551, in obj_to_primitive\n raise
exception.InvalidTargetVersion(version=target_version)\n',
u'InvalidTargetVersion: Invalid target version 1.5\n'].
2019-11-26 08:36:45.975 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
AND
Failed to get details for device 227d33c9-a8c5-45aa-bd98-8bc3ff2f6029:
RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5
[u'Traceback (most recent call last):\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 166, in _process_incoming\n res =
self.dispatcher.dispatch(message)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 229, in inner\n return func(*args, **kwargs)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 148, in bulk_pull\n **filter_kwargs)]\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
line 551, in obj_to_primitive\n raise
exception.InvalidTargetVersion(version=target_version)\n',
u'InvalidTargetVersion: Invalid target version 1.5\n'].
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc Traceback (most recent call
last):
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/rpc.py",
line 303, in get_devices_details_list_and_failed_devices
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc agent_restarted))
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/rpc.py",
line 312, in get_device_details
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc resources.PORT, device,
agent_restarted)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py",
line 61, in get_resource_by_id
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc
agent_restarted=agent_restarted)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py",
line 79, in _flood_cache_for_query
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc
filter_kwargs=filter_kwargs)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_log/helpers.py",
line 67, in wrapper
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc return method(*args,
**kwargs)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 114, in bulk_pull
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc
version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron_lib/rpc.py",
line 157, in call
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc return
self._original_context.call(ctxt, method, **kwargs)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
line 178, in call
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc retry=self.retry)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
line 128, in _send
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc retry=retry)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 645, in send
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc call_monitor_timeout,
retry=retry)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 636, in _send
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc raise result
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc RemoteError: Remote error:
InvalidTargetVersion Invalid target version 1.5
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc [u'Traceback (most recent
call last):\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 166, in _process_incoming\n res =
self.dispatcher.dispatch(message)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 229, in inner\n return func(*args, **kwargs)\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py",
line 148, in bulk_pull\n **filter_kwargs)]\n', u' File
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
line 551, in obj_to_primitive\n raise
exception.InvalidTargetVersion(version=target_version)\n',
u'InvalidTargetVersion: Invalid target version 1.5\n'].
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc
AND
Clearing cache of registered ports, retries to resync were > 5:
RemoteError: Remote error: InvalidTargetVersion Invalid target version
1.5
I have switched back to 14.0.2 (5ish weeks old) and things are back on
track in the environment.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1854050/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
