Reviewed:  https://review.openstack.org/561047
Committed: 
https://git.openstack.org/cgit/openstack/neutron/commit/?id=544597c6ef9fb297693dbeb0f2d7dc22f3a1b25d
Submitter: Zuul
Branch:    master

commit 544597c6ef9fb297693dbeb0f2d7dc22f3a1b25d
Author: Ihar Hrachyshka <ihrac...@redhat.com>
Date:   Thu Apr 12 20:30:20 2018 +0000

    ovs: survive errors from check_ovs_status
    
    Instead of allowing an error to bubble up and exit from rpc_loop, catch
    it and assume the switch is dead which will make the agent to wait until
    the switch is back without failing the service.
    
    Change-Id: Ic3095dd42b386f56b1f75ebb6a125606f295551b
    Closes-Bug: #1731494


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1731494

Title:
  neutron-openvswitch-agent crashes due to TypeError exception in
  ovs_ryuapp

Status in neutron:
  Fix Released

Bug description:
  At some point during some rally test, we saw this exception in ovs
  agent logs:

  2017-11-07 13:35:51.428 597682 DEBUG 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
[req-62f85bb3-db4c-4485-b35c-b7c1cafb3970 3d527bdd3ede4c6a97f91b701393b8e3 
5f753e92a5d740fc97252bd39f868561 - - -] port_delete message processed for port 
3e8348d0-40e1-4146-b803-1e6c6eddba53 port_delete 
/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:430
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp 
[req-141ecd16-22d7-4b1c-aa91-25d5077414f5 - - - - -] Agent main thread died of 
an exception: TypeError: int() can't convert non-string with explicit base
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp 
Traceback (most recent call last):
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py",
 line 40, in agent_main_wrapper
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     
ovs_agent.main(bridge_classes)
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 2205, in main
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     
agent.daemon_loop()
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File 
"/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     
return f(*args, **kwargs)
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 2120, in daemon_loop
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     
self.rpc_loop(polling_manager=pm)
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File 
"/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     
return f(*args, **kwargs)
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 1985, in rpc_loop
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     
ovs_status = self.check_ovs_status()
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File 
"/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     
return f(*args, **kwargs)
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 1787, in check_ovs_status
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     
status = self.int_br.check_canary_table()
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py",
 line 52, in check_canary_table
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     
flows = self.dump_flows(constants.CANARY_TABLE)
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py",
 line 141, in dump_flows
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     
(dp, ofp, ofpp) = self._get_dp()
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_bridge.py",
 line 68, in _get_dp
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     
self._cached_dpid = int(new_dpid_str, 16)
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp 
TypeError: int() can't convert non-string with explicit base
  2017-11-07 13:35:51.439 597682 ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp
  2017-11-07 13:35:54.861 597682 WARNING ovsdbapp.backend.ovs_idl.vlog [-] 
tcp:127.0.0.1:6640: receive error: Connection reset by peer: RuntimeError: OVS 
transaction timed out

  
  This makes the agent crash and when restarted, perform a full sync which 
slows things down a lot.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1731494/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

Reply via email to