** Changed in: neutron/kilo
Status: New => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1439472
Title:
OVS doesn't restart properly when Exception occurred
Status in neutron:
Fix Released
Status in neutron kilo series:
Fix Released
Bug description:
Wish this fix can be fixed into kilo. If it's not able due to the bad timing,
wish this fix can be merged into stable/kilo.
-------------------
[The problem]
If there is an Exception (such as DBConnectionError) occurred/occurring when
OVS restart,
OVS will return "every thing is OK :-)",
while flow of created network in br-tun will NOT be recovered. :-(
Unless user operation of OVS restart has been executed.
-------------------
[action and log]
[q-agent.log]
[[[create network and subnet and add it to DHCP agent]]]
[[[I turned off MySQL]]]
[[[But nothing happened]]]
[[[Then I restarted OVS]]]
[[[Here it goes...]]]
...
...
...
2015-04-01 22:06:48.237 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Unable to sync tunnel IP
192.168.122.96: Remote error: DBConnectionError (OperationalError) (2003,
"Can't connect to MySQL server on '127.0.0.1' (111)") None None
...
...
...
2015-04-01 22:06:56.060 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent tunnel out of sync
with plugin!
2015-04-01 22:06:56.061 DEBUG oslo_messaging._drivers.amqpdriver
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is
705639bc86ae44f4b4cc28715ce981e8 _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:06:56.062 DEBUG oslo_messaging._drivers.amqp
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is
41f997f166c04cff986ff08eb298b3eb. _add_unique_id
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:06:56.085 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Unable to sync tunnel IP
192.168.122.96: Remote error: DBConnectionError (OperationalError) (2003,
"Can't connect to MySQL server on '127.0.0.1' (111)") None None
...
...
...
2015-04-01 22:06:56.111 DEBUG oslo_messaging._drivers.amqpdriver
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is
6f012243c4844978a7b8181bedcafcc9 _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:06:56.112 DEBUG oslo_messaging._drivers.amqp
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is
04ed1ccb78bb4ab495b9ebf40c2338f5. _add_unique_id
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:06:56.138 ERROR
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Error while processing VIF
ports
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call
last):
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent File
"/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py",
line 1522, in rpc_loop
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent ovs_restarted)
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent File
"/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py",
line 1260, in process_network_ports
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
port_info.get('updated', set()))
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent File
"/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 360, in
setup_port_filters
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.prepare_devices_filter(new_devices)
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent File
"/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 219, in
decorated_function
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent File
"/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 229, in
prepare_devices_filter
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent self.context,
list(device_ids))
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent File
"/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 116, in
security_group_info_for_devices
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent devices=devices)
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line
156, in call
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent retry=self.retry)
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90,
in _send
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent timeout=timeout,
retry=retry)
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 350, in send
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent retry=retry)
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 341, in _send
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent raise result
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent RemoteError: Remote error:
DBConnectionError (OperationalError) (2003, "Can't connect to MySQL server on
'127.0.0.1' (111)") None None
...
...
...
2015-04-01 22:06:56.138 3698 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
2015-04-01 22:06:56.139 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop -
iteration:186 completed. Processed ports statistics: {'ancillary': {'removed':
0, 'added': 0}, 'regular': {'updated': 0, 'added': 0, 'removed': 0}}.
Elapsed:0.091 loop_count_and_wait
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1411
2015-04-01 22:06:58.049 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop -
iteration:187 started rpc_loop
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1448
2015-04-01 22:06:58.049 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent out of sync with
plugin!
2015-04-01 22:06:58.050 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23']
execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:06:58.058 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23']
Exit code: 0
Stdin:
Stdout: NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=9.952s, table=23, n_packets=0, n_bytes=0, idle_age=9,
priority=0 actions=drop
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
...
...
...
------------------------------------------
[[[OK, enough. I'll just start MySQL again...]]]
...
...
...
2015-04-01 22:07:00.192 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['iptables-save', '-c']
Exit code: 0
Stdin:
Stdout: # Generated by iptables-save v1.4.21 on Wed Apr 1 22:07:00 2015
*raw
:PREROUTING ACCEPT [20290:4833947]
:OUTPUT ACCEPT [20523:5745904]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-PREROUTING - [0:0]
[19974:4765842] -A PREROUTING -j neutron-openvswi-PREROUTING
[20207:5677799] -A OUTPUT -j neutron-openvswi-OUTPUT
COMMIT
# Completed on Wed Apr 1 22:07:00 2015
# Generated by iptables-save v1.4.21 on Wed Apr 1 22:07:00 2015
*mangle
...
...
...
COMMIT
# Completed on Wed Apr 1 22:07:00 2015
Stdout:
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.221 DEBUG neutron.agent.linux.iptables_manager
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] IPTablesManager.apply
completed with success _apply_synchronized
/opt/stack/neutron/neutron/agent/linux/iptables_manager.py:468
2015-04-01 22:07:00.222 DEBUG oslo_concurrency.lockutils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Releasing file lock
"/opt/stack/data/neutron/lock/neutron-iptables" after holding it for 0.038s
release /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:227
2015-04-01 22:07:00.223 DEBUG oslo_concurrency.lockutils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Releasing semaphore
"iptables" lock
/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:404
2015-04-01 22:07:00.223 DEBUG neutron.agent.linux.iptables_manager
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Semaphore / lock released
"iptables" _apply /opt/stack/neutron/neutron/agent/linux/iptables_manager.py:414
2015-04-01 22:07:00.223 DEBUG oslo_messaging._drivers.amqpdriver
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is
4140d8d1d9b14bd2b2e0e008c12278a4 _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.225 DEBUG oslo_messaging._drivers.amqp
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is
4f17d058945744dabfd7e4dffb7c1c85. _add_unique_id
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.414 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Processing port:
6dd3f67e-ea8c-4c26-868d-a51cfc10581c treat_devices_added_or_updated
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1148
2015-04-01 22:07:00.415 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=external_ids,name,ofport', 'find', 'Interface',
'external_ids:iface-id=6dd3f67e-ea8c-4c26-868d-a51cfc10581c',
'external_ids:attached-mac!=""'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.420 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=external_ids,name,ofport', 'find', 'Interface',
u'external_ids:iface-id=6dd3f67e-ea8c-4c26-868d-a51cfc10581c',
'external_ids:attached-mac!=""']
Exit code: 0
Stdin:
Stdout:
{"data":[[["map",[["attached-mac","fa:16:3e:c7:c9:84"],["iface-id","6dd3f67e-ea8c-4c26-868d-a51cfc10581c"],["iface-status","active"]]],"tap6dd3f67e-ea",4]],"headings":["external_ids","name","ofport"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.421 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'iface-to-br', 'tap6dd3f67e-ea'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.427 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'iface-to-br', u'tap6dd3f67e-ea']
Exit code: 0
Stdin:
Stdout: br-int
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.427 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Port
6dd3f67e-ea8c-4c26-868d-a51cfc10581c updated. Details: {u'profile': {},
u'admin_state_up': True, u'network_id':
u'dc2e0fc7-b6f3-4602-8c8e-14e1fd9865a9', u'segmentation_id': 1003,
u'device_owner': u'network:dhcp', u'physical_network': None, u'mac_address':
u'fa:16:3e:c7:c9:84', u'device': u'6dd3f67e-ea8c-4c26-868d-a51cfc10581c',
u'port_id': u'6dd3f67e-ea8c-4c26-868d-a51cfc10581c', u'fixed_ips':
[{u'subnet_id': u'c6984a53-e9f1-40c2-8504-d166981f1cc6', u'ip_address':
u'192.168.50.2'}], u'network_type': u'vxlan'}
2015-04-01 22:07:00.428 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=tag', 'list', 'Port', 'tap6dd3f67e-ea'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.434 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=tag', 'list', 'Port', u'tap6dd3f67e-ea']
Exit code: 0
Stdin:
Stdout: {"data":[[1]],"headings":["tag"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.434 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Setting status for
6dd3f67e-ea8c-4c26-868d-a51cfc10581c to UP treat_devices_added_or_updated
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1175
2015-04-01 22:07:00.435 DEBUG oslo_messaging._drivers.amqpdriver
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is
c1e45d5795b442f8933d3a5ec51e4517 _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.435 DEBUG oslo_messaging._drivers.amqp
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is
70ee9752b47b48878c59634d52b77c30. _add_unique_id
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.505 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Configuration for device
6dd3f67e-ea8c-4c26-868d-a51cfc10581c completed.
2015-04-01 22:07:00.505 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Processing port:
9d380fa6-81ca-4392-92dc-49016dff05da treat_devices_added_or_updated
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1148
2015-04-01 22:07:00.506 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=external_ids,name,ofport', 'find', 'Interface',
'external_ids:iface-id=9d380fa6-81ca-4392-92dc-49016dff05da',
'external_ids:attached-mac!=""'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.511 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=external_ids,name,ofport', 'find', 'Interface',
u'external_ids:iface-id=9d380fa6-81ca-4392-92dc-49016dff05da',
'external_ids:attached-mac!=""']
Exit code: 0
Stdin:
Stdout:
{"data":[[["map",[["attached-mac","fa:16:3e:df:1f:b8"],["iface-id","9d380fa6-81ca-4392-92dc-49016dff05da"],["iface-status","active"]]],"tap9d380fa6-81",2]],"headings":["external_ids","name","ofport"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.512 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'iface-to-br', 'tap9d380fa6-81'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.518 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'iface-to-br', u'tap9d380fa6-81']
Exit code: 0
Stdin:
Stdout: br-int
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.519 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Port
9d380fa6-81ca-4392-92dc-49016dff05da updated. Details: {u'profile': {},
u'admin_state_up': True, u'network_id':
u'233087a9-5dd9-4f95-a55e-4e42c2954530', u'segmentation_id': 1001,
u'device_owner': u'network:dhcp', u'physical_network': None, u'mac_address':
u'fa:16:3e:df:1f:b8', u'device': u'9d380fa6-81ca-4392-92dc-49016dff05da',
u'port_id': u'9d380fa6-81ca-4392-92dc-49016dff05da', u'fixed_ips':
[{u'subnet_id': u'bfa89ecb-deca-468d-8fcd-b68973dd5337', u'ip_address':
u'10.0.0.2'}], u'network_type': u'vxlan'}
2015-04-01 22:07:00.520 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=tag', 'list', 'Port', 'tap9d380fa6-81'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.527 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=tag', 'list', 'Port', u'tap9d380fa6-81']
Exit code: 0
Stdin:
Stdout: {"data":[[2]],"headings":["tag"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.527 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Setting status for
9d380fa6-81ca-4392-92dc-49016dff05da to UP treat_devices_added_or_updated
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1175
2015-04-01 22:07:00.527 DEBUG oslo_messaging._drivers.amqpdriver
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is
3d08426bf697496fab546a6604e4d802 _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.528 DEBUG oslo_messaging._drivers.amqp
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is
9cd3621f640549278d77bed8b4f874cd. _add_unique_id
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.597 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Configuration for device
9d380fa6-81ca-4392-92dc-49016dff05da completed.
2015-04-01 22:07:00.597 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Processing port:
f677b5e6-2996-49b2-acca-14fe20543cc6 treat_devices_added_or_updated
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1148
2015-04-01 22:07:00.598 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=external_ids,name,ofport', 'find', 'Interface',
'external_ids:iface-id=f677b5e6-2996-49b2-acca-14fe20543cc6',
'external_ids:attached-mac!=""'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.603 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=external_ids,name,ofport', 'find', 'Interface',
u'external_ids:iface-id=f677b5e6-2996-49b2-acca-14fe20543cc6',
'external_ids:attached-mac!=""']
Exit code: 0
Stdin:
Stdout:
{"data":[[["map",[["attached-mac","fa:16:3e:62:d4:05"],["iface-id","f677b5e6-2996-49b2-acca-14fe20543cc6"],["iface-status","active"]]],"qr-f677b5e6-29",3]],"headings":["external_ids","name","ofport"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.603 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'iface-to-br', 'qr-f677b5e6-29'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.607 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'iface-to-br', u'qr-f677b5e6-29']
Exit code: 0
Stdin:
Stdout: br-int
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.608 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Port
f677b5e6-2996-49b2-acca-14fe20543cc6 updated. Details: {u'profile': {},
u'admin_state_up': True, u'network_id':
u'233087a9-5dd9-4f95-a55e-4e42c2954530', u'segmentation_id': 1001,
u'device_owner': u'network:router_interface', u'physical_network': None,
u'mac_address': u'fa:16:3e:62:d4:05', u'device':
u'f677b5e6-2996-49b2-acca-14fe20543cc6', u'port_id':
u'f677b5e6-2996-49b2-acca-14fe20543cc6', u'fixed_ips': [{u'subnet_id':
u'bfa89ecb-deca-468d-8fcd-b68973dd5337', u'ip_address': u'10.0.0.1'}],
u'network_type': u'vxlan'}
2015-04-01 22:07:00.608 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=tag', 'list', 'Port', 'qr-f677b5e6-29'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.612 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=tag', 'list', 'Port', u'qr-f677b5e6-29']
Exit code: 0
Stdin:
Stdout: {"data":[[2]],"headings":["tag"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.612 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Setting status for
f677b5e6-2996-49b2-acca-14fe20543cc6 to UP treat_devices_added_or_updated
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1175
2015-04-01 22:07:00.612 DEBUG oslo_messaging._drivers.amqpdriver
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is
3f3cb1b8657e407da96eb7c5bf26f087 _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.613 DEBUG oslo_messaging._drivers.amqp
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is
75c68e8426674f00bb9c2b8078f2c70b. _add_unique_id
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.670 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Configuration for device
f677b5e6-2996-49b2-acca-14fe20543cc6 completed.
2015-04-01 22:07:00.671 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Processing port:
36c9baf7-a45d-4207-98b5-8155523c7a1b treat_devices_added_or_updated
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1148
2015-04-01 22:07:00.671 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=external_ids,name,ofport', 'find', 'Interface',
'external_ids:iface-id=36c9baf7-a45d-4207-98b5-8155523c7a1b',
'external_ids:attached-mac!=""'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.677 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=external_ids,name,ofport', 'find', 'Interface',
u'external_ids:iface-id=36c9baf7-a45d-4207-98b5-8155523c7a1b',
'external_ids:attached-mac!=""']
Exit code: 0
Stdin:
Stdout:
{"data":[[["map",[["attached-mac","fa:16:3e:de:6e:c1"],["iface-id","36c9baf7-a45d-4207-98b5-8155523c7a1b"],["iface-status","active"],["vm-uuid","c3a7dfd8-0871-4a7b-9a5f-1b0022ae0bf0"]]],"qvo36c9baf7-a4",5]],"headings":["external_ids","name","ofport"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.678 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'iface-to-br', 'qvo36c9baf7-a4'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.681 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'iface-to-br', u'qvo36c9baf7-a4']
Exit code: 0
Stdin:
Stdout: br-int
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.682 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Port
36c9baf7-a45d-4207-98b5-8155523c7a1b updated. Details: {u'profile': {},
u'admin_state_up': True, u'network_id':
u'dc2e0fc7-b6f3-4602-8c8e-14e1fd9865a9', u'segmentation_id': 1003,
u'device_owner': u'compute:None', u'physical_network': None, u'mac_address':
u'fa:16:3e:de:6e:c1', u'device': u'36c9baf7-a45d-4207-98b5-8155523c7a1b',
u'port_id': u'36c9baf7-a45d-4207-98b5-8155523c7a1b', u'fixed_ips':
[{u'subnet_id': u'c6984a53-e9f1-40c2-8504-d166981f1cc6', u'ip_address':
u'192.168.50.3'}], u'network_type': u'vxlan'}
2015-04-01 22:07:00.682 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=tag', 'list', 'Port', 'qvo36c9baf7-a4'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.686 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--columns=tag', 'list', 'Port', u'qvo36c9baf7-a4']
Exit code: 0
Stdin:
Stdout: {"data":[[1]],"headings":["tag"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.687 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Setting status for
36c9baf7-a45d-4207-98b5-8155523c7a1b to UP treat_devices_added_or_updated
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1175
2015-04-01 22:07:00.687 DEBUG oslo_messaging._drivers.amqpdriver
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is
6e87e064d5b34ef7b293dbcfdfa60340 _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.687 DEBUG oslo_messaging._drivers.amqp
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is
b6d420f5532d4be9ba30311656b3090e. _add_unique_id
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.764 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Configuration for device
36c9baf7-a45d-4207-98b5-8155523c7a1b completed.
2015-04-01 22:07:00.765 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] process_network_ports -
iteration:188 - treat_devices_added_or_updated completed. Skipped 0 devices of
4 devices currently available. Time elapsed: 0.541 process_network_ports
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1281
2015-04-01 22:07:00.765 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] process_network_ports -
iteration:188 - treat_devices_removed completed in 0.000 process_network_ports
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1299
2015-04-01 22:07:00.765 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop -
iteration:188 - ports processed. Elapsed:0.716 rpc_loop
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1526
2015-04-01 22:07:00.765 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'list-ports', 'br-ex'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.770 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'list-ports', u'br-ex']
Exit code: 0
Stdin:
Stdout: qg-66217dea-97
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.771 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--if-exists', '--columns=name,external_ids,ofport', 'list', 'Interface',
'qg-66217dea-97'] execute_rootwrap_daemon
/opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.777 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'--if-exists', '--columns=name,external_ids,ofport', 'list', 'Interface',
u'qg-66217dea-97']
Exit code: 0
Stdin:
Stdout:
{"data":[["qg-66217dea-97",["map",[["attached-mac","fa:16:3e:6e:00:93"],["iface-id","66217dea-974a-4f2b-bba7-7d4a85cd52b7"],["iface-status","active"]]],1]],"headings":["name","external_ids","ofport"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.777 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop -
iteration:188 - ancillary port info retrieved. Elapsed:0.728 rpc_loop
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1542
2015-04-01 22:07:00.778 DEBUG oslo_messaging._drivers.amqpdriver
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is
ed1df33c378445579c01e81312539430 _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.778 DEBUG oslo_messaging._drivers.amqp
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is
b9871a477d554b83ab9229588f9644fa. _add_unique_id
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.848 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Ancillary Port
66217dea-974a-4f2b-bba7-7d4a85cd52b7 added
2015-04-01 22:07:00.850 DEBUG oslo_messaging._drivers.amqpdriver
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is
de4643d4fed4494dbb34e465c3f26bc0 _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.851 DEBUG oslo_messaging._drivers.amqp
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is
d4231d08eedc44c8a4e05ff4a3d9ce1a. _add_unique_id
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.913 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
process_ancillary_network_ports - iteration: 188 -
treat_ancillary_devices_added completed in 0.136
process_ancillary_network_ports
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1314
2015-04-01 22:07:00.914 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
process_ancillary_network_ports - iteration: 188 -
treat_ancillary_devices_removed completed in 0.000
process_ancillary_network_ports
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1330
2015-04-01 22:07:00.914 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop -
iteration: 188 - ancillary ports processed. Elapsed:0.865 rpc_loop
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1551
2015-04-01 22:07:00.915 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop -
iteration:188 completed. Processed ports statistics: {'ancillary': {'removed':
0, 'added': 1}, 'regular': {'updated': 0, 'added': 4, 'removed': 0}}.
Elapsed:0.865 loop_count_and_wait
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1411
2015-04-01 22:07:02.050 DEBUG
neutron.plugins.openvswitch.agent.ovs_neutron_agent
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop -
iteration:189 started rpc_loop
/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1448
2015-04-01 22:07:02.051 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap
daemon): ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23']
execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:02.061 DEBUG neutron.agent.linux.utils
[req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23']
Exit code: 0
Stdin:
Stdout: NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=13.955s, table=23, n_packets=0, n_bytes=0, idle_age=13,
priority=0 actions=drop
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
...
...
...
[[[Seems like the log back to peace, again...]]]
[[[But flow of created network does not recovered...]]]
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1439472/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
