** Also affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1624494
Title:
[master] metadata is not working on multi-node setup
Status in devstack:
New
Status in neutron:
New
Bug description:
Setup:
1. One controller
2. KVM and ESX nova compute
3. Two Network nodes running q-dhcp and q-meta
Nodes: Ubuntu 14.04.3 amd64 Trusty
On above setup, tried to run few metadata queries we see 500 internal
server error.
$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 1.1.107.1 0.0.0.0 UG 0 0 0 eth0
1.1.107.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.169.254 1.1.107.1 255.255.255.255 UGH 0 0 0 eth0
$ curl http://169.254.169.254
500 Internal Server Error
Remote metadata server experienced an internal server error.
$
NOTE: Moved all services to single node(controller) and metadata
queries seems to work fine.
vmware@cntr11:~$ neutron agent-list
+--------------------------------------+----------------+-----------+-------------------+-------+----------------+------------------------+
| id | agent_type | host |
availability_zone | alive | admin_state_up | binary |
+--------------------------------------+----------------+-----------+-------------------+-------+----------------+------------------------+
| 0168206e-6c13-40df-a0b8-7772220ab9cd | DHCP agent | network-1 | nova
| :-) | True | neutron-dhcp-agent |
| 08c8bb6e-0c7f-41e3-b134-81e2d5334aea | Metadata agent | network-1 |
| :-) | True | neutron-metadata-agent |
| 25e9fa6d-6acc-4b8e-bb0b-f2d3ac20981d | Metadata agent | network-2 |
| :-) | True | neutron-metadata-agent |
| 534d349c-8830-4648-814b-611a30f59287 | DHCP agent | network-2 | nova
| :-) | True | neutron-dhcp-agent |
+--------------------------------------+----------------+-----------+-------------------+-------+----------------+------------------------+
vmware@cntr11:~$
vmware@cntr11:~$ nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State |
Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 7 | nova-conductor | cntr11 | internal | enabled | up |
2016-09-16T10:00:22.000000 | - |
| 9 | nova-compute | esx-ubuntu-01 | nova | enabled | up |
2016-09-16T10:00:14.000000 | - |
| 10 | nova-compute | kvm-3 | nova | enabled | up |
2016-09-16T10:00:23.000000 | - |
| 11 | nova-compute | kvm-2 | nova | enabled | up |
2016-09-16T10:00:19.000000 | - |
| 12 | nova-compute | kvm-1 | nova | enabled | up |
2016-09-16T10:00:19.000000 | - |
| 13 | nova-scheduler | cntr11 | internal | enabled | up |
2016-09-16T10:00:15.000000 | - |
| 14 | nova-consoleauth | cntr11 | internal | enabled | up |
2016-09-16T10:00:20.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
vmware@cntr11:~$
Logs:
2016-09-13 13:31:50.713 14309 DEBUG eventlet.wsgi.server [-] (14309) accepted
'' server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
2016-09-13 13:31:50.715 14309 DEBUG neutron.agent.metadata.agent [-] Request:
GET / HTTP/1.0^M
Accept-Encoding: gzip, deflate^M
Content-Length: 0^M
Content-Type: text/plain^M
Host: 169.254.169.254^M
User-Agent: Python-httplib2/0.9.2 (gzip)^M
X-Forwarded-For: 1.1.107.3^M
X-Neutron-Router-Id: bbe453a5-db77-4cd9-af02-31232a222f16 __call__
/opt/stack/neutron/neutron/agent/metadata/agent.py:86
2016-09-13 13:31:50.716 14309 DEBUG oslo_messaging._drivers.amqpdriver [-]
CALL msg_id: 50b283bdeab945cb93872300abd5b47c exchange 'neutron' topic
'q-plugin' _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-09-13 13:31:50.807 14309 DEBUG oslo_messaging._drivers.amqpdriver [-]
received reply msg_id: 50b283bdeab945cb93872300abd5b47c __call__
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-09-13 13:31:50.808 14309 DEBUG oslo_messaging._drivers.amqpdriver [-]
CALL msg_id: 52646ead6313405fb688b80cbb4bfc73 exchange 'neutron' topic
'q-plugin' _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-09-13 13:31:50.846 14309 DEBUG oslo_messaging._drivers.amqpdriver [-]
received reply msg_id: 52646ead6313405fb688b80cbb4bfc73 __call__
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent [-]
Unexpected error.
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent Traceback
(most recent call last):
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File
"/opt/stack/neutron/neutron/agent/metadata/agent.py", line 90, in __call__
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent return
self._proxy_request(instance_id, tenant_id, req)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File
"/opt/stack/neutron/neutron/agent/metadata/agent.py", line 193, in
_proxy_request
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent
body=req.body)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File
"/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in
request
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent
(response, content) = self._request(conn, authority, uri, request_uri, method,
body, headers, redirections, cachekey)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File
"/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in
_request
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent
(response, content) = self._conn_request(conn, request_uri, method, body,
headers)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File
"/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1307, in
_conn_request
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent response
= conn.getresponse()
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File
"/usr/lib/python2.7/httplib.py", line 1129, in getresponse
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent response
= self.response_class(*args, **kwds)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File
"/usr/lib/python2.7/httplib.py", line 390, in __init__
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent self.fp
= sock.makefile('rb', 0)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File
"/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 298, in
makefile
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent dupped =
self.dup()
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File
"/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 289, in
dup
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent newsock
= type(self)(sock, set_nonblocking=False)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent TypeError:
__init__() got an unexpected keyword argument 'set_nonblocking'
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent
2016-09-13 13:31:50.849 14309 INFO eventlet.wsgi.server [-] 1.1.107.3,<local>
- - [13/Sep/2016 13:31:50] "GET / HTTP/1.1" 500 229 0.134582
2016-09-13 13:31:51.854 13766 DEBUG oslo_messaging._drivers.amqpdriver [-]
CAST unique_id: 757d1802bce344a18d2ae1df8221058f exchange 'neutron' topic
'q-reports-plugin' _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1624494/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
