Hello Jonathan, Have you seen this discussion on the openstack-dev [1] that discusses the bug 1361357 [2]? I have no idea if it is related to your issue but FWIW a fix backport in the latest Icehouse release introduced performance regression for the metadata agent. BR Simon
[1] http://lists.openstack.org/pipermail/openstack-dev/2014-October/048916.html [2] https://bugs.launchpad.net/cloud-archive/+bug/1361357 On Wed, Oct 22, 2014 at 2:33 AM, Jonathan Proulx <[email protected]> wrote: > Ah there's the log many instances of: > > 2014-10-21 19:50:15.527 12931 INFO neutron.wsgi [-] 10.10.167.98 - - > [21/Oct/2014 19:50:15] "GET /openstack/2012-08-10 HTTP/1.1" 500 343 > 120.411705 > > 2014-10-21 19:50:15.528 12931 ERROR > neutron.agent.metadata.namespace_proxy [-] Unexpected error. > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy Traceback (most recent call > last): > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy File > > "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", > line 74, in __cal > l__ > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy req.body) > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy File > > "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", > line 105, in _pro > xy_request > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy > connection_type=UnixDomainHTTPConnection) > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy File > "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1569, in > request > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy (response, content) = > self._request(conn, authority, uri, request_uri, method, body, > headers, redi > rections, cachekey) > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy File > "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1316, in > _request > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy (response, content) = > self._conn_request(conn, request_uri, method, body, headers) > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy File > "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1285, in > _conn_request > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy response = > conn.getresponse() > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy File > "/usr/lib/python2.7/httplib.py", line 1045, in getresponse > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy response.begin() > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy File > "/usr/lib/python2.7/httplib.py", line 409, in begin > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy version, status, reason = > self._read_status() > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy File > "/usr/lib/python2.7/httplib.py", line 373, in _read_status > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy raise BadStatusLine(line) > 2014-10-21 19:50:15.528 12931 TRACE > neutron.agent.metadata.namespace_proxy BadStatusLine: '' > 2014-10-21 19:50:15.528 12931 TRACE neutron.agent.metadata.namespace_proxy > > On Tue, Oct 21, 2014 at 8:17 PM, Jonathan Proulx <[email protected]> > wrote: > > running Icehouse + Neutron ML2/OVS and network names spaces. > > > > Was running well unitl recently, most recent change was switching to > > Ceph RBD for ephemeral storage on the hypervisors (and glance). I > > suspect this of being relevant because it makes the instances launch > > much more quickly. > > > > I haven't classified the breaking point but launching 64 instances > > deterministically breaks the metadata agent. > > > > The service seems to be running on the controller, but is not > > listening in the network namespace. It seems to require restarting > > both the dhcp-agent and the metadata agent to get it to go again. > > > > Even in debug mode I get no errors in the logs. > > > > Anyone seen this? > > > > -Jon > > _______________________________________________ > OpenStack-operators mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >
_______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
