Reviewed: https://review.openstack.org/317616 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=70ea188f5d87c45fb60ace8b8405274e5f6dd489 Submitter: Jenkins Branch: master
commit 70ea188f5d87c45fb60ace8b8405274e5f6dd489 Author: venkata anil <[email protected]> Date: Tue May 17 16:30:13 2016 +0000 New option for num_threads for state change server Currently max number of client connections(i.e greenlets spawned at a time) opened at any time by the WSGI server is set to 100 with wsgi_default_pool_size[1]. This configuration may be fine for neutron api server. But with wsgi_default_pool_size(=100) requests, state change server is creating heavy cpu load on agent. So this server(which run on agents) need lesser value i.e can be configured to half the number of cpu on agent We use "ha_keepalived_state_change_server_threads" config option to configure number of threads in state change server instead of wsgi_default_pool_size. [1] https://review.openstack.org/#/c/278007/ DocImpact: Add new config option - ha_keepalived_state_change_server_threads, to configure number of threads in state change server. Closes-Bug: #1581580 Change-Id: I822ea3844792a7731fd24419b7e90e5aef141993 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1581580 Title: Heavy cpu load seen when keepalived state change server gets wsgi_default_pool_size requests at same time Status in neutron: Fix Released Bug description: With wsgi_default_pool_size=100[1], if the keepalived state change server gets 100 requests at the same time, while processing the requests heavy load is seen on cpu, making the network node unresponsive. For each request, keepalived state change server spawns a new meta data proxy process(i.e neutron-ns-metadata-proxy). During heavy cpu load, with "top" command, I can see many metadata proxy processes in "running" state at same time(see the attachment). When wsgi_default_pool_size=8, I see state change server spawning 8 metadata proxy processes at a time("top" command shows 8 meta data proxy processes in "running" state at a time), cpu load is less and metadata proxy processes(for example, 100) spawned for all requests without failures(as backlog=KEEPALIVED_STATE_CHANGE_SERVER_BACKLOG i.e 4096). We can keep wsgi_default_pool_size=100 for neutron API server, and use seperate configuration for UnixDomainWSGIServer(for example CONF.unix_domain_wsgi_default_pool_size). neutron/agent/linux/utils.py class UnixDomainWSGIServer(wsgi.Server): def _run(self, application, socket): """Start a WSGI service in a new green thread.""" logger = logging.getLogger('eventlet.wsgi.server') eventlet.wsgi.server(socket, application, max_size=CONF.unix_domain_wsgi_default_pool_size, protocol=UnixDomainHttpProtocol, log=logger) [1] https://github.com/openstack/neutron/commit/9d573387f1e33ce85269d3ed9be501717eed4807 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1581580/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
