Reviewed: https://review.opendev.org/c/openstack/neutron/+/807335 Committed: https://opendev.org/openstack/neutron/commit/e610a5eb9e71aa2549fb11e2139370d227787da2 Submitter: "Zuul (22348)" Branch: master
commit e610a5eb9e71aa2549fb11e2139370d227787da2 Author: Slawek Kaplonski <[email protected]> Date: Fri Sep 3 16:04:02 2021 +0200 Don't use singleton in routes.middleware.RoutesMiddleware It seems that using default singleton=True in the routes.middleware.RoutesMiddleware which is leading to use thread-local RequestConfig singleton object is not working well with eventlet monkeypatching of threading library which we are doing in Neutron. As a result it leaks memory in neutron-api workers every time when API request to not existing API endpoint is made by user. To avoid that memory leak, let's use singletone=False in that RoutesMiddleware object, at least until problem with thread-local singleton and eventlet monkey patching will be solved. Closes-Bug: #1942179 Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1942179 Title: neutron api worker leaks memory when processing requests to not existing controllers Status in neutron: Fix Released Status in OpenStack Security Advisory: Confirmed Bug description: Authorized cloud user may do API requests to neutron to not existing endpoints, like e.g.: curl -g -i -X GET http://10.120.0.30:9696/v2.0/blabla -H "Accept: application/json" -H "User-Agent: openstacksdk/0.59.0 keystoneauth1/4.3.1 python-requests/2.26.0 CPython/3.6.8" -H "X-Auth- Token: $token" and each such request will increase memory consumption of the neutron- api worker process. What I did was: * start neutron server with just one api worker (easier to calculate memory consumption but it would be the same leak in case of more workers too). Memory consumption was: sudo pmap 212436 | tail -n 1 total 183736K * now run command like: $ i=1; while [ $i -lt 2000 ]; do echo "Request $i"; curl -g -i -X GET http://10.120.0.30:9696/v2.0/blabla -H "Accept: application/json" -H "User-Agent: openstacksdk/0.59.0 keystoneauth1/4.3.1 python- requests/2.26.0 CPython/3.6.8" -H "X-Auth-Token: $token" 2>1 >/dev/null; i=$(( i+1 )); sleep 0.01; done * check memory consumption of the same api worker now: sudo pmap 212436 | tail -n 1 total 457896K To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1942179/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
