Reviewed: https://review.openstack.org/610034 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=00d08a3288f61268f8a823991fadf0d83c36c04d Submitter: Zuul Branch: master
commit 00d08a3288f61268f8a823991fadf0d83c36c04d Author: Sean Mooney <w...@seanmooney.info> Date: Fri Oct 12 14:26:14 2018 +0100 Harden placement init under wsgi - This change tries to address an edge case discovered when running placement under mod_wsgi where if the placement wsgi application is re-initialize the db_api configure method attempts to reconfigure a started transaction factory. - This since oslo.db transaction factories do not support reconfiguration at runtime this result in an exception being raised preventing reloading of the Placement API without restarting apache to force mod_wsgi to recreate the python interpreter. - This change introduces a run once decorator to allow annotating functions that should only be executed once for the lifetime fo an interpreter. - This change applies the run_once decorator to the db_api configure method, to suppress the attempt to reconfigure the current TransactionFactory on application reload. Co-Authored-By: Balazs Gibizer <balazs.gibi...@ericsson.com> Closes-Bug: #1799246 Related-Bug: #1784155 Change-Id: I704196711d30c1124e713ac31111a8ea6fa2f1ba ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1799246 Title: module level init of db transation contexts cause failure under mod_wsgi on module reload Status in OpenStack Compute (nova): Fix Released Bug description: Description =========== This is related to a downstream bug first reported here:https://bugzilla.redhat.com/show_bug.cgi?id=1630069 where in it was discovers that due to how triplo currently deploys the placemetn api under mod_wsgi when deployed under mod_wsgi if the wsgi application exits with an error it is reloaded back into the same python interperter instance. As a result of this behavior module level variable have a longer lifetime then the lifetime of the application. when run under uwsgi when the application is reloaded it is reloaded in a new python interperter meaning the lifetime of the module level variables is scoped to the life time of the application. As a result of the life time semantics of mod_wsgi the placment api and nova-api must assume that the wsgi applications init can be invoked multiple times on failure. The current use of the sqlalcamey enginefacade transaction_context is not reentrant resulting in a type error being raised on subsequet calls to configure when the wsgi application is reloaded. Expected result =============== it should be possible to reload the nova and placement api wsgi application under mod_wsgi on failure Actual result ============= 46087 [Wed Oct 10 15:10:49.433284 2018] [:error] [pid 14] [remote 172.25.0.10:208] mod_wsgi (pid=14): Target WSGI script '/var/www/cgi-bin/nova/nova-placement-api' cannot be loaded as Python module. 46088 [Wed Oct 10 15:10:49.433305 2018] [:error] [pid 14] [remote 172.25.0.10:208] mod_wsgi (pid=14): Exception occurred processing WSGI script '/var/www/cgi-bin/nova/nova-placement-api'. 46089 [Wed Oct 10 15:10:49.433320 2018] [:error] [pid 14] [remote 172.25.0.10:208] Traceback (most recent call last): 46090 [Wed Oct 10 15:10:49.433333 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/var/www/cgi-bin/nova/nova-placement-api", line 54, in <module> 46091 [Wed Oct 10 15:10:49.433354 2018] [:error] [pid 14] [remote 172.25.0.10:208] application = init_application() 46092 [Wed Oct 10 15:10:49.433361 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/wsgi.py", line 108, in init_application 46093 [Wed Oct 10 15:10:49.433386 2018] [:error] [pid 14] [remote 172.25.0.10:208] db_api.configure(conf.CONF) 46094 [Wed Oct 10 15:10:49.433392 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/db_api.py", line 35, in configure 46095 [Wed Oct 10 15:10:49.433403 2018] [:error] [pid 14] [remote 172.25.0.10:208] **_get_db_conf(conf.placement_database)) 46096 [Wed Oct 10 15:10:49.433408 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 788, in configure 46097 [Wed Oct 10 15:10:49.433420 2018] [:error] [pid 14] [remote 172.25.0.10:208] self._factory.configure(**kw) 46098 [Wed Oct 10 15:10:49.433425 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator 46099 [Wed Oct 10 15:10:49.433435 2018] [:error] [pid 14] [remote 172.25.0.10:208] return wrapped(*args, **kwargs) 46100 [Wed Oct 10 15:10:49.433440 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 312, in configure 46101 [Wed Oct 10 15:10:49.433449 2018] [:error] [pid 14] [remote 172.25.0.10:208] self._configure(False, kw) 46102 [Wed Oct 10 15:10:49.433453 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 317, in _configure 46103 [Wed Oct 10 15:10:49.433462 2018] [:error] [pid 14] [remote 172.25.0.10:208] raise TypeError("this TransactionFactory is already started") 46104 [Wed Oct 10 15:10:49.433473 2018] [:error] [pid 14] [remote 172.25.0.10:208] TypeError: this TransactionFactory is already started To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1799246/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp