Reviewed: https://review.openstack.org/278501 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a5a7b892df46f5d48840f600f03e7be2db999a1d Submitter: Jenkins Branch: master
commit a5a7b892df46f5d48840f600f03e7be2db999a1d Author: Jakub Libosvar <[email protected]> Date: Wed Feb 10 16:35:30 2016 +0000 fullstack: Gracefully stop neutron-server process There is possible scenario that neutron-server loses connection to AMQP bus and that can cause timeouts when hanging on rpc calls. On the other hand, we should be able to stop service gracefully even if AMQP server isn't running. During teardown there were still neutron-server orphaned api workers processes running that had open connection to database which caused unexpected failures. Note that this change applies only for neutron-server because previous investigations showed agents using oslo service can hang on rpc causing other failures. Next step should be making sure rabbitmq is stopped as last or decrease rpc timeouts in agents. Change-Id: I3d793ba924dc3fd229d8588f4be1e943614f22bb Related-bug: 1494363 Closes-bug: 1541742 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1541742 Title: fullstack tests break when tearing down database Status in neutron: Fix Released Bug description: http://logs.openstack.org/41/265041/7/check/gate-neutron-dsvm- fullstack/8ac64cd/testr_results.html.gz Late runs fail with the same errors: Traceback (most recent call last): File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/fixtures/fixture.py", line 125, in cleanUp return self._cleanups(raise_errors=raise_first) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/fixtures/callmany.py", line 88, in __call__ reraise(error[0], error[1], error[2]) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/fixtures/callmany.py", line 82, in __call__ cleanup(*args, **kwargs) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/testresources/__init__.py", line 797, in tearDownResources resource[1].finishedWith(getattr(test, resource[0]), result) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/testresources/__init__.py", line 509, in finishedWith self._clean_all(resource, result) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/testresources/__init__.py", line 478, in _clean_all self.clean(resource) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/provision.py", line 127, in clean resource.database.engine) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/provision.py", line 263, in drop_all_objects self.impl.drop_all_objects(engine) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/provision.py", line 415, in drop_all_objects conn.execute(schema.DropConstraint(fkc)) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute return meth(self, multiparams, params) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection return connection._execute_ddl(self, multiparams, params) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 968, in _execute_ddl compiled File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context context) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception util.raise_from_cause(newraise, exc_info) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context context) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute cursor.execute(statement, parameters) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/cursors.py", line 146, in execute result = self._query(query) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/cursors.py", line 296, in _query conn.query(q) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/connections.py", line 819, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/connections.py", line 1001, in _read_query_result result.read() File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/connections.py", line 1285, in read first_packet = self.connection._read_packet() File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/connections.py", line 945, in _read_packet packet_header = self._read_bytes(4) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/connections.py", line 971, in _read_bytes data = self._rfile.read(num_bytes) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/_socketio.py", line 59, in readinto return self._sock.recv_into(b) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 353, in recv_into return self._recv_loop(self.fd.recv_into, buffer, nbytes, flags) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 341, in _recv_loop self._read_trampoline() File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 312, in _read_trampoline timeout_exc=socket.timeout("timed out")) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 201, in _trampoline mark_as_closed=self._mark_as_closed) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 162, in trampoline return hub.switch() File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch return self.greenlet.switch() File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in run self.wait(sleep_time) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 85, in wait presult = self.do_poll(seconds) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll return self.poll.poll(seconds) File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler raise TimeoutException() IndexError: tuple index out of range To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1541742/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
