Public bug reported: This started recently (last backport successfully merged was on July 20th), functional tests now fail 100% on recent backports with TIMED_OUT. For example: https://review.opendev.org/c/openstack/neutron/+/801882 https://review.opendev.org/c/openstack/neutron/+/802528
I confirmed it with dummy change: https://review.opendev.org/c/openstack/neutron/+/802552 Many tests fail with: 2021-07-27 17:34:20.939074 | controller | File "/usr/lib/python3.6/threading.py", line 295, in wait 2021-07-27 17:34:20.939093 | controller | waiter.acquire() 2021-07-27 17:34:20.939107 | controller | 2021-07-27 17:34:20.939120 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/semaphore.py", line 115, in acquire 2021-07-27 17:34:20.939134 | controller | hubs.get_hub().switch() 2021-07-27 17:34:20.939147 | controller | 2021-07-27 17:34:20.939161 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch 2021-07-27 17:34:20.939177 | controller | return self.greenlet.switch() 2021-07-27 17:34:20.939191 | controller | 2021-07-27 17:34:20.939205 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 350, in run 2021-07-27 17:34:20.939219 | controller | self.wait(sleep_time) 2021-07-27 17:34:20.939233 | controller | 2021-07-27 17:34:20.939246 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 80, in wait 2021-07-27 17:34:20.939260 | controller | presult = self.do_poll(seconds) 2021-07-27 17:34:20.939316 | controller | 2021-07-27 17:34:20.939338 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll 2021-07-27 17:34:20.939352 | controller | return self.poll.poll(seconds) 2021-07-27 17:34:20.939366 | controller | 2021-07-27 17:34:20.939380 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler 2021-07-27 17:34:20.939394 | controller | raise TimeoutException() 2021-07-27 17:34:20.939407 | controller | 2021-07-27 17:34:20.939421 | controller | fixtures._fixtures.timeout.TimeoutException Start of backtrace depends on tests (namespace creation, ip_lib, ...) so it does look like generic issue in related package Note this is specific to stable/ussuri, the first backport mentioned passed in newer branches and in stable/train without issue. Functional tests are passing in train with same OS, same python version 3.6 Nothing suspicious logged in the tests functional output itself ** Affects: neutron Importance: Undecided Status: New ** Tags: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1938262 Title: [stable/ussuri] Functional jobs timeout, many tests with fixtures._fixtures.timeout.TimeoutException Status in neutron: New Bug description: This started recently (last backport successfully merged was on July 20th), functional tests now fail 100% on recent backports with TIMED_OUT. For example: https://review.opendev.org/c/openstack/neutron/+/801882 https://review.opendev.org/c/openstack/neutron/+/802528 I confirmed it with dummy change: https://review.opendev.org/c/openstack/neutron/+/802552 Many tests fail with: 2021-07-27 17:34:20.939074 | controller | File "/usr/lib/python3.6/threading.py", line 295, in wait 2021-07-27 17:34:20.939093 | controller | waiter.acquire() 2021-07-27 17:34:20.939107 | controller | 2021-07-27 17:34:20.939120 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/semaphore.py", line 115, in acquire 2021-07-27 17:34:20.939134 | controller | hubs.get_hub().switch() 2021-07-27 17:34:20.939147 | controller | 2021-07-27 17:34:20.939161 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch 2021-07-27 17:34:20.939177 | controller | return self.greenlet.switch() 2021-07-27 17:34:20.939191 | controller | 2021-07-27 17:34:20.939205 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 350, in run 2021-07-27 17:34:20.939219 | controller | self.wait(sleep_time) 2021-07-27 17:34:20.939233 | controller | 2021-07-27 17:34:20.939246 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 80, in wait 2021-07-27 17:34:20.939260 | controller | presult = self.do_poll(seconds) 2021-07-27 17:34:20.939316 | controller | 2021-07-27 17:34:20.939338 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll 2021-07-27 17:34:20.939352 | controller | return self.poll.poll(seconds) 2021-07-27 17:34:20.939366 | controller | 2021-07-27 17:34:20.939380 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler 2021-07-27 17:34:20.939394 | controller | raise TimeoutException() 2021-07-27 17:34:20.939407 | controller | 2021-07-27 17:34:20.939421 | controller | fixtures._fixtures.timeout.TimeoutException Start of backtrace depends on tests (namespace creation, ip_lib, ...) so it does look like generic issue in related package Note this is specific to stable/ussuri, the first backport mentioned passed in newer branches and in stable/train without issue. Functional tests are passing in train with same OS, same python version 3.6 Nothing suspicious logged in the tests functional output itself To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1938262/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
