I'm changing the project to neutron as it seems the problem is within neutron tempest plugin, not tempest.
** Project changed: tempest => neutron -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1881311 Title: Neutron Tempest Pagination test cases fail if run in parallel Status in neutron: New Bug description: Neutron tempest Pagination tet cases fail if run in parallel. ##### Issue The pagination test cases see items created by other test cases and hence expected vs actual differ and the test cases fail. ##### Proposed solution: 1. Update pagination test cases to query neutron resource only for a specific project. OR 2. Check the project ID in expected test data and use that Project ID to match the results in Actual. Ignore any other Project ID Open to further discussion or any other solution. For Eg: ###### These two test cases fail: Test case 1: neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68] ```code Traceback neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68] Traceback: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 353, in test_list_pagination_page_reverse_with_href_links self._test_list_pagination_page_reverse_with_href_links() File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1346, in _test_list_pagination_page_reverse_with_href_links self.assertSameOrder(expected_resources, reversed(resources)) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1160, in assertSameOrder self.assertEqual(len(original), len(actual)) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 5 != 6 ``` Test case 2: neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links [id-dcd02a7a-f07e-4d5e-b0ca-b58e48927a9b] ```code Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 337, in test_list_pagination_with_href_links self._test_list_pagination_with_href_links() File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1312, in _test_list_pagination_with_href_links self._test_list_pagination_iteratively(self._list_all_with_hrefs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1241, in _test_list_pagination_iteratively len(expected_resources), sort_args File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1302, in _list_all_with_hrefs self.assertEqual(1, len(resources_)) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 1 != 0 ``` ###### Reason for failure: More neutron trunk ports are being returned than expected. Code has to be fixed for 2 and 3 below such that trunks are returned only for one project (specify project_id in GET trunks call). A parallel test might be creating trunks so this test case is failing. 2. test_list_pagination_page_reverse_with_href_links -> Expected returns trunks for project: 864acee2d6c64faa8750cfe53437a158 -> Actual paginated returns trunk for more projects: 89e63227c3b6405498f8fb1973cd055d and 864acee2d6c64faa8750cfe53437a158 3. test_list_pagination_with_href_links Same issue as in 2. ###### Able to run the tests if run in serial mode: concurrency = 1 {0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [3.404399s] ... ok{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_href_links [9.412321s] ... ok {0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_marker [4.566547s] ... ok{0} neutron_tempest_plugin.api.test_ports.PortsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [4.636547s] ... ok {0} neutron_tempest_plugin.api.test_subnets.SubnetsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [1.971421s] ... ok{0} neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [0.881594s] ... ok {0} neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links [2.173380s] ... ok{0} neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_asc [0.180619s] ... ok {0} neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_desc [0.176105s] ... ok ====== Totals ====== Ran: 9 tests in 107.7512 sec. Passed: 9 Skipped: 0 Expected Fail: 0 Unexpected Success: 0 Failed: 0 Sum of execute time for each test: 27.4029 sec. ============== Worker Balance ============== Worker 0 (9 tests) => 0:01:21.001099 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1881311/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
