Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23842 )
Change subject: IMPALA-14668: Upgrade to pytest 6.2.5 ...................................................................... IMPALA-14668: Upgrade to pytest 6.2.5 This patch upgrades pytest version from 2.9.2 to 6.2.5, the highest pytest version available without the need to upgrade setuptools. The pytest requirement is moved to py3-requirements.txt. We can not go back to test with Python2 + pytest-2.9.2 after this patch because our test script need to be adjusted as well. This is OK since we have default testing using Python3 since IMPALA-14333 and stop building impala-shell for Python2 since IMPALA-14606. The adjustment are follows: - Replace deprecated @pytest.yield_fixture with plain @pytest.fixture. - Replace --resultlog parameter (removed in version 6.0) with --report-log from pytest-reportlog plugin. - Make impala-shell.sh bootstrap Python3 venv (infra/python/env-gcc10.4.0-py3/) by default. Python2 venv (infra/python/env-gcc10.4.0/) is not bootstrapped automatically anymore. - Upgrade execnet to version 1.9.0. This is required by pytest-xdist==2.4.0. - Remove unused pytest-runner plugin - Add -Wonce argument in pytest.ini to slightly suppress warnings. - Add "junit_logging = system-err" option at pytest.ini to continue logging stderr output to junit xml file. - Fix rootdir and pytest.ini path programmatically at run-tests.py because individual pytest-xdist worker often does not pick this up correctly during parallel run. With pytest-xdist==2.4.0, parallel EE tests does not show verbose individual test names. It only shows pytest progress marker like following line ...ss....s................................s............................. This is a known limitation in pytest-xdist because execnet, the underlying library used for communication between master and workers, does not support transferring stdout/stderr from workers. https://pytest-xdist.readthedocs.io/en/stable/known-limitations.html Read following links for more detail about the deprecation notes: https://docs.pytest.org/en/stable/deprecations.html Change SKIP_SSL_MSG default to empty string because skipif does not accept None reason anymore. Removed run-process-failure-tests.sh (unused after IMPALA-5534) and unused pytest fixtures. Fixed erroneous log formatting in test_restart_services.py. Fixed small warnings found by pytest-6.2.5 that help stabilize exhaustive tests run at: - test_automatic_invalidation.py - test_calcite_planner.py - test_events_custom_configs.py - test_session_expiration.py - test_shell_interactive.py - auto_scaler.py - concurrent_workload.py - hdfs_util.py Most of the remaining warnings are about not closing resources properly at the end of test. These warnings should be addressed in follow up patches. Testing: - Pass exhaustive tests. Change-Id: Ic3812fe976ef09ac48753dee30151714f4752c24 Reviewed-on: http://gerrit.cloudera.org:8080/23842 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M bin/impala-shell.sh M bin/run-all-tests.sh M infra/python/deps/py3-requirements.txt M infra/python/deps/requirements.txt M tests/common/network.py M tests/comparison/tests/conftest.py M tests/comparison/tests/test_cursor.py M tests/comparison/tests/test_query_objects.py M tests/conftest.py M tests/custom_cluster/test_automatic_invalidation.py M tests/custom_cluster/test_calcite_planner.py M tests/custom_cluster/test_client_ssl.py M tests/custom_cluster/test_events_custom_configs.py M tests/custom_cluster/test_ipv6.py M tests/custom_cluster/test_restart_services.py M tests/custom_cluster/test_session_expiration.py M tests/pytest.ini M tests/query_test/test_observability.py M tests/run-custom-cluster-tests.sh D tests/run-process-failure-tests.sh M tests/run-tests.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py M tests/util/auto_scaler.py M tests/util/concurrent_workload.py M tests/util/hdfs_util.py 27 files changed, 192 insertions(+), 298 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/23842 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic3812fe976ef09ac48753dee30151714f4752c24 Gerrit-Change-Number: 23842 Gerrit-PatchSet: 13 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]>
