Hello Jason Fehr, Peter Rozsa, Joe McDonnell, Michael Smith, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23842
to look at the new patch set (#5).
Change subject: IMPALA-14668: Upgrade to pytest 6.2.5
......................................................................
IMPALA-14668: Upgrade to pytest 6.2.5
This patch upgrade pytest version from 2.9.2 to 6.2.5, the highest
pytest version available without the need to upgrade setuptools.
The pytest requirement is moved to py3-requirements.txt. We can not go
back to test with Python2 + pytest-2.9.2 after this patch because our
test script need to be adjusted as well. This is OK since we have
default testing using Python3 since IMPALA-14333 and stop building
impala-shell for Pyton2 since IMPALA-14606. The adjustment are follows:
- Replace deprecated @pytest.yield_fixture with plain @pytest.fixture.
- Replace --resultlog parameter (removed in version 6.0) with
--report-log from pytest-reportlog plugin.
- Make impala-shell.sh bootstrap Python3
venv (infra/python/env-gcc10.4.0-py3/) by default. Python2
venv (infra/python/env-gcc10.4.0/) is not bootstrapped automatically
anymore.
- Upgrade execnet to version 1.9.0. This is required by
pytest-xdist==2.4.0.
- Remove unused pytest-runner plugin
- Add -Wonce argument in pytest.ini to slightly suppress warnings.
- Add "junit_logging = system-err" option at pytest.ini to continue
logging stderr output to junit xml file.
- Fix rootdir and pytest.ini path programmatically at run-tests.py
because individual pytest-xdist worker often does not pick this up
correctly during parallel run.
With pytest-xdist==2.4.0, parallel EE tests does not show verbose
individual test names. It only shows pytest progress marker like
following line
...ss....s................................s.............................
This is a known limitation in pytest-xdist because execnet, the
underlying library used for communication between master and workers,
does not support transferring stdout/stderr from workers.
https://pytest-xdist.readthedocs.io/en/stable/known-limitations.html
Read following links for more detail about the deprecation notes:
https://docs.pytest.org/en/stable/deprecations.html
Change SKIP_SSL_MSG default to empty string because skipif does not
accept None reason anymore. Removed run-process-failure-tests.sh (unused
after IMPALA-5534) and unused pytest fixtures. Fixed erroneous log
formatting in test_restart_services.py. Changed test_calcite_planner.py
to use a more stable ImpalaConnection client rather than cursor fixture.
This patch also fix small warnings found by pytest-6.2.5 at
test_shell_interactive.py and hdfs_util.py. Most of the warnings are
about not closing resources properly at the end of test. These warnings
should be addressed in follow up patches.
Testing:
- Pass core tests.
Change-Id: Ic3812fe976ef09ac48753dee30151714f4752c24
---
M bin/impala-shell.sh
M bin/run-all-tests.sh
M infra/python/deps/py3-requirements.txt
M infra/python/deps/requirements.txt
M tests/common/network.py
M tests/comparison/tests/conftest.py
M tests/comparison/tests/test_cursor.py
M tests/comparison/tests/test_query_objects.py
M tests/conftest.py
M tests/custom_cluster/test_calcite_planner.py
M tests/custom_cluster/test_client_ssl.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_restart_services.py
M tests/pytest.ini
M tests/query_test/test_observability.py
M tests/run-custom-cluster-tests.sh
D tests/run-process-failure-tests.sh
M tests/run-tests.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/util/hdfs_util.py
22 files changed, 105 insertions(+), 207 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/23842/5
--
To view, visit http://gerrit.cloudera.org:8080/23842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic3812fe976ef09ac48753dee30151714f4752c24
Gerrit-Change-Number: 23842
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Jason Fehr <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
