Hello Jason Fehr, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#4).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Switch parallel loops in test_stress_infra.py to a simple loop that
fail-proof in Python3.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to vector permutation
difference (see IMPALA-13125) and other intermittent issue in Python3
pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Set timezone='Europe/Budapest' in test_sort and test_top_n to get
consistent result for reading functional_parquet.complextypes_structs
and functional_orc_def.complextypes_structs.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return rows where alltypesagg.tinyint_col is null.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny are contrained only for json/none/none (functional_json).
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M
testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/infra/test_stress_infra.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
30 files changed, 379 insertions(+), 243 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/4
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Jason Fehr <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
