Hello Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23468
to look at the new patch set (#4).
Change subject: IMPALA-14501: Migrate most scripts from impala-python to
impala-python3
......................................................................
IMPALA-14501: Migrate most scripts from impala-python to impala-python3
To remove the dependency on Python 2, existing scripts need to use
impala-python3 rather than impala-python. This command finds those
locations:
git grep impala-python | grep -v impala-python3 | grep -v impala-python-common
| grep -v init-impala-python
This removes or switches most of these locations by various means:
1. If a python file has a #!/bin/env impala-python but doesn't have a main
function, it removes the hash-bang and makes sure that the file is
not executable.
2. Most scripts can simply switch from impala-python to impala-python3
with minimal changes.
3. The cm-api pypi package (which doesn't support Python 3) has been
replaced by the cm-client pypi package and interfaces have changed.
Rather than migrating the code (which hasn't been used in years), this
deletes the old code and stops installing cm-api into the virtualenv.
The code can be restored and revamped if there is any interest in
interacting with CM clusters.
4. This switches tests/comparison over to impala-python3, but this code has
bit-rotted. Some pieces can be run manually, but it can't be fully
verified with Python 3. It shouldn't hold back the migration on its own.
5. This also replaces locations of impala-python in comments / documentation /
READMEs.
6. kazoo (used for interacting with HBase) needed to be upgraded to a
version that supports Python 3. The newest version of kazoo requires
upgrades of other component versions, so this uses kazoo 2.8.0 to avoid
needing other upgrades.
The two remaining uses of impala-python are:
- impala-shell packaging
- bin/impala-env-versioned-python
These will be removed separately when we drop Python 2 support for
impala-shell.
Testing:
- Ran core job
- Manual testing of individual scripts (except some bitrotted areas like the
random query generator)
Change-Id: If209b761290bc7e7c716c312ea757da3e3bca6dc
---
M be/src/codegen/gen_ir_descriptions.py
M bin/bootstrap_system.sh
M bin/dump_breakpad_symbols.py
M bin/gen_build_version.py
M bin/generate_minidump_collection_testdata.py
M bin/get_code_size.py
M bin/inline_pom.py
M bin/load-data.py
M bin/parse-thrift-profile.py
M bin/run-workload.py
M bin/set-pythonpath.sh
M bin/single_node_perf_run.py
M docker/setup_build_context.py
M fe/src/test/java/org/apache/impala/customcluster/LdapImpylaHttpTest.java
M infra/python/README
M infra/python/deps/py2-requirements.txt
M infra/python/deps/py3-requirements.txt
M infra/python/deps/requirements.txt
M lib/python/impala_py_lib/jenkins/generate_junitxml.py
M lib/python/impala_py_lib/profiles.py
M testdata/bin/check-hbase-nodes.py
M testdata/bin/create-load-data.sh
M testdata/bin/create-tpc-jdbc-tables.py
M testdata/bin/generate-schema-statements.py
M testdata/bin/generate-test-vectors.py
M testdata/bin/load-tpc-kudu.py
M testdata/bin/load_nested.py
M testdata/bin/random_avro_schema.py
M testdata/bin/rewrite-iceberg-metadata.py
M testdata/bin/wait-for-hiveserver2.py
M testdata/bin/wait-for-metastore.py
M testdata/common/text_delims_table.py
M testdata/common/widetable.py
M tests/comparison/ORACLE.txt
M tests/comparison/cli_options.py
M tests/comparison/cluster.py
M tests/comparison/data_generator.py
M tests/comparison/data_generator_mapper.py
M tests/comparison/data_generator_mapred_common.py
M tests/comparison/discrepancy_searcher.py
M tests/comparison/leopard/controller.py
M tests/comparison/leopard/front_end.py
M tests/comparison/tests/README
M tests/comparison/tests/conftest.py
M tests/comparison/util/verify-oracle-connection.py
M tests/custom_cluster/test_auto_scaling.py
M tests/custom_cluster/test_executor_groups.py
M tests/custom_cluster/test_no_delete_stats_in_truncate.py
M tests/custom_cluster/test_shared_catalogd.py
M tests/metadata/test_column_unicode.py
M tests/shell/test_beeline.py
M tests/shell/test_cookie_util.py
M tests/shell/test_kerberos_util.py
M tests/shell/test_shell_client.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_commandline_jwt_auth.py
M tests/shell/test_shell_commandline_kerberos_auth.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/stress/concurrent_select.py
M tests/stress/extract_min_mem.py
M tests/stress/mem_broker.py
M tests/stress/queries.py
M tests/stress/query_retries_stress_runner.py
M tests/stress/query_runner.py
M tests/stress/runtime_info.py
M tests/stress/util.py
M tests/util/acid_txn.py
M tests/util/auto_scaler.py
M tests/util/concurrent_workload.py
M tests/util/run_impyla_http_query.py
71 files changed, 65 insertions(+), 342 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/23468/4
--
To view, visit http://gerrit.cloudera.org:8080/23468
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If209b761290bc7e7c716c312ea757da3e3bca6dc
Gerrit-Change-Number: 23468
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
