Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22231
Change subject: IMPALA-13620: Refresh compute_table_stats.py script ...................................................................... IMPALA-13620: Refresh compute_table_stats.py script This patch refreshes compute_table_stats.py script with the following changes: - Limit parallelism to 16 at maximum if --parallelism argument is not set. - Change its default connection to hs2, leveraging existing ImpylaHS2Connection. - Change OptionParser to ArgumentParser. - Use impala-python3 to run the script. - Add --exclude_table_names to skip running COMPUTE STATS on certain tables/views. - Round robin compute stats around different DB to reduce chance of lock contention over single DB. This patch also improves query handle logging in ImpylaHS2Connection. collect_profile_and_log argument is added to control whether to pull logs and runtime profile at the end of __fetch_results(). The default behavior remains unchanged. Skip COMPUTE STATS for functional_kudu.alltypesagg and functional_kudu.manynulls because it is invalid to run COMPUTE STATS over view. compute-table-stats.sh is reordered to slowly ramp up COMPUTE STATS query concurrency over single DB. Testing: Repeatedly run compute-table-stats.sh from cold state and confirm there is no error occurs. This is the script to do so from active minicluster: cd $IMPALA_HOME ./bin/start-impala-cluster.py --kill ./testdata/bin/kill-hive-server.sh ./testdata/bin/run-hive-server.sh ./bin/start-impala-cluster.py ./testdata/bin/compute-table-stats.sh > /tmp/compute-stats.txt 2>&1 grep error /tmp/compute-stats.txt Change-Id: I1ebf02f95b957e7dda3a30622b87e8fca3197699 --- M testdata/bin/compute-table-stats.sh M tests/common/impala_connection.py M tests/util/compute_table_stats.py 3 files changed, 195 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/22231/1 -- To view, visit http://gerrit.cloudera.org:8080/22231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1ebf02f95b957e7dda3a30622b87e8fca3197699 Gerrit-Change-Number: 22231 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto <[email protected]>
