Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12503
Change subject: IMPALA-8207: Fix query loading for perf and stress tests ...................................................................... IMPALA-8207: Fix query loading for perf and stress tests Problems with perf queries (run-workload.py): - TPCH picks up stress test specific queries (TPCH-AGG1/2/3) - TPCDS picks up queries that were intended just to validate that data was loaded properly but that aren't interesting from a perf perspective (TPCDS-COUNT-<table>) - TPCDS picks up both decimal_v1 and decimal_v2 queries. This is mostly harmless as for queries with matching names only one gets run but it causes some queries with mismatched names to be run twice (TPCDS-Q39-1/2 vs. TPCDS-Q39.1/2) Problems with stress queries (concurrent_select.py): - TPCDS fails to pick up Q22A as it does not use the decimal_v2 queries, even though decimal_v2 is the default now. This problem is exacerbated by the fact that the two scripts have different code paths for selecting the queries, so in the past changes that were made to one path were not always made to the other. This patch merges the two paths to reduce code duplication and prevent these sorts of issues in the future, and fixes the above issues. One complication is that historically the stress test has used query names in the form 'q1' whereas the perf test has used query names in the form 'TPCH-Q1'. This patch standardizes on using 'TPCH-Q1'. Testing: - Added a test that checks that the perf tests pick up the expected number of queries. - Manually ran the scripts and verified that the correct queries are selected. Change-Id: Id1966d6ca8babdda07d47e089b75ba06d0318c0d --- M testdata/workloads/tpcds/queries/tpcds-decimal_v2-q39-1.test M testdata/workloads/tpcds/queries/tpcds-decimal_v2-q39-2.test A tests/infra/test_perf_infra.py M tests/infra/test_stress_infra.py M tests/performance/workload.py M tests/stress/concurrent_select.py M tests/util/parse_util.py M tests/util/test_file_parser.py 8 files changed, 106 insertions(+), 63 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/12503/1 -- To view, visit http://gerrit.cloudera.org:8080/12503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id1966d6ca8babdda07d47e089b75ba06d0318c0d Gerrit-Change-Number: 12503 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall <[email protected]>
