Hello David Knupp, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12503
to look at the new patch set (#2).
Change subject: IMPALA-8207: Fix query loading for perf and stress tests
......................................................................
IMPALA-8207: Fix query loading for perf and stress tests
Problems with perf queries (run-workload.py):
- TPCH picks up stress test specific queries (TPCH-AGG1/2/3)
- TPCDS picks up queries that were intended just to validate that data
was loaded properly but that aren't interesting from a perf
perspective (TPCDS-COUNT-<table>)
- TPCDS picks up both decimal_v1 and decimal_v2 queries. This is
mostly harmless as for queries with matching names only one gets run
but it causes some queries with mismatched names to be run twice
(TPCDS-Q39-1/2 vs. TPCDS-Q39.1/2)
Problems with stress queries (concurrent_select.py):
- TPCDS fails to pick up Q22A as it does not use the decimal_v2
queries, even though decimal_v2 is the default now.
This problem is exacerbated by the fact that the two scripts have
different code paths for selecting the queries, so in the past changes
that were made to one path were not always made to the other.
This patch merges the two paths to reduce code duplication and prevent
these sorts of issues in the future, and fixes the above issues.
One complication is that historically the stress test has used query
names in the form 'q1' whereas the perf test has used query names in
the form 'TPCH-Q1'. This patch standardizes on using 'TPCH-Q1'.
Testing:
- Added a test that checks that the perf tests pick up the expected
number of queries.
- Manually ran the scripts and verified that the correct queries are
selected.
Change-Id: Id1966d6ca8babdda07d47e089b75ba06d0318c0d
---
M testdata/workloads/tpcds/queries/tpcds-decimal_v2-q39-1.test
M testdata/workloads/tpcds/queries/tpcds-decimal_v2-q39-2.test
A tests/infra/test_perf_infra.py
M tests/infra/test_stress_infra.py
M tests/performance/workload.py
M tests/stress/concurrent_select.py
M tests/util/parse_util.py
M tests/util/test_file_parser.py
8 files changed, 107 insertions(+), 63 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/12503/2
--
To view, visit http://gerrit.cloudera.org:8080/12503
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id1966d6ca8babdda07d47e089b75ba06d0318c0d
Gerrit-Change-Number: 12503
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Marshall <[email protected]>
Gerrit-Reviewer: David Knupp <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
