Alex Behm has submitted this change and it was merged. Change subject: IMPALA-5039: Fix variability in parquet dictionary filtering test ......................................................................
IMPALA-5039: Fix variability in parquet dictionary filtering test The tests for dictionary filtering look at how many row groups are processed and how many are filtered by matching text in the profile. However, the number of row groups processed and filtered by any individual fragment depends on how the work is split and how many impalads are running. This causes variability in the test output. To fix this, the test needs a way to aggregate the results across fragments. This fix introduces the following syntax for specifying these aggregates: aggregate(function_name, field_name): expected_value This searches the runtime profile for lines that contain 'field_name: number'. It skips the averaged fragment, as this is derived from all the other fragments. Currently, only SUM is implemented, and the expected_value is required to be an integer. It should be easy to implement other interesting functions like COUNT and MIN/MAX. It would also be possible to extend it to floats. Switching the dictionary filtering tests over to this new syntax eliminates the variability in the tests. Change-Id: I6b7b84d973b3ac678a24e82900f2637d569158bb Reviewed-on: http://gerrit.cloudera.org:8080/6301 Tested-by: Impala Public Jenkins Reviewed-by: Alex Behm <[email protected]> --- D testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-filtering.test M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test M tests/common/test_result_verifier.py M tests/query_test/test_mt_dop.py M tests/query_test/test_scanners.py 5 files changed, 134 insertions(+), 317 deletions(-) Approvals: Impala Public Jenkins: Verified Alex Behm: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/6301 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I6b7b84d973b3ac678a24e82900f2637d569158bb Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Marcel Kornacker <[email protected]>
