Hello Quanlong Huang, Qifan Chen, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18327
to look at the new patch set (#6).
Change subject: IMPALA-11123: Optimize count(star) for ORC scans
......................................................................
IMPALA-11123: Optimize count(star) for ORC scans
IMPALA-5036 added optimization for count(star) in Parquet scans that
avoid materializing dummy rows. This change provides similar
optimization for ORC tables. We use the stripes num rows statistics when
computing the count star instead of materializing empty rows. The
aggregate function changed from a count to a special sum function
initialized to 0.
This count(star) optimization is disabled for the full ACID table
because the scanner might need to read and validate the
'currentTransaction' column in table's special schema.
This patch drops 'parquet' from names related to the count star
optimization. It also improves the count(star) operation by serving the
result just from the file's footer stats for both Parquet and ORC.
Testing:
- Add PlannerTest.testOrcStatsAgg
- Add TestAggregationQueries::test_orc_count_star_optimization
- Exercise count(star) in TestOrc::test_misaligned_orc_stripes
- Pass core tests
Change-Id: I0fafa1182f97323aeb9ee39dd4e8ecd418fa6091
---
M be/src/exec/hdfs-columnar-scanner.cc
M be/src/exec/hdfs-columnar-scanner.h
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-orc-scanner.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.h
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/orc-stats-agg.test
M
testdata/workloads/functional-planner/queries/PlannerTest/parquet-stats-agg.test
M
testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-upper-lower-bound-metrics.test
A testdata/workloads/functional-query/queries/QueryTest/orc-stats-agg.test
M tests/custom_cluster/test_executor_groups.py
M tests/custom_cluster/test_query_retries.py
M tests/query_test/test_aggregation.py
M tests/query_test/test_scanners.py
22 files changed, 761 insertions(+), 95 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/18327/6
--
To view, visit http://gerrit.cloudera.org:8080/18327
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0fafa1182f97323aeb9ee39dd4e8ecd418fa6091
Gerrit-Change-Number: 18327
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Qifan Chen <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
