[
https://issues.apache.org/jira/browse/IMPALA-11123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820332#comment-17820332
]
ASF subversion and git services commented on IMPALA-11123:
----------------------------------------------------------
Commit 15e471563d22ca82fea3c435a98f859bfaf99d6a in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=15e471563 ]
IMPALA-11123: Reimplement ORC optimized count star
Commit 7ca20b3c94b1c9c1ddd4ed1e89f0969a0df55330 revert the original
optimized count(star) for ORC scan from commit
f932d78ad0a30e322d59fc39072f710f889d2135 (gerrit review
http://gerrit.cloudera.org:8080/18327). The revert is necessary since
the unification of count star and zero slot functions into
HdfsColumnarScanner and causing significant regression for non-optimized
counts star query in parquet format (over 15% slower
MaterializeTupleTime).
This patch reimplements optimized count(star) for ORC scan code path
while minimizing the code changes needed for parquet scan code path.
After this patch, ORC and parquet code path will have only the following
new things in common:
- THdfsScanNode.count_star_slot_offset renamed to
THdfsScanNode.star_slot_offset
- HdfsScanner::IssueFooterRanges will only issue footer ranges if
IsZeroSlotTableScan() or optimize_count_star() is true (made possible
for parquet by IMPALA-12631).
The structure of HdfsParquetScanner::GetNextInternal() remains
unchanged. Its zero scan slot code path is still served through num_rows
metadata from the parquet footer, while the optimized count star code
path still loops over row groups metadata (also from parquet footer).
The following table shows single-node benchmark result of 3 count query
variant on TPC-DS scale 10, both in ORC and parquet format, looped 9
times.
+-----------+---------------------------+---------+--------+-------------+------------+
| Workload | Query | Format | Avg(s) | Base Avg(s) |
Delta(Avg) |
+-----------+---------------------------+---------+--------+-------------+------------+
| TPCDS(10) | TPCDS-Q_COUNT_UNOPTIMIZED | orc | 0.30 | 0.28 |
+6.50% |
| TPCDS(10) | TPCDS-Q_COUNT_OPTIMIZED | parquet | 0.14 | 0.14 |
+1.56% |
| TPCDS(10) | TPCDS-Q_COUNT_ZERO_SLOT | parquet | 0.27 | 0.27 |
+1.42% |
| TPCDS(10) | TPCDS-Q_COUNT_ZERO_SLOT | orc | 0.28 | 0.29 |
-3.03% |
| TPCDS(10) | TPCDS-Q_COUNT_UNOPTIMIZED | parquet | 0.21 | 0.22 |
-4.45% |
| TPCDS(10) | TPCDS-Q_COUNT_OPTIMIZED | orc | 0.14 | 0.21 | I
-35.92% |
+-----------+---------------------------+---------+--------+-------------+------------+
Testing:
- Restore PlannerTest.testOrcStatsAgg
- Restore TestAggregationQueriesRunOnce and
TestAggregationQueriesRunOnce::test_orc_count_star_optimization
- Exercise count(star) in TestOrc::test_misaligned_orc_stripes
- Pass core tests
Change-Id: I5971c8f278e1dee44e2a8dd4d2f043d22ebf5d17
Reviewed-on: http://gerrit.cloudera.org:8080/19927
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> Optimize count(*) for ORC scans
> -------------------------------
>
> Key: IMPALA-11123
> URL: https://issues.apache.org/jira/browse/IMPALA-11123
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Quanlong Huang
> Assignee: Riza Suminto
> Priority: Critical
> Labels: ramp-up
> Fix For: Impala 4.1.0, Impala 4.4.0
>
>
> IMPALA-5036 and IMPALA-6501 added an optimisation for count\(*\) in Parquet
> and Kudu scans that avoids materialising dummy rows. We should do something
> similar for ORC.
> Refer IMPALA-5036 for more details.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
