[
https://issues.apache.org/jira/browse/IMPALA-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956465#comment-16956465
]
ASF subversion and git services commented on IMPALA-5036:
---------------------------------------------------------
Commit 3f1d0472243f25eacf611b12a5e1150294683957 in impala's branch
refs/heads/master from norbert.luksa
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3f1d047 ]
IMPALA-6501: Optimize count(star) for Kudu scans
IMPALA-5036 added an optimisation for count(star) in Parquet scans
that avoids materialising dummy rows. This change provides similar
optimization for Kudu tables.
Instead of materializing empty rows when computing count star, we use
the NumRows field from the Kudu API. The Kudu scanner tuple is
modified to have one slot into which we will write the
num rows statistic. The aggregate function is changed from count to a
special sum function that gets initialized to 0.
Tests:
* Added end-to-end tests
̣* Added planner tests
* Run performance tests on tpch.lineitem Kudu table with 25 set as
scaling factor, on 1 node, with mt_dop set to 1, just to measure
the speedup gained when scanning. Counting the rows before the
optimization took around 400ms, and around 170ms after.
Change-Id: Ic99e0f954d0ca65779bd531ca79ace1fcb066fb9
Reviewed-on: http://gerrit.cloudera.org:8080/14347
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Improve COUNT(*) performance of Parquet scans.
> ----------------------------------------------
>
> Key: IMPALA-5036
> URL: https://issues.apache.org/jira/browse/IMPALA-5036
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0
> Reporter: Alexander Behm
> Assignee: Taras Bobrovytsky
> Priority: Major
> Labels: parquet, performance, ramp-up
> Fix For: Impala 2.10.0
>
>
> {code}
> select count(*) from parquet_table;
> select count(*) from parquet_table group by partition_col;
> {code}
> Impala already has a special code path for fast Parquet scans when no columns
> are scanned and materialized, but the performance can be significantly
> improved with a plan+execution change, as follows:
> *Execution change*
> Instead of returning empty batches until num_rows have been returned, the
> Parquet scanner can populate a single slot with the num_rows from the Parquet
> row groups
> *Plan change*
> The count(*) local aggregation needs to be changed to a sum(num_rows_slot)
> aggregation.
> The final distributed plan will be:
> scan -> local agg with sum(num_rows_slot) -> merge agg sum(sum(num_rows_slot))
> This optimization is applicable where is only a count(*) and there are no
> scan predicates.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
