[
https://issues.apache.org/jira/browse/IMPALA-12631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815207#comment-17815207
]
ASF subversion and git services commented on IMPALA-12631:
----------------------------------------------------------
Commit 637c750a3e83e8cf01ee0bd5141087d423cf6253 in impala's branch
refs/heads/master from zhangyifan27
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=637c750a3 ]
IMPALA-12631: Improve count star performance for parquet scans
Before this patch frontend generates multiple scan ranges for a
parquet file when count star optimization is enabled. Backend function
HdfsParquetScanner::GetNextInternal() also call NextRowGroup()
multiple times to find row groups and sum up RowGroup.num_rows. This
could be inefficient because we only need to read file metadata to
compute count star. This patch optimizes it by creating only one
scan range that contains the file footer for each parquet file.
The following table shows a performance comparison before and after
the patch. primitive_count_star_multiblock query is a modified
primitive_count_star query that targets a multi-block
tpch10_parquet.lineitem table. The files of the table are generated
by the command `hdfs dfs -Ddfs.block.size=1048576 -cp -f -d`.
+-------------------+---------------------------------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+--------+
| Workload | Query | File Format |
Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters |
Median Diff(%) | MW Zval | Tval |
+-------------------+---------------------------------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+--------+
| TPCDS(10) | TPCDS-Q_COUNT_OPTIMIZED | parquet / none / none |
0.17 | 0.16 | +2.58% | * 29.53% * | * 27.16% * | 30 |
+1.20% | 0.58 | 0.35 |
| TPCDS(10) | TPCDS-Q_COUNT_UNOPTIMIZED | parquet / none / none |
0.27 | 0.26 | +2.96% | 8.97% | 9.94% | 30 |
+0.16% | 0.44 | 1.19 |
| TPCDS(10) | TPCDS-Q_COUNT_ZERO_SLOT | parquet / none / none |
0.18 | 0.18 | -0.69% | 1.65% | 1.99% | 30 |
-0.34% | -1.55 | -1.47 |
| TARGETED-PERF(10) | primitive_count_star_multiblock | parquet / none / none |
0.06 | 0.12 | I -49.88% | 4.11% | 3.53% | 30 | I
-99.97% | -6.54 | -66.81 |
+-------------------+---------------------------------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+--------+
Testing:
- Ran PlannerTest#testParquetStatsAgg
- Added new test cases to query_test/test_aggregation.py
Change-Id: Ib9cd2448fe51a420d4559d0cc861c4d30822f4fd
Reviewed-on: http://gerrit.cloudera.org:8080/20804
Reviewed-by: Zoltan Borok-Nagy <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Improve count star performance for parquet scans
> ------------------------------------------------
>
> Key: IMPALA-12631
> URL: https://issues.apache.org/jira/browse/IMPALA-12631
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: YifanZhang
> Assignee: YifanZhang
> Priority: Major
>
> The code in the backend function HdfsParquetScanner::GetNextInternal() is not
> efficient now. We use row group statistics instead of file meta statistics,
> which leads to unnecessary materialization overhead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
