[
https://issues.apache.org/jira/browse/IMPALA-9956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17158806#comment-17158806
]
ASF subversion and git services commented on IMPALA-9956:
---------------------------------------------------------
Commit dabcedeca363051a9d73f2abb1a88ff230e41cc7 in impala's branch
refs/heads/master from Tim Armstrong
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=dabcede ]
IMPALA-9956: inline hot functions in Sorter
Add some compiler hints to force inlining of small
functions into the hot Partition() loop.
Performance:
A single node perf run on TPC-H showed no perf change.
A single node performance run with the queries that target
sort performance showed up to a 19% reduction in time spent
in the sort.
+-------------------+-----------------------+---------+------------+------------+----------------+
| Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s)
| Delta(GeoMean) |
+-------------------+-----------------------+---------+------------+------------+----------------+
| TARGETED-PERF(30) | parquet / none / none | 5.52 | -5.82% | 4.00
| -9.74% |
+-------------------+-----------------------+---------+------------+------------+----------------+
+-------------------+-------------------------------------+-----------------------+--------+-------------+------------+-----------+----------------+-------+----------------+---------+--------+
| Workload | Query | File Format
| Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters |
Median Diff(%) | MW Zval | Tval |
+-------------------+-------------------------------------+-----------------------+--------+-------------+------------+-----------+----------------+-------+----------------+---------+--------+
| TARGETED-PERF(30) | primitive_orderby_all | parquet / none /
none | 11.89 | 12.22 | -2.73% | 1.07% | 1.20% | 10 |
-2.88% | -3.13 | -5.42 |
| TARGETED-PERF(30) | primitive_orderby_bigint_expression | parquet / none /
none | 2.61 | 2.94 | I -11.27% | 0.83% | 1.14% | 10 |
I -12.56% | -3.58 | -26.25 |
| TARGETED-PERF(30) | primitive_orderby_bigint | parquet / none /
none | 2.06 | 2.42 | I -14.80% | 0.94% | 0.68% | 10 |
I -17.43% | -3.58 | -44.37 |
+-------------------+-------------------------------------+-----------------------+--------+-------------+------------+-----------+----------------+-------+----------------+---------+--------+
(I) Improvement: TARGETED-PERF(30) primitive_orderby_bigint_expression [parquet
/ none / none] (2.94s -> 2.61s [-11.27%])
+---------------------+------------+----------+----------+------------+-----------+----------+----------+------------+--------+-------+-------+-----------+
| Operator | % of Query | Avg | Base Avg | Delta(Avg) |
StdDev(%) | Max | Base Max | Delta(Max) | #Hosts | #Inst | #Rows | Est
#Rows |
+---------------------+------------+----------+----------+------------+-----------+----------+----------+------------+--------+-------+-------+-----------+
| 02:ANALYTIC | 11.84% | 332.95ms | 337.56ms | -1.37% | 4.86%
| 360.86ms | 379.52ms | -4.92% | 1 | 1 | 5.09M | 18.00M |
| F00:EXCHANGE SENDER | 15.61% | 439.03ms | 454.63ms | -3.43% | 4.86%
| 478.29ms | 485.79ms | -1.55% | 1 | 1 | -1 | -1 |
| 01:SORT | 67.05% | 1.89s | 2.21s | -14.88% | 0.98%
| 1.92s | 2.26s | -15.07% | 1 | 1 | 5.09M | 18.00M |
+---------------------+------------+----------+----------+------------+-----------+----------+----------+------------+--------+-------+-------+-----------+
(I) Improvement: TARGETED-PERF(30) primitive_orderby_bigint [parquet / none /
none] (2.42s -> 2.06s [-14.80%])
+---------------------+------------+----------+----------+------------+-----------+----------+----------+------------+--------+-------+-------+-----------+
| Operator | % of Query | Avg | Base Avg | Delta(Avg) |
StdDev(%) | Max | Base Max | Delta(Max) | #Hosts | #Inst | #Rows | Est
#Rows |
+---------------------+------------+----------+----------+------------+-----------+----------+----------+------------+--------+-------+-------+-----------+
| 02:ANALYTIC | 15.39% | 367.90ms | 373.26ms | -1.44% | 3.48%
| 390.03ms | 393.01ms | -0.76% | 1 | 1 | 5.09M | 18.00M |
| F00:EXCHANGE SENDER | 15.64% | 373.88ms | 374.12ms | -0.07% | 2.83%
| 389.96ms | 386.36ms | +0.93% | 1 | 1 | -1 | -1 |
| 01:SORT | 56.28% | 1.35s | 1.68s | -20.10% | 1.14%
| 1.38s | 1.70s | -18.92% | 1 | 1 | 5.09M | 18.00M |
| 00:SCAN HDFS | 9.67% | 231.18ms | 231.77ms | -0.25% | 7.06%
| 247.79ms | 250.70ms | -1.16% | 1 | 1 | 5.09M | 18.00M |
+---------------------+------------+----------+----------+------------+-----------+----------+----------+------------+--------+-------+-------+-----------+
Change-Id: I7a8034ab6d2e3c71a2d2f2fcc3d6b788e9398194
Reviewed-on: http://gerrit.cloudera.org:8080/16202
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Inlining functions in Sorter::Partition() gives a significant speedup.
> ----------------------------------------------------------------------
>
> Key: IMPALA-9956
> URL: https://issues.apache.org/jira/browse/IMPALA-9956
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Labels: performance
>
> In TPC-H q67, the sort gets ~9% faster just from this small change.
> {noformat}
> AFTER
> Fetched 100 row(s) in 3.27s
> [localhost:21000] tpcds_parquet> summary;
> +-----------------+--------+-------+----------+----------+---------+------------+-----------+---------------+---------------------------+
> | Operator | #Hosts | #Inst | Avg Time | Max Time | #Rows | Est.
> #Rows | Peak Mem | Est. Peak Mem | Detail |
> +-----------------+--------+-------+----------+----------+---------+------------+-----------+---------------+---------------------------+
> | F00:ROOT | 1 | 1 | 402.93us | 402.93us | |
> | 0 B | 0 B | |
> | 11:TOP-N | 1 | 1 | 9.85ms | 9.85ms | 100 | 100
> | 7.04 MB | 9.38 KB | |
> | 10:ANALYTIC | 1 | 1 | 102.50ms | 102.50ms | 514.59K | 15.09M
> | 11.02 MB | 4.00 MB | |
> | 09:SORT | 1 | 1 | 304.39ms | 304.39ms | 514.59K | 15.09M
> | 80.02 MB | 38.00 MB | |
> | 08:AGGREGATE | 1 | 1 | 178.87ms | 178.87ms | 514.59K | 15.09M
> | 114.16 MB | 1.42 GB | FINALIZE |
> | 07:AGGREGATE | 1 | 1 | 794.68ms | 794.68ms | 514.59K | 15.09M
> | 257.67 MB | 16.98 GB | FINALIZE |
> | 06:HASH JOIN | 1 | 1 | 33.95ms | 33.95ms | 535.97K | 2.88M
> | 3.33 MB | 2.88 MB | INNER JOIN |
> | |--03:SCAN HDFS | 1 | 1 | 2.77ms | 2.77ms | 18.00K | 18.00K
> | 2.69 MB | 80.00 MB | tpcds_parquet.item |
> | 05:HASH JOIN | 1 | 1 | 8.02ms | 8.02ms | 535.97K | 2.88M
> | 2.05 MB | 1.94 MB | INNER JOIN |
> | |--02:SCAN HDFS | 1 | 1 | 1.18ms | 1.18ms | 12 | 12
> | 69.00 KB | 32.00 MB | tpcds_parquet.store |
> | 04:HASH JOIN | 1 | 1 | 10.22ms | 10.22ms | 535.97K | 2.88M
> | 2.01 MB | 1.94 MB | INNER JOIN |
> | |--01:SCAN HDFS | 1 | 1 | 2.62ms | 2.62ms | 365 | 7.30K
> | 1.33 MB | 80.00 MB | tpcds_parquet.date_dim |
> | 00:SCAN HDFS | 1 | 1 | 9.06ms | 9.06ms | 535.97K | 2.88M
> | 14.37 MB | 64.00 MB | tpcds_parquet.store_sales |
> +-----------------+--------+-------+----------+----------+---------+------------+-----------+---------------+---------------------------+
> BEFORE
> Fetched 100 row(s) in 3.33s
> [localhost:21000] tpcds_parquet> summary;
> +-----------------+--------+-------+----------+----------+---su------+------------+-----------+---------------+---------------------------+
> | Operator | #Hosts | #Inst | Avg Time | Max Time | #Rows | Est.
> #Rows | Peak Mem | Est. Peak Mem | Detail |
> +-----------------+--------+-------+----------+----------+---------+------------+-----------+---------------+---------------------------+
> | F00:ROOT | 1 | 1 | 414.22us | 414.22us | |
> | 0 B | 0 B | |
> | 11:TOP-N | 1 | 1 | 10.03ms | 10.03ms | 100 | 100
> | 7.04 MB | 9.38 KB | |
> | 10:ANALYTIC | 1 | 1 | 101.26ms | 101.26ms | 514.59K | 15.09M
> | 11.02 MB | 4.00 MB | |
> | 09:SORT | 1 | 1 | 334.83ms | 334.83ms | 514.59K | 15.09M
> | 80.02 MB | 38.00 MB | |
> | 08:AGGREGATE | 1 | 1 | 221.69ms | 221.69ms | 514.59K | 15.09M
> | 114.16 MB | 1.42 GB | FINALIZE |
> | 07:AGGREGATE | 1 | 1 | 811.38ms | 811.38ms | 514.59K | 15.09M
> | 257.68 MB | 16.98 GB | FINALIZE |
> | 06:HASH JOIN | 1 | 1 | 30.35ms | 30.35ms | 535.97K | 2.88M
> | 3.33 MB | 2.88 MB | INNER JOIN |
> | |--03:SCAN HDFS | 1 | 1 | 2.71ms | 2.71ms | 18.00K | 18.00K
> | 2.63 MB | 80.00 MB | tpcds_parquet.item |
> | 05:HASH JOIN | 1 | 1 | 7.48ms | 7.48ms | 535.97K | 2.88M
> | 2.07 MB | 1.94 MB | INNER JOIN |
> | |--02:SCAN HDFS | 1 | 1 | 869.04us | 869.04us | 12 | 12
> | 69.00 KB | 32.00 MB | tpcds_parquet.store |
> | 04:HASH JOIN | 1 | 1 | 9.77ms | 9.77ms | 535.97K | 2.88M
> | 2.01 MB | 1.94 MB | INNER JOIN |
> | |--01:SCAN HDFS | 1 | 1 | 2.83ms | 2.83ms | 365 | 7.30K
> | 1.33 MB | 80.00 MB | tpcds_parquet.date_dim |
> | 00:SCAN HDFS | 1 | 1 | 8.31ms | 8.31ms | 535.97K | 2.88M
> | 14.35 MB | 64.00 MB | tpcds_parquet.store_sales |
> +-----------------+--------+-------+----------+----------+---------+------------+-----------+---------------+---------------------------+
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
