Tim Armstrong has uploaded a new patch set (#2).
Change subject: IMPALA-4008: don't bake in hash table and hash join pointers
......................................................................
IMPALA-4008: don't bake in hash table and hash join pointers
This fixes some of the cases where memory addresses are baked into
codegen'd code.
Testing:
Ran exhaustive build.
Perf:
Ran a local perf run. No significant changes. I was able to see some
small improvements on microbenchmarks.
+-----------+-----------------------+---------+------------+------------+----------------+
| Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) |
Delta(GeoMean) |
+-----------+-----------------------+---------+------------+------------+----------------+
| TPCH(_20) | parquet / none / none | 9.07 | +0.46% | 5.88 |
+0.34% |
+-----------+-----------------------+---------+------------+------------+----------------+
+-----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------------+-------+
| Workload | Query | File Format | Avg(s) | Base Avg(s) |
Delta(Avg) | StdDev(%) | Base StdDev(%) | Num Clients | Iters |
+-----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------------+-------+
| TPCH(_20) | TPCH-Q2 | parquet / none / none | 2.12 | 1.89 |
+12.29% | * 10.85% * | * 20.30% * | 1 | 10 |
| TPCH(_20) | TPCH-Q13 | parquet / none / none | 9.84 | 9.34 |
+5.39% | 9.01% | 3.79% | 1 | 10 |
| TPCH(_20) | TPCH-Q17 | parquet / none / none | 14.61 | 14.19 |
+2.97% | 2.15% | 1.52% | 1 | 10 |
| TPCH(_20) | TPCH-Q18 | parquet / none / none | 14.76 | 14.35 |
+2.82% | 3.20% | 2.64% | 1 | 10 |
| TPCH(_20) | TPCH-Q9 | parquet / none / none | 13.72 | 13.54 |
+1.30% | 1.75% | 0.70% | 1 | 10 |
| TPCH(_20) | TPCH-Q8 | parquet / none / none | 5.71 | 5.64 |
+1.30% | 1.21% | 1.23% | 1 | 10 |
| TPCH(_20) | TPCH-Q19 | parquet / none / none | 47.35 | 46.75 |
+1.28% | 2.39% | 1.88% | 1 | 10 |
| TPCH(_20) | TPCH-Q5 | parquet / none / none | 4.57 | 4.52 |
+1.20% | 1.30% | 0.88% | 1 | 10 |
| TPCH(_20) | TPCH-Q16 | parquet / none / none | 2.07 | 2.05 |
+1.12% | 2.59% | 1.79% | 1 | 10 |
| TPCH(_20) | TPCH-Q11 | parquet / none / none | 1.45 | 1.45 |
+0.15% | 2.69% | 2.06% | 1 | 10 |
| TPCH(_20) | TPCH-Q3 | parquet / none / none | 4.65 | 4.65 |
-0.09% | 2.12% | 2.17% | 1 | 10 |
| TPCH(_20) | TPCH-Q4 | parquet / none / none | 3.22 | 3.23 |
-0.26% | 1.03% | 1.33% | 1 | 10 |
| TPCH(_20) | TPCH-Q7 | parquet / none / none | 15.84 | 15.92 |
-0.50% | 0.91% | 1.15% | 1 | 10 |
| TPCH(_20) | TPCH-Q14 | parquet / none / none | 3.29 | 3.31 |
-0.59% | 3.31% | 1.58% | 1 | 10 |
| TPCH(_20) | TPCH-Q22 | parquet / none / none | 2.65 | 2.67 |
-0.78% | 3.03% | 1.46% | 1 | 10 |
| TPCH(_20) | TPCH-Q15 | parquet / none / none | 4.50 | 4.55 |
-1.19% | 2.87% | 2.45% | 1 | 10 |
| TPCH(_20) | TPCH-Q20 | parquet / none / none | 3.84 | 3.91 |
-1.76% | 2.20% | 1.94% | 1 | 10 |
| TPCH(_20) | TPCH-Q10 | parquet / none / none | 5.58 | 5.70 |
-2.00% | 1.01% | 1.79% | 1 | 10 |
| TPCH(_20) | TPCH-Q21 | parquet / none / none | 22.84 | 23.42 |
-2.47% | 0.68% | 0.56% | 1 | 10 |
| TPCH(_20) | TPCH-Q1 | parquet / none / none | 11.25 | 11.60 |
-3.06% | 0.48% | 1.74% | 1 | 10 |
| TPCH(_20) | TPCH-Q12 | parquet / none / none | 3.81 | 3.98 |
-4.38% | 1.62% | 1.14% | 1 | 10 |
| TPCH(_20) | TPCH-Q6 | parquet / none / none | 1.94 | 2.04 |
-4.85% | 2.40% | 1.58% | 1 | 10 |
+-----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------------+-------+
+--------------------+-----------------------+---------+------------+------------+----------------+
| Workload | File Format | Avg (s) | Delta(Avg) |
GeoMean(s) | Delta(GeoMean) |
+--------------------+-----------------------+---------+------------+------------+----------------+
| TARGETED-PERF(_20) | parquet / none / none | 8.17 | -1.66% | 2.96
| -1.48% |
+--------------------+-----------------------+---------+------------+------------+----------------+
+--------------------+--------------------------------------------------------+-----------------------+--------+-------------+------------+------------+----------------+-------------+-------+
| Workload | Query
| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) |
Base StdDev(%) | Num Clients | Iters |
+--------------------+--------------------------------------------------------+-----------------------+--------+-------------+------------+------------+----------------+-------------+-------+
| TARGETED-PERF(_20) | primitive_topn_bigint
| parquet / none / none | 3.32 | 2.87 | +15.31% | * 17.22% * |
1.64% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_AGG-Q4
| parquet / none / none | 7.07 | 6.61 | +6.93% | * 15.59% * |
5.08% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_AGG-Q1
| parquet / none / none | 1.18 | 1.12 | +5.57% | 1.94% |
2.96% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_empty_build_join_1
| parquet / none / none | 10.75 | 10.47 | +2.76% | 1.15% |
0.94% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_top-n_all
| parquet / none / none | 24.30 | 23.85 | +1.87% | 1.40% |
0.82% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_broadcast_join_2
| parquet / none / none | 2.45 | 2.42 | +1.38% | 1.93% |
1.33% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_filter_bigint_selective
| parquet / none / none | 0.57 | 0.57 | +1.16% | 3.66% |
4.19% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_STRING-Q3
| parquet / none / none | 1.70 | 1.68 | +1.06% | 1.88% |
2.69% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_broadcast_join_3
| parquet / none / none | 4.15 | 4.13 | +0.47% | 1.27% |
1.28% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_broadcast_join_1
| parquet / none / none | 1.46 | 1.46 | +0.32% | 1.68% |
2.43% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_orderby_all
| parquet / none / none | 12.92 | 12.89 | +0.22% | 1.69% |
1.12% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_STRING-Q4
| parquet / none / none | 1.68 | 1.67 | +0.21% | 2.31% |
2.44% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_STRING-Q7
| parquet / none / none | 3.35 | 3.35 | +0.14% | 1.10% |
1.84% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_conjunct_ordering_4
| parquet / none / none | 0.46 | 0.46 | +0.07% | 0.29% |
0.27% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_groupby_bigint_pk
| parquet / none / none | 35.99 | 36.04 | -0.13% | 8.55% |
6.95% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_AGG-Q6
| parquet / none / none | 0.97 | 0.97 | -0.16% | 2.68% |
2.66% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_filter_decimal_selective
| parquet / none / none | 0.84 | 0.84 | -0.31% | 2.85% |
3.58% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_shuffle_join_union_all_with_groupby
| parquet / none / none | 20.46 | 20.53 | -0.34% | 0.60% |
0.60% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_AGG-Q7
| parquet / none / none | 0.98 | 0.98 | -0.44% | 2.78% |
2.33% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_exchange_shuffle
| parquet / none / none | 25.21 | 25.34 | -0.49% | 1.94% |
1.30% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_filter_string_like
| parquet / none / none | 6.27 | 6.31 | -0.56% | 0.59% |
0.43% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_STRING-Q5
| parquet / none / none | 1.96 | 1.97 | -0.60% | 1.98% |
1.25% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_filter_bigint_non_selective
| parquet / none / none | 0.53 | 0.53 | -0.71% | 2.64% |
0.28% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_exchange_broadcast
| parquet / none / none | 14.77 | 14.97 | -1.33% | 2.59% |
2.39% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_LIMIT-Q1
| parquet / none / none | 0.01 | 0.01 | -1.40% | 3.77% |
3.51% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_orderby_bigint
| parquet / none / none | 2.58 | 2.61 | -1.44% | 2.85% |
1.04% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_STRING-Q1
| parquet / none / none | 1.47 | 1.49 | -1.63% | 2.79% |
1.62% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_filter_decimal_non_selective
| parquet / none / none | 0.79 | 0.81 | -1.65% | 1.91% |
2.67% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_STRING-Q2
| parquet / none / none | 1.56 | 1.58 | -1.73% | 2.86% |
2.57% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_STRING-Q6
| parquet / none / none | 4.48 | 4.59 | -2.22% | 1.21% |
0.85% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_conjunct_ordering_5
| parquet / none / none | 11.80 | 12.10 | -2.50% | 3.06% |
2.09% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_conjunct_ordering_1
| parquet / none / none | 7.96 | 8.18 | -2.58% | 1.69% |
2.17% | 1 | 10 |
| TARGETED-PERF(_20) |
primitive_shuffle_join_one_to_many_string_with_groupby | parquet / none / none
| 93.11 | 95.78 | -2.78% | 1.07% | 1.00% | 1
| 10 |
| TARGETED-PERF(_20) | primitive_filter_string_selective
| parquet / none / none | 0.88 | 0.91 | -3.03% | 4.68% |
5.34% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_AGG-Q5
| parquet / none / none | 2.44 | 2.52 | -3.35% | 2.39% |
2.04% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_groupby_bigint_highndv
| parquet / none / none | 9.16 | 9.50 | -3.59% | 0.98% |
1.56% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_conjunct_ordering_3
| parquet / none / none | 1.44 | 1.50 | -4.14% | 1.10% |
1.08% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_filter_string_non_selective
| parquet / none / none | 0.89 | 0.93 | -4.31% | 7.20% |
5.12% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_groupby_decimal_highndv
| parquet / none / none | 13.98 | 14.94 | -6.40% | * 11.24% * |
8.66% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_AGG-Q2
| parquet / none / none | 2.87 | 3.13 | -8.36% | 0.57% |
1.59% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_groupby_bigint_lowndv
| parquet / none / none | 1.48 | 1.62 | -8.91% | 1.62% |
1.28% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_conjunct_ordering_2
| parquet / none / none | 13.68 | 15.05 | -9.14% | 2.91% |
1.83% | 1 | 10 |
| TARGETED-PERF(_20) | primitive_groupby_decimal_lowndv.test
| parquet / none / none | 1.47 | 1.61 | -9.26% | 0.19% |
1.50% | 1 | 10 |
| TARGETED-PERF(_20) | PERF_AGG-Q3
| parquet / none / none | 4.25 | 4.82 | -11.82% | 0.79% |
0.84% | 1 | 10 |
+--------------------+--------------------------------------------------------+-----------------------+--------+-------------+------------+------------+----------------+-------------+-------+
Change-Id: Ie353666dbb5c958f0094d169306fe930ec3014c5
---
M be/src/exec/hash-table.cc
M be/src/exec/hash-table.h
M be/src/exec/hash-table.inline.h
M be/src/exec/partitioned-aggregation-node.cc
M be/src/exec/partitioned-hash-join-node.cc
5 files changed, 224 insertions(+), 239 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/4326/2
--
To view, visit http://gerrit.cloudera.org:8080/4326
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie353666dbb5c958f0094d169306fe930ec3014c5
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <[email protected]>
Gerrit-Reviewer: Michael Ho
