Hello Aman Sinha, Fang-Yu Rao, Riza Suminto, Joe McDonnell, Michael Smith,
Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23315
to look at the new patch set (#5).
Change subject: IMPALA-14102: [part 3] Calcite Planner: Customize
LoptOptimizeJoinRule
......................................................................
IMPALA-14102: [part 3] Calcite Planner: Customize LoptOptimizeJoinRule
This commit provides a customized version of the LoptOptimizeJoinRule.
The ImpalaLoptOptimizeJoin class was grabbed from
https://github.com/apache/calcite/blob/calcite-1.37.0/core/src/main/java/org/apache/...
/calcite/rel/rules/LoptOptimizeJoinRule.java
The use of runtime filters within Impala throw a wrench into the join
optimization
ordering. We should factor this in to come up with much better join ordering
choices.
While this first attempt in this file does not do this, it does put some
band-aids
on cases where the Calcite algorithm can go awry. Namely, the following changes
are made from the original Calcite algorithm:
- When attempting the check of whether to push down a join versus keeping the
current
join, the epsilon on the tiebreaker has been made a little wider. If it does
go
to the tiebreaker, it compares the "leftest most" cardinality on the 2
choices. This
allows a more effective runtime filter to be used, if found.
- "Complex" trees through a bit of a wrench into runtime filtering. A "simple"
tree is
defined here as a tree with only projects and filters. If a tree is complex,
the row
count will have gone through some major transformation, and it isn't quite
apparent
based on the row count at the top level that a runtime filter could have a
major
impact. It especially caused problems when the complex tree was processed in
between
simple trees. Because the row count showed up low, a swap might occur with a
simple
tree where it shouldn't have. In practice, it helped extremely to delay the
processing of complex trees until after all the simple trees have been
processed.
- Similar to the last point, we also do not want to swap the left and right side
if we detect a complex tree that has its "leftest" most input with a high
cardinality because a runtime filter could help improve performance greatly.
To sum up: These are band-aid patches that take guesses as to when a runtime
filter will be used. A more complete analysis should be done at some point, but
these changes should improve performance, as tested on the tpcds queries.
Change-Id: I95bba7bafcc7dd86f17921c11740ac349daac6c7
---
R
java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ImpalaLoptOptimizeJoinRule.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q01.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q02.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q03.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q04.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q05.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q06.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q07.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q08.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q09.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q10a.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q11.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q12.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q13.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q14a.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q14b.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q16.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q17.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q18.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q19.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q20.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q21.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q22.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q23a.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q23b.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q24a.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q24b.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q25.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q26.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q27.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q28.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q29.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q30.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q31.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q32.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q33.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q34.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q35a.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q36.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q37.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q38.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q39a.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q39b.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q40.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q41.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q42.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q43-verbose.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q43.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q44.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q45.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q46.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q47.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q48.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q49.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q50.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q52.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q53.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q54.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q55.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q56.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q57.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q58.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q59.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q60.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q61.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q62.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q63.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q64.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q65.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q66.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q67.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q68.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q69.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q70.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q72.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q73.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q74.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q75.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q77.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q80.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q81.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q82.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q83.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q84.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q85.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q86.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q87.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q88.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q89.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q90.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q91.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q92.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q94.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q95.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q96.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q98.test
97 files changed, 58,709 insertions(+), 57,963 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/23315/5
--
To view, visit http://gerrit.cloudera.org:8080/23315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I95bba7bafcc7dd86f17921c11740ac349daac6c7
Gerrit-Change-Number: 23315
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Fang-Yu Rao <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
