Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16480 )
Change subject: IMPALA-10179: After join inversion inherit the left child's parallelism ...................................................................... IMPALA-10179: After join inversion inherit the left child's parallelism The planner makes a cost based decision to invert a join - i.e swap left and right child - based on cardinality, row size and parallelism of both sides. The join node's numNodes and numInstances attributes were originally set based on the left child's values. These were not getting reset after the inversion and the plan shows the original value, although the backend actually uses the correct value from the left input..so this mainly affects planner estimates. This patch fixes this behavior by inheriting the new left child's parallelism (numNodes and numInstances) after inversion. A common case where the join inputs get flipped is when the original query has a left semi join (IN, EXISTS etc.) and the subquery has a large cardinality compared to the outer query. This gets converted to a right semi join. This pattern occurs for example in TPC-DS q10, q35, q69. Testing: - Updated the plans for several TPC-DS queries to reflect the modified parallelism. 11 queries had the join fragment 'instances' increase from either 1 or 2 to 6 (for planner tests hosts = 3 and mt_dop = 2). There were also 4 queries where the parallelism reduced. Based on an initial check, these changes seem expected. TODO: Check if this reduction is expected based on the costing. One option may be to use a max(left parallelism, right parallelism) value to avoid unintended side effects. - Couple of TPC-H queries also had increase in parallelism. - Ran e2e TPC-DS queries. - TODO: test performance on larger scale factor Change-Id: I01ba559034ad76f7ccee41f237c81b29d8402950 Reviewed-on: http://gerrit.cloudera.org:8080/16480 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q03.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q04.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q06.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q07.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q08.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q09.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q11.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q13.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q17.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q19.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q24a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q24b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q25.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q27.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q28.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q29.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q31.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q34.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q36.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q37.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q38.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q42.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q43.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q44.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q46.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q47.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q48.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q50.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q51.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q52.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q53.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q55.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q58.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q59.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q61.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q63.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q64.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q65.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q66.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q68.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q69.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q70.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q72.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q73.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q74.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q78.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q79.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q82.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q85.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q87.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q88.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q89.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q93.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q96.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q97.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q98.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test 78 files changed, 2,771 insertions(+), 2,760 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I01ba559034ad76f7ccee41f237c81b29d8402950 Gerrit-Change-Number: 16480 Gerrit-PatchSet: 4 Gerrit-Owner: Aman Sinha <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
