LuciferYang commented on a change in pull request #29434:
URL: https://github.com/apache/spark/pull/29434#discussion_r473579507
##########
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/StarJoinCostBasedReorderSuite.scala
##########
@@ -329,7 +329,7 @@ class StarJoinCostBasedReorderSuite extends PlanTest with
StatsEstimationTestBas
//
// Number of generated plans: 46 (vs. 82)
val query =
-
d1.join(t3).join(t4).join(f1).join(d2).join(t5).join(t6).join(d3).join(t1).join(t2)
+
d1.join(t3).join(t4).join(f1).join(d3).join(d2).join(t5).join(t6).join(t1).join(t2)
Review comment:
@cloud-fan It seems that using "LinkedHashMap" does not make the results
uniform, still two possibilities optimization results. The level 0 `foundPlans`
is produced by `items.zipWithIndex`, the `items` produced by
https://github.com/apache/spark/blob/3092527f7557b64ff9a5bedadfac8bb2f189a9b4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala#L78-L91
If `itemIndex` has different order with different input order, use
`LinkedHashMap` should be possible to guarantee the same optimization result
for the same input order with Scala 2.12 and Scala 2.13, unlikely to guarantee
the same optimization results for different input order, If the `items` is
ordered, the output may be fully guaranteed, but I haven't verified it yet.
Now I am counting the optimization results of full permutation of 10 tables
involved this case as input, divided into
- Scala 2.12 use HashMap
- Scala 2.12 use LinkedHashMap
- Scala 2.13 use HashMap
- Scala 2.13 use LinkedHashMap
The distribution of optimization results will be fed back later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
