[
https://issues.apache.org/jira/browse/SPARK-28621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-28621:
-------------------------------
Summary: CheckCartesianProducts may throw some error which mismatch
generated physical plan (was: may throw some mismatching error )
> CheckCartesianProducts may throw some error which mismatch generated physical
> plan
> ----------------------------------------------------------------------------------
>
> Key: SPARK-28621
> URL: https://issues.apache.org/jira/browse/SPARK-28621
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.3
> Reporter: Weichen Xu
> Priority: Major
>
> CheckCartesianProducts check logical plan which mismatch the physical plan.
> So when option "spark.sql.crossJoin.enabled" turn on, it may throw some
> mismatching error which make user confusing.
>
> There're some cases:
> 1) The sql optimizer will possibly do join reordering. So that if one join
> marked by "CROSS JOIN" syntax, it may be reordered in the physical plan. So
> here CheckCartesianProducts only check logical plan which will possibly
> mismatch the physical plan.
> 2) Other places which may be inconsistent with physical plan:
> providing:
> {code:java}
> spark.range(2).createOrReplaceTempView("sm1") // can be broadcast
> spark.range(50000000).createOrReplaceTempView("bg1") // cannot be broadcast
> spark.range(60000000).createOrReplaceTempView("bg2") // cannot be broadcast
> {code}
> and suppose `spark.sql.crossJoin.enabled=false` (by default)
> 1) Some join could be convert to broadcast nested loop join, but
> CheckCartesianProducts raise error. e.g.
> {code:java}
> select sm1.id, bg1.id from bg1 join sm1 where sm1.id < bg1.id
> {code}
> 2) Some join will run by CartesianJoin but CheckCartesianProducts DO NOT
> raise error. e.g.
> {code:java}
> select bg1.id, bg2.id from bg1 join bg2 where bg1.id < bg2.id
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
