Xiangrui Meng created SPARK-18390:
-------------------------------------
Summary: Optimized plan tried to use Cartesian join when it is not
enabled
Key: SPARK-18390
URL: https://issues.apache.org/jira/browse/SPARK-18390
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.0.1
Reporter: Xiangrui Meng
{code}
val df2 = spark.range(1e9.toInt).withColumn("one", lit(1))
val df3 = spark.range(1e9.toInt)
df3.join(df2, df3("id") === df2("one")).count()
{code}
throws
{noformat}
org.apache.spark.sql.AnalysisException: Cartesian joins could be prohibitively
expensive and are disabled by default. To explicitly enable them, please set
spark.sql.crossJoin.enabled = true;
{noformat}
This is probably not the right behavior because it was not the user who
suggested using cartesian product. SQL picked it while knowing it is not
enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
