Stavros Kontopoulos created SPARK-17959:

             Summary: spark.sql.join.preferSortMergeJoin has no effect for 
simple join
                 Key: SPARK-17959
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.0.1
            Reporter: Stavros Kontopoulos

Example code:   
val df = spark.sparkContext.parallelize(List(("A", 10, "dss@s1"), ("A", 20, 
      ("B", 1, "dss@qqa"), ("B", 2, "dss@qqb"))).toDF("Group", "Amount", 
      .where($"a.Group" === $"b.Group")

I always get the SortMerge strategy even if i set 
spark.sql.join.preferSortMergeJoin to false since:
sinzeInBytes = 2^63-1
and thus:
condition here:
is always false...

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to