Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/20345
  
    When I re-checked the code of the `ReorderJoin` rule, I found 
`ExtractFiltersAndInnerJoins` was applied into a join tree multiple times. IIUC 
we can use `OrderedJoin` to avoid this case though, any reason not to do so? I 
just made [a trivial 
patch](https://github.com/apache/spark/compare/master...maropu:GuardReAppliedReorder)
 for that and checked the metrics for the rule;
    ```
    scala> import org.apache.spark.sql.catalyst.rules.RuleExecutor
    scala> :paste
    RuleExecutor.resetMetrics()
    val numJoins = 9
    spark.range(1).selectExpr((0 until numJoins).map { i => s"id AS k$i" }: 
_*).write.saveAsTable("t")
    (0 until numJoins).foreach { i =>
      spark.range(1).selectExpr(s"id AS k$i").write.saveAsTable(s"t$i")
    }
    val joinSql = s"""
      SELECT *
        FROM t, ${ (0 until numJoins).map(i => s"t$i").mkString(", ") }
        WHERE ${(0 until numJoins).map(i => s"t.k$i = t$i.k$i").mkString(" AND 
")}
    """
    sql(joinSql).explain
    println(RuleExecutor.dumpTimeSpent())
    
    -- master
    Rule                                                 Effective Time / Total 
Time  Effective Runs / Total Runs    
    org.apache.spark.sql.catalyst.optimizer.ReorderJoin  97010505 / 126269245   
      2 / 26  
    
    -- w/ the patch
    Rule                                                 Effective Time / Total 
Time  Effective Runs / Total Runs    
    org.apache.spark.sql.catalyst.optimizer.ReorderJoin  20498471 / 34859643    
      2 / 26 
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to