Github user bogdanrdc commented on the issue:
https://github.com/apache/spark/pull/22205
To justify copying ConvertToLocalRelation earlier: this code produces 10 ms
after this change vs 45 ms before:
```
val manyCols = (0 until 1000).map { i => s"value as col$i"}
val local = Seq(1, 2, 3).toDS().selectExpr(manyCols: _*).where($"col3"
> 4)
val plan = local.queryExecution.analyzed
val numIter = 1000
val start = System.nanoTime()
for (i <- 0 until numIter) {
spark.sessionState.optimizer.execute(plan)
}
val durationMs = (System.nanoTime() - start) / numIter / 1000000
println(s"duration $durationMs")
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
