Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/19752
@gatorsmile I added a test case to check that the execution plan is
`WholeStageCodegenExec` as expected. I also made some performance test using
almost the same code, ie.:
```
val N = 30
val nRows = 1000000
var expr1 = when($"id" === lit(0), 0)
var expr2 = when($"id" === lit(0), 10)
(1 to N).foreach { i =>
expr1 = expr1.when($"id" === lit(i), -i)
expr2 = expr2.when($"id" === lit(i + 10), i)
}
time { spark.range(nRows).select(expr1.as("c1"),
expr2.otherwise(0).as("c2")).sort("c1").show }
```
before this PR, it takes on average 1091.690996ms. After the PR, it takes
on average 106.894443ns.
Actually there is a problem which is fixed in #18641 and it is not fixed
here, ie. when the code contains deeply nested exceptions, the 64KB limit
exception can still happen. But this should be handled in a more generic way in
#19813.
@kiszk What do you think?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
