Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21952
This is serious and we should fix it before Spark 2.4.
For the benchmark, I have 2 questions:
1. will the regression caused by the df cache? we can run
`df.queryExecution.toRdd.foreach(_ => ())` to verify the cache reading
performance.
2. is this related to array type only? if we remove the array column, does
the regression disappear?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
