Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/21952
Hi @dbtsai , nice catch!
I think we can also check the nullability here:
https://github.com/apache/spark/pull/21952/files#diff-01fea32e6ec6bcf6f34d06282e08705aR160
If the input data is from data source, I doubt the improvement of this PR.
As the data schema is always nullable for data source:
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L379
Anyway we should add these checks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
