cloud-fan commented on pull request #31281: URL: https://github.com/apache/spark/pull/31281#issuecomment-767007988
This is a much simpler solution and makes sense to me: almost all databases enforce the data constraints at the write side, Spark should expect the data sources or vendors to enforce it at the write side, instead of doing read side checks. This changes the storage of char type (because of the write side padding), which makes more sense to better preserve the char type semantic. Usually, we can't change storage afterward, but fortunately, Spark 3.1 is not released yet. It's a good chance to get this merged and fix the perf regression regarding CBO and predicate pushdown. Thanks, merging to master/3.1! ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
