hvanhovell opened a new pull request, #39627: URL: https://github.com/apache/spark/pull/39627
### What changes were proposed in this pull request? This PR makes `RowEncoder` produce an `AgnosticEncoder`. The expression generation for these encoders is moved to `ScalaReflection` (this will be moved out in a subsequent PR). The generated serializer and deserializer expressions will slightly change for both schema and type based encoders. These are not semantically different from the old expressions. Concretely the following changes have been introduced: - There is more type validation in maps/arrays/seqs for type based encoders. This should be a positive change, since it disallows users to pass wrong data through erasure hackd. - Array/Seq serialization is a bit more strict. In the old scenario it was possible to pass in sequences/arrays with the wrong type and/or nullability. ### Why are the changes needed? For the Spark Connect Scala Client we also want to be able to use `Row` based results. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? This is a refactoring so mostly existing tests. I have added test to the catalyst tests that triggered failures downstream (typed arrays in `WrappedArray` & `Seq[_]` change in Scala 2.13). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
