kevinwilfong opened a new pull request, #10443: URL: https://github.com/apache/incubator-gluten/pull/10443
## What changes are proposed in this pull request? This adds general support for casting complex types. Previously, casting of complex types was limited to arrays where the elements were casted from/to specific types. This change allows casting of arrays/maps/rows as long as all conversions of child types are generally supported in Gluten, including support for nested complex types. From testing, the main blocker to supporting this was the different failure semantics between Spark and Velox. In particular if a child conversion failed Velox would return NULL for the entire complex value, while Spark would simply make that child NULL within the larger complex value. This was addressed in https://github.com/facebookincubator/velox/pull/14451 ## How was this patch tested? Spark already has existing tests for casting complex types in org.apache.spark.sql.catalyst.expressions.CastSuite I verified these do not fallback, unless Gluten doesn't support casting the child types, and those tests pass. I noticed there weren't many tests in Spark, and many of them involved casting child types Gluten doesn't support, so I added additional unit tests in org.apache.gluten.expressions.GlutenCastSuite I ran these tests against Spark versions 3.2-3.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
