michaelmitchell-bit opened a new pull request, #4822: URL: https://github.com/apache/datafusion-comet/pull/4822
## Which issue does this PR close? Closes #4788. ## Rationale for this change Spark `flatten` returns `NULL` for an output row when any nested sub-array in that row is `NULL`. Comet was using DataFusion's `flatten`, which preserves only the outer array null bitmap and can return incorrect values when an inner sub-array is null. ## What changes are included in this PR? This adds a Spark-compatible `flatten` implementation for Comet that preserves DataFusion's offset/value flattening while computing Spark-compatible output nulls. It also registers the new UDF and extends the existing `flatten.sql` coverage to compare Comet against Spark for null sub-array cases. ## How are these changes tested? Ran: ``` cargo test -p datafusion-comet-spark-expr array_funcs::flatten cargo build --release ./mvnw test -Prelease -Dtest=none -Dsuites="org.apache.comet.CometSqlFileTestSuite flatten" -Dscalastyle.skip=true ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
