andygrove opened a new pull request, #3795: URL: https://github.com/apache/datafusion-comet/pull/3795
## Summary Two fixes to `CometArrayAppend`: **Return type mismatch.** The serde was passing `ArrayType(elementType)` (always `containsNull=true`) as the return type to DataFusion. Spark's actual return type is `if (right.nullable) left.dataType.asNullable else left.dataType` — meaning when the element is non-nullable, the input array's `containsNull` flag is preserved. Fixed by passing `expr.dataType` directly, which Spark has already computed correctly. **Incorrect `Incompatible` classification.** `CometArrayAppend` was marked `Incompatible(None)` with no explanation. The `CaseWhen(IsNotNull(arr), array_append(arr, elem), null)` wrapper already handles the only genuine incompatibility (DataFusion's `array_append` does not preserve null top-level array rows on its own). With the return type fix, the expression now matches Spark's behavior fully. Changed to `Compatible()`. **Test cleanup.** Removed `spark.comet.expression.ArrayAppend.allowIncompatible=true` from `array_append.sql` since `Compatible` expressions don't need it. Added a comment explaining why `ArrayInsert.allowIncompatible=true` is still needed (on Spark 4.0, `array_append` is a `RuntimeReplaceable` that rewrites to `array_insert(-1)`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
