peterxcli commented on issue #4515: URL: https://github.com/apache/datafusion-comet/issues/4515#issuecomment-5447362029
Filed the upstream issue for the `collect_set` row in the table (it also covers `collect_list`): apache/datafusion#24741. Two additions to what the table records: - The drift is not specific to `Int32` — running the `collect_set` SQL file suites on current `main` (Spark 4.2 profile), the shuffle writer's alignment cast fires for every element type tested: `Int8`/`Int16`/`Int32`/`Int64`, `Boolean`, `Utf8`, `Binary`, `Decimal128(10,2)`, `Date32`, and `Timestamp(us, "UTC")`. Root cause is `SparkCollectList`/`SparkCollectSet::return_type` hardcoding a nullable list element while Catalyst declares `containsNull = false` (both functions always drop nulls, so non-nullable is correct). - For nested inputs there is a second inconsistency upstream: `return_type` clones the input type verbatim (preserving non-nullable inner fields) while the accumulators produce all-nullable nested fields — that is what Comet's `coerce_collect_child_nullability` workaround in the native planner papers over. If upstream fixes both, that workaround can be dropped along with the per-batch cast. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
