andygrove opened a new pull request, #7649:
URL: https://github.com/apache/datafusion-comet/pull/7649
## Which issue does this PR close?
Backport of #5903 to `branch-1.2`. Closes #5239 on this branch.
## Rationale for this change
Native `collect_set` and `collect_list` over a struct with a non-nullable
nested field fail the query with `column types must match schema types` when
the aggregate emits its state. The Scala side declared the partial state's
element type with the child's own nullability, while the native planner
normalizes collect arguments to nullable nested fields, so the two schemas
disagree. `branch-1.2` has the same code. See #5903 and #5239 for the full
analysis.
## What changes are included in this PR?
Cherry-pick of 92be8e8abb (`git cherry-pick -x`) with no conflicts and no
adaptations. It declares the `CollectSet` and `CollectList` state element type
with `asNullable` in `CometBaseAggregate`, and includes the source PR's SQL
file tests (`collect_list.sql`, `collect_set.sql`) and the
`CometAggregateSuite` test. The diff against the source commit is identical
apart from line numbers and blob hashes.
Other release branches: `branch-1.2` is the newest release branch, so there
is no newer one to port to.
## How are these changes tested?
On this branch, with JDK 17 and the default Spark profile only:
`CometAggregateSuite` "grouped collect_list and collect_set over non-nullable
nested fields" and `CometSqlFileTestSuite` for `collect_list` and `collect_set`
pass against a native library built from this branch (11 tests, 0 failures).
With `operators.scala` restored to `branch-1.2`'s version, the
`CometAggregateSuite` test and both `collect_list.sql` runs fail with `column
types must match schema types, expected List(Struct("flag": non-null Boolean,
...)) but found List(Struct("flag": Boolean, ...))`, so the branch has the bug
and the tests catch it. `collect_set.sql` passes either way on this branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]