andygrove opened a new pull request, #7649:
URL: https://github.com/apache/datafusion-comet/pull/7649

   ## Which issue does this PR close?
   
   Backport of #5903 to `branch-1.2`. Closes #5239 on this branch.
   
   ## Rationale for this change
   
   Native `collect_set` and `collect_list` over a struct with a non-nullable 
nested field fail the query with `column types must match schema types` when 
the aggregate emits its state. The Scala side declared the partial state's 
element type with the child's own nullability, while the native planner 
normalizes collect arguments to nullable nested fields, so the two schemas 
disagree. `branch-1.2` has the same code. See #5903 and #5239 for the full 
analysis.
   
   ## What changes are included in this PR?
   
   Cherry-pick of 92be8e8abb (`git cherry-pick -x`) with no conflicts and no 
adaptations. It declares the `CollectSet` and `CollectList` state element type 
with `asNullable` in `CometBaseAggregate`, and includes the source PR's SQL 
file tests (`collect_list.sql`, `collect_set.sql`) and the 
`CometAggregateSuite` test. The diff against the source commit is identical 
apart from line numbers and blob hashes.
   
   Other release branches: `branch-1.2` is the newest release branch, so there 
is no newer one to port to.
   
   ## How are these changes tested?
   
   On this branch, with JDK 17 and the default Spark profile only: 
`CometAggregateSuite` "grouped collect_list and collect_set over non-nullable 
nested fields" and `CometSqlFileTestSuite` for `collect_list` and `collect_set` 
pass against a native library built from this branch (11 tests, 0 failures). 
With `operators.scala` restored to `branch-1.2`'s version, the 
`CometAggregateSuite` test and both `collect_list.sql` runs fail with `column 
types must match schema types, expected List(Struct("flag": non-null Boolean, 
...)) but found List(Struct("flag": Boolean, ...))`, so the branch has the bug 
and the tests catch it. `collect_set.sql` passes either way on this branch.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to