ErikBPF commented on code in PR #5903:
URL: https://github.com/apache/datafusion-comet/pull/5903#discussion_r4365035600
##########
spark/src/main/scala/org/apache/spark/sql/comet/operators.scala:
##########
@@ -2488,11 +2488,12 @@ trait CometBaseAggregate[T <: BaseAggregateExec]
extends CometOperatorSerde[T] {
val bufferAttrs = aggFunc.aggBufferAttributes
aggFunc match {
case cs: CollectSet =>
- val elementType = cs.children.head.dataType
+ // Match the native planner's collect argument normalization,
including nested fields.
+ val elementType = cs.children.head.dataType.asNullable
val nativeStateType = ArrayType(elementType, containsNull = true)
output(bufferIdx) = output(bufferIdx).withDataType(nativeStateType)
case cl: CollectList =>
- val elementType = cl.children.head.dataType
+ val elementType = cl.children.head.dataType.asNullable
val nativeStateType = ArrayType(elementType, containsNull = true)
output(bufferIdx) = output(bufferIdx).withDataType(nativeStateType)
Review Comment:
Done in `8c5700c5a`. Both branches are now one, exactly as you proposed:
```scala
case _: CollectSet | _: CollectList =>
// Match the native planner's collect argument normalization, including
nested fields.
val elementType = aggFunc.children.head.dataType.asNullable
val nativeStateType = ArrayType(elementType, containsNull = true)
output(bufferIdx) = output(bufferIdx).withDataType(nativeStateType)
```
`rg 'case _: CollectSet \| _: CollectList'` matches once, and no `case cs:
CollectSet` / `case cl: CollectList` remains. The commit is rebased onto
current `main` (`5a50bd6f2`), so the diff is the PR's change plus the
test/coverage edits, not a merge of newer work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
