jubins commented on code in PR #59094:
URL: https://github.com/apache/spark/pull/59094#discussion_r4148233008
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala:
##########
@@ -905,6 +905,10 @@ case class ApproxTopKCombine(
*/
override def update(buffer: CombineInternal[Any], input: InternalRow):
CombineInternal[Any] = {
val inputState = state.eval(input).asInstanceOf[InternalRow]
+ if (inputState == null) {
+ // A NULL sketch carries no items, so it contributes nothing to the
combined sketch.
+ return buffer
Review Comment:
couldn't reproduce this, I tried separate partials via grouping keys,
shuffle partitions of 1/2/8/64 with AQE disabled, and 200 NULL rows against one
real sketch, all returning identical correct results, which I think is because
`uncheckedItemDataType` comes from `state.dataType` and is one fixed type per
query, so every partial stamps the same type. added your verification cases as
tests in d6d289d.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
