david-mollitor-db commented on code in PR #59162:
URL: https://github.com/apache/spark/pull/59162#discussion_r4283492018
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/stat/FrequentItems.scala:
##########
@@ -88,6 +88,21 @@ case class CollectFrequentItems(
override def createAggregationBuffer(): mutable.Map[Any, Long] =
mutable.Map.empty[Any, Long]
+ /**
+ * Returns the map key for `value`. A BinaryType value is wrapped in a
[[BinaryKey]] so that the
Review Comment:
Thanks @zhengruifeng! `collect_set` (and `collect_union`) handle the same
problem in `convertToBufferElement`: a binary value is keyed as
`UnsafeArrayData.fromPrimitiveArray(bytes)`, and `eval` converts it back with
`toByteArray()`. This PR follows the same structure (wrap in
`update`/`deserialize`, unwrap in `eval`/`serialize`) and the same shape as the
`CollationKeyedElement` wrapper `collect_set` uses for collated keys. The
difference is that `BinaryKey` doesn't copy the array: `freqItems` builds a key
for every row, including rows whose value is already counted, and `child.eval`
already returns a fresh array. I'm happy to switch to `UnsafeArrayData` if
you'd prefer the two to match exactly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
