jiwen624 opened a new pull request, #57177:
URL: https://github.com/apache/spark/pull/57177
### What changes were proposed in this pull request?
For a non-`UTF8_BINARY` collated STRING column, `approx_top_k` (and
`approx_top_k_accumulate` / `_combine` / `_estimate`) stored the collation sort
key in the sketch and returned it verbatim as the item. This makes the sketch
store `CollatedString` instead: it dedupes by the collation key (so
collation-equal values still count as one) but retains an actual input value to
return, the way `mode()` does. The serde writes only the original value and
recomputes the key on read, so the `UTF8_BINARY` format is unchanged.
### Why are the changes needed?
The returned items were wrong: raw ICU sort-key bytes (invalid UTF-8) for
ICU collations, and the lower-cased form for `UTF8_LCASE` (a value that may
never have appeared). Counts were correct, but the item labels were garbage.
Silent wrong result, no error.
Before the fix:
```sql
scala> spark.sql("""SELECT approx_top_k(c, 5, 100) FROM (
| SELECT CAST(col AS STRING COLLATE UNICODE_CI) AS c
| FROM VALUES ('HELLO'), ('HELLO'), ('HELLO'), ('hello'), ('world')
AS t(col)
| )""").show(100, false)
+--------------------------------+
|approx_top_k(c, 5, 100) |
+--------------------------------+
|[{93AAG\t, 4}, {WGMA1\t, 1}]|
+--------------------------------+
scala> spark.sql("""SELECT approx_top_k(c, 5, 100) FROM (
| SELECT CAST(col AS STRING COLLATE UNICODE) AS c
| FROM VALUES ('HELLO'), ('HELLO'), ('HELLO'), ('hello'), ('world')
AS t(col)
| )""").show(100, false)
+------------------------------------------------------------+
|approx_top_k(c, 5, 100) |
+------------------------------------------------------------+
|[{93AAG\t�����, 3}, {93AAG\t\t, 1}, {WGMA1\t\t, 1}]|
+------------------------------------------------------------+
scala> spark.sql("""SELECT approx_top_k(c, 5, 100) FROM (
| SELECT CAST(col AS STRING COLLATE UTF8_LCASE) AS c
| FROM VALUES ('HELLO'), ('HELLO'), ('HELLO'), ('hello'), ('world')
AS t(col)
| )""").show(100, false)
+------------------------+
|approx_top_k(c, 5, 100) |
+------------------------+
|[{hello, 4}, {world, 1}]|
+------------------------+
```
### Does this PR introduce _any_ user-facing change?
Yes. approx_top_k over a collated string column now returns an actual input
value instead of the internal collation key. Sketches persisted by
approx_top_k_accumulate over a non-binary collated column before this change
are not readable after it (they contained wrong data); the UTF8_BINARY format
is unchanged.
### How was this patch tested?
Added test cases.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
