jiwen624 opened a new pull request, #57327:
URL: https://github.com/apache/spark/pull/57327
### What changes were proposed in this pull request?
`approx_top_k_estimate` picked its sketch deserialization SerDe from the
static item type in the sketch-state struct (field 2). A `UNION ALL` of two
sketch columns with coercible item types (e.g. INT + BIGINT) widens that static
type while the serialized bytes and the per-row DDL (field 3) keep the original
type. Estimation then decoded the bytes with the wrong SerDe.
This PR makes `estimate` deserialize using the true item type recorded
per-row in the state DDL (recovering collation from the static type, exactly as
`combine` already does), then widens each decoded item to the declared output
type via a lossless cast so results still match the statically-declared schema.
### Why are the changes needed?
A query like:
```sql
SELECT approx_top_k_estimate(approx_top_k_combine(sketch, 5))
FROM (SELECT approx_top_k_accumulate(c) AS sketch, 'int' AS tag
FROM VALUES (1), (1), (2) AS t(c)
UNION ALL
SELECT approx_top_k_accumulate(c) AS sketch, 'long' AS tag
FROM VALUES (CAST(9 AS BIGINT)) AS t(c))
WHERE tag = 'int';
```
fails at runtime with SketchesArgumentException: Bounds Violation, e.g.,
```Java
26/07/16 22:29:33 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 3)
org.apache.datasketches.common.SketchesArgumentException: Bounds Violation:
reqOffset: 0, reqLength: 16, (reqOff + reqLen): 16, allocSize: 10
at org.apache.datasketches.common.Util.checkBounds(Util.java:659)
```
Or, depending on the widening it can also throw ClassCastException or, when
byte sizes
happen to match (e.g. BIGINT→DOUBLE), silently return wrong results.
### Does this PR introduce _any_ user-facing change?
Yes. A query that may crash or return wrong results now gets fixed.
### How was this patch tested?
Added UT.
### Was this patch authored or co-authored using generative AI tooling?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
