dtenedor opened a new pull request, #53463:
URL: https://github.com/apache/spark/pull/53463
### What changes were proposed in this pull request?
This PR adds a restriction that the rank/quantile arguments to
`kll_sketch_get_quantile_*` and `kll_sketch_get_rank_*` functions must be
foldable (compile-time constants).
**Changes:**
1. Added `checkInputDataTypes()` validation to `KllSketchGetQuantileBase`
and `KllSketchGetRankBase` that returns a `NON_FOLDABLE_INPUT` error if the
rank/quantile argument is not foldable.
2. Fixed a bug in `nullSafeEval` where `right.eval()` was being called
instead of using the already-evaluated `rightInput` parameter.
3. Added negative test cases in `kllquantiles.sql` to verify the new
foldability restriction for:
- Non-foldable scalar rank argument to `kll_sketch_get_quantile_bigint`
- Non-foldable array rank argument to `kll_sketch_get_quantile_bigint`
- Non-foldable scalar quantile argument to `kll_sketch_get_rank_bigint`
- Non-foldable array quantile argument to `kll_sketch_get_rank_bigint`
### Why are the changes needed?
The foldability restriction is reasonable to have, since this is the
intended usage of the functions, and is consistent with other existing
functions in Spark.
Additionally, fixing the `right.eval()` bug ensures we use the
already-evaluated value passed to `nullSafeEval`, avoiding redundant evaluation.
### Does this PR introduce _any_ user-facing change?
Yes. Users will now receive an error if they pass a non-constant expression
(e.g., a column reference) as the rank argument to `kll_sketch_get_quantile_*`
or the quantile argument to `kll_sketch_get_rank_*`. For example:
-- This will now fail with NON_FOLDABLE_INPUT error
SELECT kll_sketch_get_quantile_bigint(sketch, col / 10.0) FROM table;
-- This is allowed (constant expression)
SELECT kll_sketch_get_quantile_bigint(sketch, 0.5) FROM table;
SELECT kll_sketch_get_quantile_bigint(sketch, array(0.25, 0.5, 0.75)) FROM
table;### How was this patch tested?
Added negative test cases in `kllquantiles.sql` that exercise the new
foldability restriction for both scalar and array inputs. The tests verify that
appropriate error messages are returned for non-foldable arguments.
### Was this patch authored or co-authored using generative AI tooling?
Yes, with assistance from `claude-4.5-opus-high` with manual review and
adjustment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
