Yan Yan created SPARK-61589:
-------------------------------
Summary: Define NULL semantics for KLL quantile and rank getter
arrays
Key: SPARK-61589
URL: https://issues.apache.org/jira/browse/SPARK-61589
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yan Yan
The KLL quantile and rank getter functions accept a constant scalar or an array
of requests. NULL elements inside a non-NULL request array are currently read
as numeric zero and produce non-NULL results. This behavior is not explained in
the public API contract, so users cannot distinguish it from scalar or
whole-array NULL propagation.
This affects both getter families for bigint, float and double:
* kll_sketch_get_quantile_bigint / float / double
* kll_sketch_get_rank_bigint / float / double
For a sketch containing -10, 0 and 10, the current bigint results are:
{code:sql}
SELECT kll_sketch_get_quantile_bigint(kll_sketch_agg_bigint(value), array(0.5D,
NULL))
FROM VALUES (-10L), (0L), (10L) AS t(value);
-- [0, -10]
SELECT kll_sketch_get_rank_bigint(kll_sketch_agg_bigint(value), array(10L,
NULL))
FROM VALUES (-10L), (0L), (10L) AS t(value);
-- [1.0, 0.6666666666666666]
{code}
The second example is significant: interpreting a NULL element as a zero-valued
quantile returns the rank of zero, which need not be zero. An all-NULL request
array also produces an array of zero-request results rather than a NULL array.
Define and regression-test the following contract consistently across SQL,
Scala and Python:
* A NULL sketch, scalar NULL request, or whole-array NULL request returns NULL.
* A NULL rank element passed to a quantile getter requests rank zero, yielding
the minimum quantile.
* A NULL quantile element passed to a rank getter requests the rank of zero.
* This applies to mixed and all-NULL request arrays for all three numeric
sketch types.
The proposed resolution preserves existing accepted-query behavior. The APIs do
not establish a clear non-NULL-element precondition today; introducing
rejection would change those queries. Documenting the existing semantics and
adding focused regression coverage makes the contract explicit without changing
runtime results.
Implementation and regression coverage:
https://github.com/apache/spark/pull/60358
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]