Yan Yan created SPARK-61589:
-------------------------------

             Summary: Define NULL semantics for KLL quantile and rank getter 
arrays
                 Key: SPARK-61589
                 URL: https://issues.apache.org/jira/browse/SPARK-61589
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Yan Yan


The KLL quantile and rank getter functions accept a constant scalar or an array 
of requests. NULL elements inside a non-NULL request array are currently read 
as numeric zero and produce non-NULL results. This behavior is not explained in 
the public API contract, so users cannot distinguish it from scalar or 
whole-array NULL propagation.

This affects both getter families for bigint, float and double:

* kll_sketch_get_quantile_bigint / float / double
* kll_sketch_get_rank_bigint / float / double

For a sketch containing -10, 0 and 10, the current bigint results are:

{code:sql}
SELECT kll_sketch_get_quantile_bigint(kll_sketch_agg_bigint(value), array(0.5D, 
NULL))
FROM VALUES (-10L), (0L), (10L) AS t(value);
-- [0, -10]

SELECT kll_sketch_get_rank_bigint(kll_sketch_agg_bigint(value), array(10L, 
NULL))
FROM VALUES (-10L), (0L), (10L) AS t(value);
-- [1.0, 0.6666666666666666]
{code}

The second example is significant: interpreting a NULL element as a zero-valued 
quantile returns the rank of zero, which need not be zero. An all-NULL request 
array also produces an array of zero-request results rather than a NULL array.

Define and regression-test the following contract consistently across SQL, 
Scala and Python:

* A NULL sketch, scalar NULL request, or whole-array NULL request returns NULL.
* A NULL rank element passed to a quantile getter requests rank zero, yielding 
the minimum quantile.
* A NULL quantile element passed to a rank getter requests the rank of zero.
* This applies to mixed and all-NULL request arrays for all three numeric 
sketch types.

The proposed resolution preserves existing accepted-query behavior. The APIs do 
not establish a clear non-NULL-element precondition today; introducing 
rejection would change those queries. Documenting the existing semantics and 
adding focused regression coverage makes the contract explicit without changing 
runtime results.

Implementation and regression coverage: 
https://github.com/apache/spark/pull/60358




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to