[ 
https://issues.apache.org/jira/browse/SPARK-61589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61589:
-----------------------------------
    Labels: pull-request-available  (was: )

> Define NULL semantics for KLL quantile and rank getter arrays
> -------------------------------------------------------------
>
>                 Key: SPARK-61589
>                 URL: https://issues.apache.org/jira/browse/SPARK-61589
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Yan Yan
>            Priority: Minor
>              Labels: pull-request-available
>
> The KLL quantile and rank getter functions accept a constant scalar or an 
> array of requests. NULL elements inside a non-NULL request array are 
> currently read as numeric zero and produce non-NULL results. This behavior is 
> not explained in the public API contract, so users cannot distinguish it from 
> scalar or whole-array NULL propagation.
> This affects both getter families for bigint, float and double:
> * kll_sketch_get_quantile_bigint / float / double
> * kll_sketch_get_rank_bigint / float / double
> For a sketch containing -10, 0 and 10, the current bigint results are:
> {code:sql}
> SELECT kll_sketch_get_quantile_bigint(kll_sketch_agg_bigint(value), 
> array(0.5D, NULL))
> FROM VALUES (-10L), (0L), (10L) AS t(value);
> -- [0, -10]
> SELECT kll_sketch_get_rank_bigint(kll_sketch_agg_bigint(value), array(10L, 
> NULL))
> FROM VALUES (-10L), (0L), (10L) AS t(value);
> -- [1.0, 0.6666666666666666]
> {code}
> The second example is significant: interpreting a NULL element as a 
> zero-valued quantile returns the rank of zero, which need not be zero. An 
> all-NULL request array also produces an array of zero-request results rather 
> than a NULL array.
> Define and regression-test the following contract consistently across SQL, 
> Scala and Python:
> * A NULL sketch, scalar NULL request, or whole-array NULL request returns 
> NULL.
> * A NULL rank element passed to a quantile getter requests rank zero, 
> yielding the minimum quantile.
> * A NULL quantile element passed to a rank getter requests the rank of zero.
> * This applies to mixed and all-NULL request arrays for all three numeric 
> sketch types.
> The proposed resolution preserves existing accepted-query behavior. The APIs 
> do not establish a clear non-NULL-element precondition today; introducing 
> rejection would change those queries. Documenting the existing semantics and 
> adding focused regression coverage makes the contract explicit without 
> changing runtime results.
> Implementation and regression coverage: 
> https://github.com/apache/spark/pull/60358



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to