[ 
https://issues.apache.org/jira/browse/SPARK-61981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61981:
-----------------------------------
    Labels: pull-request-available  (was: )

> kll_sketch_merge_*: add a conf to take the lowest k among the first non-empty 
> input and a compacting right input
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61981
>                 URL: https://issues.apache.org/jira/browse/SPARK-61981
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Milos Stojko
>            Priority: Major
>              Labels: pull-request-available
>
> Problem - {{kll_sketch_merge_bigint}}, {{kll_sketch_merge_float}} and 
> {{kll_sketch_merge_double}} return a sketch with the k of {{left}}, whatever 
> the k of {{right}} is:
>  * If {{right}} has started compacting at a smaller k, the result keeps the 
> larger k of {{left}}. The result is larger, but not more accurate: 
> DataSketches already uses the smaller k for the error bounds.
>  * An empty {{left}} also decides k. For example, the empty sketch of an 
> all-NULL group has k = 200. Merged with an exact k = 1000 sketch of 500 
> values, the result has k = 200 and compacts, so it loses accuracy that the 
> data had.
>  * SPARK-61368 adds a rule for k to {{kll_merge_agg_*}}. Without the same 
> rule here, {{kll_sketch_merge_*(a, b)}} and {{kll_merge_agg_*}} over the rows 
> a, b can return different k.
> Proposal:
>  * Add an internal SQL conf, off by default: 
> {{spark.sql.function.kllSketchMerge.lowestCompactingK.enabled}}.
>  * When it is on, {{kll_sketch_merge_*(left, right)}} returns k = min(k of 
> the first non-empty input, k of {{right}} if {{right}} has started compacting 
> (estimation mode)). An empty input counts as absent, and two empty inputs 
> give the default k 200. A NULL input still gives NULL.
>  * This is the rule of SPARK-61368 for two inputs, and the rule of BigQuery's 
> KLL merge.
> Repro: with the conf off, this query returns k = 1000. With the conf on, it 
> returns k = 100.
> {code:sql}
> SET spark.sql.function.kllSketchMerge.lowestCompactingK.enabled = true;
> SELECT regexp_extract(kll_sketch_to_string_bigint(kll_sketch_merge_bigint(l, 
> r)), 'K +: ([0-9]+)', 1) AS k
> FROM (SELECT kll_sketch_agg_bigint(id, 1000) AS l FROM range(10)),
>      (SELECT kll_sketch_agg_bigint(id, 100) AS r FROM range(100000));
> {code}



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to