Milos Stojko created SPARK-61981:
------------------------------------

             Summary: kll_sketch_merge_*: add a conf to take the lowest k among 
the first non-empty input and a compacting right input
                 Key: SPARK-61981
                 URL: https://issues.apache.org/jira/browse/SPARK-61981
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Milos Stojko


Problem - {{kll_sketch_merge_bigint}}, {{kll_sketch_merge_float}} and 
{{kll_sketch_merge_double}} return a sketch with the k of {{left}}, whatever 
the k of {{right}} is:
 * If {{right}} has started compacting at a smaller k, the result keeps the 
larger k of {{left}}. The result is larger, but not more accurate: DataSketches 
already uses the smaller k for the error bounds.
 * An empty {{left}} also decides k. For example, the empty sketch of an 
all-NULL group has k = 200. Merged with an exact k = 1000 sketch of 500 values, 
the result has k = 200 and compacts, so it loses accuracy that the data had.
 * SPARK-61368 adds a rule for k to {{kll_merge_agg_*}}. Without the same rule 
here, {{kll_sketch_merge_*(a, b)}} and {{kll_merge_agg_*}} over the rows a, b 
can return different k.

Proposal:
 * Add an internal SQL conf, off by default: 
{{spark.sql.function.kllSketchMerge.lowestCompactingK.enabled}}.
 * When it is on, {{kll_sketch_merge_*(left, right)}} returns k = min(k of the 
first non-empty input, k of {{right}} if {{right}} has started compacting 
(estimation mode)). An empty input counts as absent, and two empty inputs give 
the default k 200. A NULL input still gives NULL.
 * This is the rule of SPARK-61368 for two inputs, and the rule of BigQuery's 
KLL merge.

Repro: with the conf off, this query returns k = 1000. With the conf on, it 
returns k = 100.
{code:sql}
SET spark.sql.function.kllSketchMerge.lowestCompactingK.enabled = true;
SELECT regexp_extract(kll_sketch_to_string_bigint(kll_sketch_merge_bigint(l, 
r)), 'K +: ([0-9]+)', 1) AS k
FROM (SELECT kll_sketch_agg_bigint(id, 1000) AS l FROM range(10)),
     (SELECT kll_sketch_agg_bigint(id, 100) AS r FROM range(100000));
{code}




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to