Milos Stojko created SPARK-61981:
------------------------------------
Summary: kll_sketch_merge_*: add a conf to take the lowest k among
the first non-empty input and a compacting right input
Key: SPARK-61981
URL: https://issues.apache.org/jira/browse/SPARK-61981
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Milos Stojko
Problem - {{kll_sketch_merge_bigint}}, {{kll_sketch_merge_float}} and
{{kll_sketch_merge_double}} return a sketch with the k of {{left}}, whatever
the k of {{right}} is:
* If {{right}} has started compacting at a smaller k, the result keeps the
larger k of {{left}}. The result is larger, but not more accurate: DataSketches
already uses the smaller k for the error bounds.
* An empty {{left}} also decides k. For example, the empty sketch of an
all-NULL group has k = 200. Merged with an exact k = 1000 sketch of 500 values,
the result has k = 200 and compacts, so it loses accuracy that the data had.
* SPARK-61368 adds a rule for k to {{kll_merge_agg_*}}. Without the same rule
here, {{kll_sketch_merge_*(a, b)}} and {{kll_merge_agg_*}} over the rows a, b
can return different k.
Proposal:
* Add an internal SQL conf, off by default:
{{spark.sql.function.kllSketchMerge.lowestCompactingK.enabled}}.
* When it is on, {{kll_sketch_merge_*(left, right)}} returns k = min(k of the
first non-empty input, k of {{right}} if {{right}} has started compacting
(estimation mode)). An empty input counts as absent, and two empty inputs give
the default k 200. A NULL input still gives NULL.
* This is the rule of SPARK-61368 for two inputs, and the rule of BigQuery's
KLL merge.
Repro: with the conf off, this query returns k = 1000. With the conf on, it
returns k = 100.
{code:sql}
SET spark.sql.function.kllSketchMerge.lowestCompactingK.enabled = true;
SELECT regexp_extract(kll_sketch_to_string_bigint(kll_sketch_merge_bigint(l,
r)), 'K +: ([0-9]+)', 1) AS k
FROM (SELECT kll_sketch_agg_bigint(id, 1000) AS l FROM range(10)),
(SELECT kll_sketch_agg_bigint(id, 100) AS r FROM range(100000));
{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]