[
https://issues.apache.org/jira/browse/SPARK-61981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Milos Stojko updated SPARK-61981:
---------------------------------
Description:
PR that adds this - [https://github.com/apache/spark/pull/60663]
Problem - {{kll_sketch_merge_bigint}}, {{kll_sketch_merge_float}} and
{{kll_sketch_merge_double}} return a sketch with the k of {{left}}, whatever
the k of {{right}} is:
* If {{right}} has started compacting at a smaller k, the result keeps the
larger k of {{left}}. The result is larger, but not more accurate: DataSketches
already uses the smaller k for the error bounds.
* An empty {{left}} also decides k. For example, the empty sketch of an
all-NULL group has k = 200. Merged with an exact k = 1000 sketch of 500 values,
the result has k = 200 and compacts, so it loses accuracy that the data had.
* SPARK-61368 adds a rule for k to {{kll_merge_agg_*}}. Without the same rule
here, {{kll_sketch_merge_*(a, b)}} and {{kll_merge_agg_*}} over the rows a, b
can return different k.
Proposal:
* Add an internal SQL conf, off by default:
{{spark.sql.function.kllSketchMerge.lowestCompactingK.enabled}}.
* When it is on, {{kll_sketch_merge_*(left, right)}} returns k = min(k of the
first non-empty input, k of {{right}} if {{right}} has started compacting
(estimation mode)). An empty input counts as absent, and two empty inputs give
the default k 200. A NULL input still gives NULL.
* This is the rule of SPARK-61368 for two inputs, and the rule of BigQuery's
KLL merge.
Repro: with the conf off, this query returns k = 1000. With the conf on, it
returns k = 100.
{code:sql}
SET spark.sql.function.kllSketchMerge.lowestCompactingK.enabled = true;
SELECT regexp_extract(kll_sketch_to_string_bigint(kll_sketch_merge_bigint(l,
r)), 'K +: ([0-9]+)', 1) AS k
FROM (SELECT kll_sketch_agg_bigint(id, 1000) AS l FROM range(10)),
(SELECT kll_sketch_agg_bigint(id, 100) AS r FROM range(100000));
{code}
was:
Problem - {{kll_sketch_merge_bigint}}, {{kll_sketch_merge_float}} and
{{kll_sketch_merge_double}} return a sketch with the k of {{left}}, whatever
the k of {{right}} is:
* If {{right}} has started compacting at a smaller k, the result keeps the
larger k of {{left}}. The result is larger, but not more accurate: DataSketches
already uses the smaller k for the error bounds.
* An empty {{left}} also decides k. For example, the empty sketch of an
all-NULL group has k = 200. Merged with an exact k = 1000 sketch of 500 values,
the result has k = 200 and compacts, so it loses accuracy that the data had.
* SPARK-61368 adds a rule for k to {{kll_merge_agg_*}}. Without the same rule
here, {{kll_sketch_merge_*(a, b)}} and {{kll_merge_agg_*}} over the rows a, b
can return different k.
Proposal:
* Add an internal SQL conf, off by default:
{{spark.sql.function.kllSketchMerge.lowestCompactingK.enabled}}.
* When it is on, {{kll_sketch_merge_*(left, right)}} returns k = min(k of the
first non-empty input, k of {{right}} if {{right}} has started compacting
(estimation mode)). An empty input counts as absent, and two empty inputs give
the default k 200. A NULL input still gives NULL.
* This is the rule of SPARK-61368 for two inputs, and the rule of BigQuery's
KLL merge.
Repro: with the conf off, this query returns k = 1000. With the conf on, it
returns k = 100.
{code:sql}
SET spark.sql.function.kllSketchMerge.lowestCompactingK.enabled = true;
SELECT regexp_extract(kll_sketch_to_string_bigint(kll_sketch_merge_bigint(l,
r)), 'K +: ([0-9]+)', 1) AS k
FROM (SELECT kll_sketch_agg_bigint(id, 1000) AS l FROM range(10)),
(SELECT kll_sketch_agg_bigint(id, 100) AS r FROM range(100000));
{code}
> kll_sketch_merge_*: add a conf to take the lowest k among the first non-empty
> input and a compacting right input
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61981
> URL: https://issues.apache.org/jira/browse/SPARK-61981
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Milos Stojko
> Priority: Major
> Labels: pull-request-available
>
> PR that adds this - [https://github.com/apache/spark/pull/60663]
> Problem - {{kll_sketch_merge_bigint}}, {{kll_sketch_merge_float}} and
> {{kll_sketch_merge_double}} return a sketch with the k of {{left}}, whatever
> the k of {{right}} is:
> * If {{right}} has started compacting at a smaller k, the result keeps the
> larger k of {{left}}. The result is larger, but not more accurate:
> DataSketches already uses the smaller k for the error bounds.
> * An empty {{left}} also decides k. For example, the empty sketch of an
> all-NULL group has k = 200. Merged with an exact k = 1000 sketch of 500
> values, the result has k = 200 and compacts, so it loses accuracy that the
> data had.
> * SPARK-61368 adds a rule for k to {{kll_merge_agg_*}}. Without the same
> rule here, {{kll_sketch_merge_*(a, b)}} and {{kll_merge_agg_*}} over the rows
> a, b can return different k.
> Proposal:
> * Add an internal SQL conf, off by default:
> {{spark.sql.function.kllSketchMerge.lowestCompactingK.enabled}}.
> * When it is on, {{kll_sketch_merge_*(left, right)}} returns k = min(k of
> the first non-empty input, k of {{right}} if {{right}} has started compacting
> (estimation mode)). An empty input counts as absent, and two empty inputs
> give the default k 200. A NULL input still gives NULL.
> * This is the rule of SPARK-61368 for two inputs, and the rule of BigQuery's
> KLL merge.
> Repro: with the conf off, this query returns k = 1000. With the conf on, it
> returns k = 100.
> {code:sql}
> SET spark.sql.function.kllSketchMerge.lowestCompactingK.enabled = true;
> SELECT regexp_extract(kll_sketch_to_string_bigint(kll_sketch_merge_bigint(l,
> r)), 'K +: ([0-9]+)', 1) AS k
> FROM (SELECT kll_sketch_agg_bigint(id, 1000) AS l FROM range(10)),
> (SELECT kll_sketch_agg_bigint(id, 100) AS r FROM range(100000));
> {code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]