[
https://issues.apache.org/jira/browse/SPARK-61368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Milos Stojko updated SPARK-61368:
---------------------------------
Description:
PR that adds this - [https://github.com/apache/spark/pull/60301]
Problem - all regarding {{{}kll_merge_agg_bigint{}}}, {{kll_merge_agg_float}}
and {{kll_merge_agg_double}} without k:
* The result takes the k of the first input sketch that the aggregate reads.
* This can cause high instability in terms of memory, performance, and output
accuracy: if the first sketch chosen is k=8, it will have high error bounds and
use very little memory, but it's also possible that the very same query uses
k=65000. Which k is chosen is non-deterministic (depends on first sketch
encountered, which is undefined)
Proposal:
* Add a SQL conf, off by default:
{{{}spark.sql.function.kllMergeAgg.lowestCompactingK.enabled{}}}.
* When it is on, a merge without k returns the lowest k among the first
non-empty input sketch and the input sketches that have started compacting
(estimation mode). An explicit k does not change. This is the rule of
BigQuery's KLL merge.
{code:sql}
{code}
was:
PR that adds this - [https://github.com/apache/spark/pull/60301]
Problem - all regarding {{{}kll_merge_agg_bigint{}}}, {{kll_merge_agg_float}}
and {{kll_merge_agg_double}} without k:
* The result takes the k of the first input sketch that the aggregate reads.
* This can cause high instability in terms of memory, performance, and output
accuracy: if the first sketch chosen is k=8, it will have high error bounds and
use very little memory, but it's also possible that the very same query uses
k=65000. Which k is chosen is non-deterministic (depends on first sketch
encountered, which is undefined)
Proposal:
* Add a SQL conf, off by default:
{{{}spark.sql.function.kllMergeAgg.lowestCompactingK.enabled{}}}.
* When it is on, a merge without k returns the lowest k among the input
sketches that have started compacting (estimation mode). If no input has
started compacting, it takes the k of the first non-empty input. An explicit k
does not change.
{code:sql}
{code}
> kll_merge_agg without k: add a conf to take the lowest k among inputs that
> have started compacting
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-61368
> URL: https://issues.apache.org/jira/browse/SPARK-61368
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Milos Stojko
> Priority: Major
> Labels: pull-request-available
>
> PR that adds this - [https://github.com/apache/spark/pull/60301]
> Problem - all regarding {{{}kll_merge_agg_bigint{}}}, {{kll_merge_agg_float}}
> and {{kll_merge_agg_double}} without k:
> * The result takes the k of the first input sketch that the aggregate reads.
> * This can cause high instability in terms of memory, performance, and
> output accuracy: if the first sketch chosen is k=8, it will have high error
> bounds and use very little memory, but it's also possible that the very same
> query uses k=65000. Which k is chosen is non-deterministic (depends on first
> sketch encountered, which is undefined)
> Proposal:
> * Add a SQL conf, off by default:
> {{{}spark.sql.function.kllMergeAgg.lowestCompactingK.enabled{}}}.
> * When it is on, a merge without k returns the lowest k among the first
> non-empty input sketch and the input sketches that have started compacting
> (estimation mode). An explicit k does not change. This is the rule of
> BigQuery's KLL merge.
> {code:sql}
> {code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]