[
https://issues.apache.org/jira/browse/SPARK-60240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60240:
-----------------------------------
Labels: pull-request-available (was: )
> tuple_sketch_agg different results
> ----------------------------------
>
> Key: SPARK-60240
> URL: https://issues.apache.org/jira/browse/SPARK-60240
> Project: Spark
> Issue Type: Bug
> Components: Bug
> Affects Versions: 4.2.0, 4.3.0, 4.4.0
> Reporter: Milos Stojko
> Priority: Minor
> Labels: pull-request-available
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Bug description - all regarding {{{}tuple_sketch_agg{}}}:
> * spark GROUP BY returns a different answer depending on if it ran on one
> task (COMPLETE agg mode) or in multiple tasks. When running on a single task
> it will return an untrimmed anwser (taking 2x more memory) when compared to
> running on multiple tasks.
> * Another side effect of this bug: when running in partial mode, the output
> for the leaf tasks will take 2x memory it would usually take
> * Another side effect of this bug: {{{}tuple_union_agg{}}}, when given the
> same inputs as {{tuple_sketch_agg}} will produce a different estimate
> * Another side effect of this bug: outputs of {{tuple_sketch_agg}} window
> aggregates take 2x more memory (they are usually untrimmed), which is non
> trivial as window aggregates produce approximately as many outputs as there
> are inputs (except for unbounded frames).
> * Another side effect of this bug - when running on one task (complete
> mode), the later entries in the sketch have a disproportionately large effect
> compared to earlier entries (this is not true when you trim), which can be
> bad if for later values are from a different distribution (for example they
> have much larger summaries). This breaks the order insensitivity of tuple agg
>
> Why this bug happens:
> * for the {{tuple_sketch_agg}} we sometimes use union internal class instead
> (for merging results, eg. in final aggregate mode). Union trims and tuple
> sketch build does not.
>
> Solution:
> * we simply make the {{TupleUpdateSketchImpl}} also return the trimmed
> version
>
> Perhaps breaking changes to customers:
> * this would require modifying preexisting spark group by, spark window. The
> outputs would sometimes be 2x smaller, and would produce a different estimate.
>
> Repro:
> Just instead of {{N}} put in either 1 or 4 to get the different versions
> (trimed vs untrimed)
> {{SELECT g, (length(s) - 24) DIV 17 AS entries
> FROM (SELECT id % 2 AS g, tuple_sketch_agg_double(id, 1.0D, 4) AS s
> FROM range(0, 1000, 1, N)
> GROUP BY id % 2);}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
