[
https://issues.apache.org/jira/browse/SPARK-60240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Milos Stojko updated SPARK-60240:
---------------------------------
Description:
PR that fixes this - [https://github.com/apache/spark/pull/59507]
Bug description - all regarding {{{}tuple_sketch_agg{}}}:
* spark GROUP BY returns a different answer depending on if it ran on one task
(COMPLETE agg mode) or in multiple tasks. When running on a single task it will
return an untrimmed anwser (taking 2x more memory) when compared to running on
multiple tasks.
* Another side effect of this bug: when running in partial mode, the output
for the leaf tasks will take 2x memory it would usually take
* Another side effect of this bug: {{{}tuple_union_agg{}}}, when given the
same inputs as {{tuple_sketch_agg}} will produce a different estimate
* Another side effect of this bug: outputs of {{tuple_sketch_agg}} window
aggregates take 2x more memory (they are usually untrimmed), which is non
trivial as window aggregates produce approximately as many outputs as there are
inputs (except for unbounded frames).
* Another side effect of this bug - when running on one task (complete mode),
the later entries in the sketch have a disproportionately large effect compared
to earlier entries (this is not true when you trim), which can be bad if for
later values are from a different distribution (for example they have much
larger summaries). This breaks the order insensitivity of tuple agg
Why this bug happens:
* for the {{tuple_sketch_agg}} we sometimes use union internal class instead
(for merging results, eg. in final aggregate mode). Union trims and tuple
sketch build does not.
Solution:
* we simply make the {{TupleUpdateSketchImpl}} also return the trimmed version
Perhaps breaking changes to customers:
* this would require modifying preexisting spark group by, spark window. The
outputs would sometimes be 2x smaller, and would produce a different estimate.
Repro:
Just instead of {{N}} put in either 1 or 4 to get the different versions
(trimed vs untrimed)
{{SELECT g, (length(s) - 24) DIV 17 AS entries
FROM (SELECT id % 2 AS g, tuple_sketch_agg_double(id, 1.0D, 4) AS s
FROM range(0, 1000, 1, N)
GROUP BY id % 2);}}
was:
Bug description - all regarding {{{}tuple_sketch_agg{}}}:
* spark GROUP BY returns a different answer depending on if it ran on one task
(COMPLETE agg mode) or in multiple tasks. When running on a single task it will
return an untrimmed anwser (taking 2x more memory) when compared to running on
multiple tasks.
* Another side effect of this bug: when running in partial mode, the output
for the leaf tasks will take 2x memory it would usually take
* Another side effect of this bug: {{{}tuple_union_agg{}}}, when given the
same inputs as {{tuple_sketch_agg}} will produce a different estimate
* Another side effect of this bug: outputs of {{tuple_sketch_agg}} window
aggregates take 2x more memory (they are usually untrimmed), which is non
trivial as window aggregates produce approximately as many outputs as there are
inputs (except for unbounded frames).
* Another side effect of this bug - when running on one task (complete mode),
the later entries in the sketch have a disproportionately large effect compared
to earlier entries (this is not true when you trim), which can be bad if for
later values are from a different distribution (for example they have much
larger summaries). This breaks the order insensitivity of tuple agg
Why this bug happens:
* for the {{tuple_sketch_agg}} we sometimes use union internal class instead
(for merging results, eg. in final aggregate mode). Union trims and tuple
sketch build does not.
Solution:
* we simply make the {{TupleUpdateSketchImpl}} also return the trimmed version
Perhaps breaking changes to customers:
* this would require modifying preexisting spark group by, spark window. The
outputs would sometimes be 2x smaller, and would produce a different estimate.
Repro:
Just instead of {{N}} put in either 1 or 4 to get the different versions
(trimed vs untrimed)
{{SELECT g, (length(s) - 24) DIV 17 AS entries
FROM (SELECT id % 2 AS g, tuple_sketch_agg_double(id, 1.0D, 4) AS s
FROM range(0, 1000, 1, N)
GROUP BY id % 2);}}
> tuple_sketch_agg different results
> ----------------------------------
>
> Key: SPARK-60240
> URL: https://issues.apache.org/jira/browse/SPARK-60240
> Project: Spark
> Issue Type: Bug
> Components: Bug
> Affects Versions: 4.2.0, 4.3.0, 4.4.0
> Reporter: Milos Stojko
> Priority: Minor
> Labels: pull-request-available
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> PR that fixes this - [https://github.com/apache/spark/pull/59507]
>
> Bug description - all regarding {{{}tuple_sketch_agg{}}}:
> * spark GROUP BY returns a different answer depending on if it ran on one
> task (COMPLETE agg mode) or in multiple tasks. When running on a single task
> it will return an untrimmed anwser (taking 2x more memory) when compared to
> running on multiple tasks.
> * Another side effect of this bug: when running in partial mode, the output
> for the leaf tasks will take 2x memory it would usually take
> * Another side effect of this bug: {{{}tuple_union_agg{}}}, when given the
> same inputs as {{tuple_sketch_agg}} will produce a different estimate
> * Another side effect of this bug: outputs of {{tuple_sketch_agg}} window
> aggregates take 2x more memory (they are usually untrimmed), which is non
> trivial as window aggregates produce approximately as many outputs as there
> are inputs (except for unbounded frames).
> * Another side effect of this bug - when running on one task (complete
> mode), the later entries in the sketch have a disproportionately large effect
> compared to earlier entries (this is not true when you trim), which can be
> bad if for later values are from a different distribution (for example they
> have much larger summaries). This breaks the order insensitivity of tuple agg
>
> Why this bug happens:
> * for the {{tuple_sketch_agg}} we sometimes use union internal class instead
> (for merging results, eg. in final aggregate mode). Union trims and tuple
> sketch build does not.
>
> Solution:
> * we simply make the {{TupleUpdateSketchImpl}} also return the trimmed
> version
>
> Perhaps breaking changes to customers:
> * this would require modifying preexisting spark group by, spark window. The
> outputs would sometimes be 2x smaller, and would produce a different estimate.
>
> Repro:
> Just instead of {{N}} put in either 1 or 4 to get the different versions
> (trimed vs untrimed)
> {{SELECT g, (length(s) - 24) DIV 17 AS entries
> FROM (SELECT id % 2 AS g, tuple_sketch_agg_double(id, 1.0D, 4) AS s
> FROM range(0, 1000, 1, N)
> GROUP BY id % 2);}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
