Milos Stojko created SPARK-60240:
------------------------------------
Summary: tuple_sketch_agg different results
Key: SPARK-60240
URL: https://issues.apache.org/jira/browse/SPARK-60240
Project: Spark
Issue Type: Bug
Components: Bug
Affects Versions: 4.2.0, 4.3.0, 4.4.0
Reporter: Milos Stojko
Bug description - all regarding {{{}tuple_sketch_agg{}}}:
* spark and photon GROUP BY returns a different answer depending on if it ran
on one task (COMPLETE agg mode) or in multiple tasks. When running on multiple
tasks it will return an untrimmed anwser (taking 2x more memory) when compared
to running on single task.
* Another side effect of this bug: when running in partial mode, the output
for the leaf tasks will take 2x memory it would usually take
* Another side effect of this bug: {{{}tuple_union_agg{}}}, when given the
same inputs as {{tuple_sketch_agg}} will produce a different estimate
* Another side effect of this bug: outputs of {{tuple_sketch_agg}} window
aggregates take 2x more memory (they are always untrimmed), which is non
trivial as window aggregates produce approximately as many outputs as there are
inputs (except for unbounded frames).
* Another side effect of this bug - when running on one task (complete mode),
the later entries in the sketch have a disproportionately large effect compared
to earlier entries (this is not true when you trim), which can be bad if for
later values are from a different distribution (for example they have much
larger summaries). This breaks the order insensitivity of tuple agg
Why this bug happens:
* for the {{tuple_sketch_agg}} we sometimes use union internal class instead
(for merging results, eg. in final aggregate mode). Union trims and tuple
sketch build does not.
Solution:
* we simply make the {{TupleUpdateSketchImpl}} also return the trimmed version
Perhaps breaking changes to customers:
* this would require modifying preexisting spark group by, spark window, and
photon window. The outputs would sometimes be 2x smaller, and would produce a
different estimate.
Repro:
Repro dbr:
Just instead of {{N}} put in either 1 or 4 to get the different versions
(trimed vs untrimed)
{{SELECT g, (length(s) - 24) DIV 17 AS entries
FROM (SELECT id % 2 AS g, tuple_sketch_agg_double(id, 1.0D, 4) AS s
FROM range(0, 1000, 1, N)
GROUP BY id % 2);}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
