Stefan Savic created SPARK-59975:
------------------------------------
Summary: Handle all-null partial and final tuple/theta
intersection aggregates
Key: SPARK-59975
URL: https://issues.apache.org/jira/browse/SPARK-59975
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.4.0
Reporter: Stefan Savic
{{{}tuple_intersection_agg_double{}}}, {{{}tuple_intersection_agg_integer{}}},
and {{theta_intersection_agg}} skip null sketch inputs. When a group or partial
sees no non-null input, its intersection remains untouched. JVM partial
serialization and final evaluation both call DataSketches {{getResult()}} on
that state, which throws {{SketchesStateException}} ("getResult() with no
intervening intersections is not a legal result"). Thus an all-null partial can
fail before another partition's valid contribution is merged, and a fully
all-null group throws instead of producing a valid empty sketch. A genuinely
empty intersection after non-null input is a different, valid result.
h3. Expected behavior
* Partial/PartialMerge preserves an explicit no-non-null-input state. Merge
treats that state as no contribution on either side; returning a valid empty
sketch from a partial would incorrectly erase another partition's non-empty
contribution.
* Final/Complete returns a valid empty sketch when the entire group had no
non-null sketch inputs, for grouped and ungrouped aggregation, without throwing.
* A valid empty input sketch remains a real contribution, and disjoint
non-null sketches legitimately produce an empty intersection.
h3. Acceptance criteria
* All-null grouped and ungrouped aggregates return a valid empty sketch
(estimate 0), with no exception during Partial, PartialMerge, Final, or
Complete.
* A group split across partitions into an all-null partial and a non-empty
partial produces the non-empty partial's result, in either merge order; other
groups remain unaffected.
* Valid empty input sketches, disjoint non-null intersections, and supported
JVM/Photon mixed-stage execution keep their correct semantics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
