[
https://issues.apache.org/jira/browse/SPARK-59975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59975:
-----------------------------------
Labels: pull-request-available (was: )
> Handle all-null partial and final tuple/theta intersection aggregates
> ---------------------------------------------------------------------
>
> Key: SPARK-59975
> URL: https://issues.apache.org/jira/browse/SPARK-59975
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.4.0
> Reporter: Stefan Savic
> Priority: Major
> Labels: pull-request-available
>
> {{{}tuple_intersection_agg_double{}}},
> {{{}tuple_intersection_agg_integer{}}}, and {{theta_intersection_agg}} skip
> null sketch inputs. When a group or partial sees no non-null input, its
> intersection remains untouched. JVM partial serialization and final
> evaluation both call DataSketches {{getResult()}} on that state, which throws
> {{SketchesStateException}} ("getResult() with no intervening intersections is
> not a legal result"). Thus an all-null partial can fail before another
> partition's valid contribution is merged, and a fully all-null group throws
> instead of producing a valid empty sketch. A genuinely empty intersection
> after non-null input is a different, valid result.
> h3. Expected behavior
> * Partial/PartialMerge preserves an explicit no-non-null-input state. Merge
> treats that state as no contribution on either side; returning a valid empty
> sketch from a partial would incorrectly erase another partition's non-empty
> contribution.
> * Final/Complete returns a valid empty sketch when the entire group had no
> non-null sketch inputs, for grouped and ungrouped aggregation, without
> throwing.
> * A valid empty input sketch remains a real contribution, and disjoint
> non-null sketches legitimately produce an empty intersection.
> h3. Acceptance criteria
> * All-null grouped and ungrouped aggregates return a valid empty sketch
> (estimate 0), with no exception during Partial, PartialMerge, Final, or
> Complete.
> * A group split across partitions into an all-null partial and a non-empty
> partial produces the non-empty partial's result, in either merge order; other
> groups remain unaffected.
> * Valid empty input sketches, disjoint non-null intersections, and supported
> JVM/Photon mixed-stage execution keep their correct semantics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
