[ https://issues.apache.org/jira/browse/SPARK-28166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen updated SPARK-28166: ------------------------------- Description: The *symmetric difference* (a.k.a. *disjunctive union*) of two sets is their set union minus their set intersection: it returns tuples which are in only one of the sets and omits tuples which are present in both sets (see [https://en.wikipedia.org/wiki/Symmetric_difference]). With the Datasets API, we can express this as either {code:java} a.union(b).except(a.intersect(b)){code} or {code:java} a.except(b).union(b.except(a)){code} Spark currently plan this query with two joins. However, it may be more efficient to represent this as a full outer join followed by a filter and a distinct (and, depending on the number of duplicates, we might want to push additional distinct clauses beneath the join, but I think that's a separate optimization). It would cool if the optimizer could automatically perform this rewrite. This is a very low priority: I'm filing this ticket mostly for tracking / reference purposes (so searches for 'symmetric difference' turn up something useful in Spark's JIRA). was: The *symmetric difference* (a.k.a. *disjunctive union*) of two sets is their set union minus their set intersection: it returns tuples which are in only one of the sets and omits tuples which are present in both sets (see [https://en.wikipedia.org/wiki/Symmetric_difference]). With the Datasets API, we can express this as either {code:java} a.union(b).except(a.intersect(b)){code} or {code:java} a.except(b).union(b.except(a)){code} Spark currently plan this query with two joins. However, it may be more efficient to represent this as a full outer join followed by a filter and a distinct (and, depending on the number of duplicates, we might want to push additional distinct clauses beneath the join, but I think that's a separate optimization). It would cool if the optimizer could automatically perform this rewrite. This is a pretty low priority: I'm filing this ticket mostly for tracking / reference purposes (so searches for 'symmetric difference' turn up something useful in Spark's JIRA). > Query optimization for symmetric difference / disjunctive union of Datasets > --------------------------------------------------------------------------- > > Key: SPARK-28166 > URL: https://issues.apache.org/jira/browse/SPARK-28166 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.0.0 > Reporter: Josh Rosen > Priority: Minor > > The *symmetric difference* (a.k.a. *disjunctive union*) of two sets is their > set union minus their set intersection: it returns tuples which are in only > one of the sets and omits tuples which are present in both sets (see > [https://en.wikipedia.org/wiki/Symmetric_difference]). > With the Datasets API, we can express this as either > {code:java} > a.union(b).except(a.intersect(b)){code} > or > {code:java} > a.except(b).union(b.except(a)){code} > Spark currently plan this query with two joins. However, it may be more > efficient to represent this as a full outer join followed by a filter and a > distinct (and, depending on the number of duplicates, we might want to push > additional distinct clauses beneath the join, but I think that's a separate > optimization). It would cool if the optimizer could automatically perform > this rewrite. > This is a very low priority: I'm filing this ticket mostly for tracking / > reference purposes (so searches for 'symmetric difference' turn up something > useful in Spark's JIRA). -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org