GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/21857
[SPARK-21274] Implement EXCEPT ALL clause.
## What changes were proposed in this pull request?
Implements EXCEPT ALL clause through query rewrites using existing
operators in Spark. In this PR, an internal UDTF (replicate_rows) is added to
aid in preserving duplicate rows. Please refer to
[Link](https://drive.google.com/open?id=1nyW0T0b_ajUduQoPgZLAsyHK8s3_dko3ulQuxaLpUXE)
for the design.
**Note** This proposed UDTF is kept as a internal function that is purely
used to aid with this particular rewrite to give us flexibility to change to a
more generalized UDTF in future.
Input Query
``` SQL
SELECT c1 FROM ut1 EXCEPT ALL SELECT c1 FROM ut2
```
Rewritten Query
```SQL
SELECT c1
FROM (
SELECT replicate_rows(sum_val, c1) AS (sum_val, c1)
FROM (
SELECT c1, cnt, sum_val
FROM (
SELECT c1, sum(vcol) AS sum_val
FROM (
SELECT 1L as vcol, c1 FROM ut1
UNION ALL
SELECT -1L as vcol, c1 FROM ut2
) AS union_all
GROUP BY union_all.c1
)
WHERE sum_val > 0
)
)
```
## How was this patch tested?
Added test cases in SQLQueryTestSuite and DataFrameSuite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dilipbiswal/spark dkb_except_all_final
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21857.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21857
----
commit 5cf8c4caa8bce874c5336498a6dc805f0bec1681
Author: Dilip Biswal <dbiswal@...>
Date: 2018-05-07T08:18:17Z
[SPARK-21274] Implement EXCEPT ALL clause.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
