GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/21240
[SPARK-21274][SQL] Add a new generator function replicate_rows to support
EXCEPT ALL and INTERSECT ALL
## What changes were proposed in this pull request?
Add a new UDTF replicate_rows. This function replicates the values based on
the first argument to the function. This will be used in EXCEPT ALL AND
INTERSECT ALL transformation (future PR) mainly
to preserve "retain duplicates" semantics. Please refer to
[Link](https://drive.google.com/open?id=1nyW0T0b_ajUduQoPgZLAsyHK8s3_dko3ulQuxaLpUXE)
for design. The transformation code changes are in
[Code](https://github.com/dilipbiswal/spark/tree/dkb_except_all_copy)
Example
```
SELECT replicate_rows(3, 1, 2)
```
Result
```
spark-sql> SELECT replicate_rows(3, 1, 2);
3 1 2
3 1 2
3 1 2
Time taken: 0.045 seconds, Fetched 3 row(s)
```
Returns 3 rows based on the first parameter value.
(Please fill in changes proposed in this fix)
## How was this patch tested?
Added tests in GeneratorFunctionSuite, TypeCoercionSuite, SQLQueryTestSuite
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dilipbiswal/spark dkb_setop_replicaterows
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21240.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21240
----
commit 90efeffb039a4c3458add840f98ea91d01cdc4a8
Author: Dilip Biswal <dbiswal@...>
Date: 2018-05-04T22:03:09Z
[SPARK-21274] Add a new generator function replicate_rows to support EXCEPT
ALL and INTERSECT ALL
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
