GitHub user scwf opened a pull request:
https://github.com/apache/spark/pull/4354
[SPARK-5583][SQL][WIP] Support unique join in hive context
Support unique join in hive context, the basic idea is transform unique
join into outer join + filter in spark sql:
FROM UNIQUEJOIN [PRESERVE] T1 a (a.key), [PRESERVE] T2 b (b.key),
[PRESERVE] T3 c (c.key) ...
If all the tables have PRESERVE keyword ==> T1 full out join T2 full out
join T3 ...
else If all the tables do not have PRESERVE keyword => T1 inner join T2
inner join T3 ...
else ==>
T = (T1 full out join T2 full out join T3 ...)
Filter on T, filter condition = keep the rows with any preserve field
is not null.
for examples:
1 T1 a (a.key), PRESERVE T2 b (b.key), PRESERVE T3 c (c.key) ==> if b.key
is not null or c.key is not null, we'll keep the row
2 T1 a (a.key), T2 b (b.key), PRESERVE T3 c (c.key) ==> if c.key is not
null we'll keep the row
Correct me if i am wrong.
todos: add tests for this
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/scwf/spark unique-join
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4354.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4354
----
commit b7e89a94cbeddcb53aac779d4b9d7de2d94e0325
Author: wangfei <[email protected]>
Date: 2015-02-03T05:29:09Z
support unique join in hive context
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
