GitHub user kozanitis opened a pull request:
https://github.com/apache/spark/pull/2939
Add range join support to spark-sql
This PR enables spark-sql to join two tables based on the interval
overlapping between their columns. This is a very useful feature to add as you
can see here: https://issues.apache.org/jira/browse/HIVE-556
Syntax:
select * from table1 RANGEJOIN table2 on OVERLAPS( (t1Attribute1,
t1Attribute2), (t2Attribute1, t2Attribute2))
Example:
Table1 (id : String, start1: Int, end1: Int) = ( ("i1", 10,20), ("i2", 50,
60) )
Table2(start2: Int, end2: Int) = ( (15,70), (80, 90) )
select * from Table1 RANGEJOIN Table2 on OVERLAPS( (start1, end1), (start2,
end2) ) produces the following entries:
("i1", 10, 20, 15, 70), ("i2", 50, 60, 15, 70)
Method:
Assuming rdd1 corresponds to the smaller table and rdd2 to the bigger one,
I create an interval tree from all regions of rdd1 which I broadcast to all
nodes and I query the tree with all entries of rdd2.
Note that the interval tree uses metadata only, not all contents of rdd1. I
keep track of the correspondence between the contents of interval tree and rdd1
through a collection of keys that I zipped with rdd1.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kozanitis/spark rangeJoins-1.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2939.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2939
----
commit 092c7c805353651f0da47378d92415dba7c3b591
Author: Kozanitis <[email protected]>
Date: 2014-10-02T01:57:17Z
Add range join support to spark-sql
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
