GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/19366
[Backport][SPARK-22141][SQL] Propagate empty relation before checking
Cartesian products
Back port https://github.com/apache/spark/pull/19362 to branch-2.2
## What changes were proposed in this pull request?
When inferring constraints from children, Join's condition can be
simplified as None.
For example,
```
val testRelation = LocalRelation('a.int)
val x = testRelation.as("x")
val y = testRelation.where($"a" === 2 && !($"a" === 2)).as("y")
x.join.where($"x.a" === $"y.a")
```
The plan will become
```
Join Inner
:- LocalRelation <empty>, [a#23]
+- LocalRelation <empty>, [a#224]
```
And the Cartesian products check will throw exception for above plan.
Propagate empty relation before checking Cartesian products, and the issue
is resolved.
## How was this patch tested?
Unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gengliangwang/spark branch-2.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19366.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19366
----
commit e30ba2be3eb0a6be0a6abd9287492a1ac0ab6063
Author: Wang Gengliang <[email protected]>
Date: 2017-09-27T10:44:10Z
[SPARK-22141][SQL] Propagate empty relation before checking Cartesian
products
When inferring constraints from children, Join's condition can be
simplified as None.
For example,
```
val testRelation = LocalRelation('a.int)
val x = testRelation.as("x")
val y = testRelation.where($"a" === 2 && !($"a" === 2)).as("y")
x.join.where($"x.a" === $"y.a")
```
The plan will become
```
Join Inner
:- LocalRelation <empty>, [a#23]
+- LocalRelation <empty>, [a#224]
```
And the Cartesian products check will throw exception for above plan.
Propagate empty relation before checking Cartesian products, and the issue
is resolved.
Unit test
Author: Wang Gengliang <[email protected]>
Closes #19362 from gengliangwang/MoveCheckCartesianProducts.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
