GitHub user zhonghaihua opened a pull request:
https://github.com/apache/spark/pull/10128
pull out nondeterministic expressions from Join
Currently,`nondeterministic expressions` are only allowed in `Project` or
`Filter`,And only when we use nondeterministic expressions in `UnaryNode` can
be pulled out.
But,Sometime in many case,we will use nondeterministic expressions to
process `join keys` avoiding data skew.for example:
```
select *
from tableA a
join
(select * from tableB) b
on upper((case when (a.brand_code is null or a.brand_code = '' ) then cast(
(-rand() * 10000000 ) as string ) else a.brand_code end )) = b.brand_code
```
This PR introduce a mechanism to pull out nondeterministic expressions from
`Join`,so we can use nondeterministic expression in `Join` appropriately.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhonghaihua/spark pulloutJoinNondeterministic
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10128.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10128
----
commit 6e166578a5c1a1faf260389509663ac8c71ec015
Author: zhonghaihua <[email protected]>
Date: 2015-11-30T07:44:49Z
pull out nondeterministic expressions from Join
pull out nondeterministic expressions from Join
pull out nondeterministic expressions from Join
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
