GitHub user chenghao-intel opened a pull request:
https://github.com/apache/spark/pull/1765
[SQL] [SPARK-2826] Reduce the memory copy while building the hashmap for
HashOuterJoin
This is a follow up for #1147 , this PR will improve the performance about
10% - 15% in my local tests.
```
Before:
LeftOuterJoin: took 16750 ms ([3000000] records)
LeftOuterJoin: took 15179 ms ([3000000] records)
RightOuterJoin: took 15515 ms ([3000000] records)
RightOuterJoin: took 15276 ms ([3000000] records)
FullOuterJoin: took 19150 ms ([6000000] records)
FullOuterJoin: took 18935 ms ([6000000] records)
After:
LeftOuterJoin: took 15218 ms ([3000000] records)
LeftOuterJoin: took 13503 ms ([3000000] records)
RightOuterJoin: took 13663 ms ([3000000] records)
RightOuterJoin: took 14025 ms ([3000000] records)
FullOuterJoin: took 16624 ms ([6000000] records)
FullOuterJoin: took 16578 ms ([6000000] records)
```
Besides the performance improvement, I also do some clean up as suggested
in #1147
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chenghao-intel/spark hash_outer_join_fixing
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1765.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1765
----
commit ab1f9e0cce75b875f2bacefe3972c7fbe6fc898c
Author: Cheng Hao <[email protected]>
Date: 2014-08-04T07:51:04Z
Reduce the memory copy while building the hashmap
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
