GitHub user jiexiong opened a pull request:
https://github.com/apache/spark/pull/15722
[SPARK-18208] [Shuffle] Executor OOM due to a memory leak in BytesToBytesMap
## What changes were proposed in this pull request?
Fixed the OOM problem in BytesToBytesMap.
## How was this patch tested?
build the package and submit a query to spark-shell with the new package.
./bin/fb-spark-shell.sh -q ad_metrics -t -- --conf
spark.dynamicAllocation.maxEx
ecutors=500 --conf spark.sql.shuffle.partitions=4000
History server UI location is https://fburl.com/484871973
Scuba dataset link for task metrics is https://fburl.com/484871984
Scuba dataset link for resource metrics is https://fburl.com/484871989
Scuba data link for profiling is https://fburl.com/484871993
Resource Manager Session link is
https://our.intern.facebook.com/intern/bumblebee_proxy/?proxy_url=http://hadoop2016.atn1.facebook.com:8810/resourceManager/session.html?sessionId=126507000000179
Spark context Web UI available at
http://[2401:db00:11:d0af:face:0:19:0]:8087
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jiexiong/spark jie_oom_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15722.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15722
----
commit 8fa8fa4bca355ac8f803132156fd3926d68190f8
Author: Jie Xiong <[email protected]>
Date: 2016-11-01T20:43:19Z
Fix the OOM failure from this operator.
commit 80fd5b394b0c9b7673c8a8a4eed65bc54afb7205
Author: Jie Xiong <[email protected]>
Date: 2016-11-01T22:48:59Z
Merge remote-tracking branch 'upstream/master' into jie_oom_fix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
