GitHub user tankkyo opened a pull request:
https://github.com/apache/spark/pull/12990
Limit total tasks displayed on Web UI to avoid History Server OOM
## What changes were proposed in this pull request?
History Server will load all task ui data, which would cause OOM if there
were too many tasks.
We should allow user to trim them by spark.ui.retainedTasks(default:1000)
## How was this patch tested?
bin/run-example SparkPi 100000, and open stages page on Web UI
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tankkyo/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12990.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12990
----
commit f131cd871d9b4202cad65d5b6d0e967d51713a01
Author: Zheng Tan <[email protected]>
Date: 2016-05-03T01:10:56Z
Fix SPARK-15059: remove fine-grained lock in ChildFirstURLClassLoader
commit f91966e8998256371666be221fa8f7a6247ff750
Author: Zheng Tan <[email protected]>
Date: 2016-05-08T12:10:12Z
Merge remote-tracking branch 'upstream/master'
commit 593b294ce36c5e99e6a4c719e4e4f4d64d6b976b
Author: Zheng Tan <[email protected]>
Date: 2016-05-08T12:48:57Z
Trim tasks in a stage to avoid history server OOM
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
