GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/6248
[SPARK-7716] [UI] Stage page hangs with many tasks
When you view the stage page while running the following:
```
sc.parallelize(1 to X, 100000).count()
```
The page never loads, the job is stalled, and you end up running into an
OOM:
```
HTTP ERROR 500
Problem accessing /stages/stage/. Reason:
Server Error
Caused by:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2367)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
at
```
This patch compresses Jetty responses in gzip. After applying this patch,
the job finishes as expected and loading the stage page no longer crashes the
Jetty server. The correct long-term fix is to add pagination.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark gzip-jetty
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6248.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6248
----
commit 304203e6dd0c2450b8181ab6e2ed275349ad6966
Author: Andrew Or <[email protected]>
Date: 2015-05-18T22:56:55Z
Use GZIP to compress Jetty responses
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
