[
https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14227842#comment-14227842
]
Patrick Wendell commented on SPARK-4598:
----------------------------------------
Having sorting with pagination seems very difficult to do correctly since we
rely on javascript for sorting in the frontent. It would be helpful to
understand the exact memory requirements of serving hundreds of thousands of
tasks. Where is the memory from? Can we just optimize the use of memory? We
need to store all of those tasks anyways in int he driver.
> Paginate stage page to avoid OOM with > 100,000 tasks
> -----------------------------------------------------
>
> Key: SPARK-4598
> URL: https://issues.apache.org/jira/browse/SPARK-4598
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: meiyoula
> Priority: Critical
>
> In HistoryServer stage page, clicking the task href in Description, it occurs
> the GC error. The detail error message is:
> 2014-11-17 16:36:30,851 | WARN | [qtp1083955615-352] | Error for
> /history/application_1416206401491_0010/stages/stage/ |
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:590)
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2014-11-17 16:36:30,851 | WARN | [qtp1083955615-364] | handle failed |
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:697)
> java.lang.OutOfMemoryError: GC overhead limit exceeded
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
