[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225566#comment-14225566 ]
xukun commented on SPARK-4598: ------------------------------ i think you can sort the task data first and then paginate the page. i think the origin data do not need much memory. the task page need much memory > Paginate stage page to avoid OOM with > 100,000 tasks > ----------------------------------------------------- > > Key: SPARK-4598 > URL: https://issues.apache.org/jira/browse/SPARK-4598 > Project: Spark > Issue Type: Bug > Components: Spark Core > Reporter: meiyoula > Priority: Critical > > In HistoryServer stage page, clicking the task href in Description, it occurs > the GC error. The detail error message is: > 2014-11-17 16:36:30,851 | WARN | [qtp1083955615-352] | Error for > /history/application_1416206401491_0010/stages/stage/ | > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:590) > java.lang.OutOfMemoryError: GC overhead limit exceeded > 2014-11-17 16:36:30,851 | WARN | [qtp1083955615-364] | handle failed | > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:697) > java.lang.OutOfMemoryError: GC overhead limit exceeded -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org