[jira] [Commented] (SPARK-4598) Paginate stage page to avoid OOM with 100,000 tasks

[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626720#comment-14626720 ] Apache Spark commented on SPARK-4598: - User 'zsxwing' has created a pull request for

[jira] [Commented] (SPARK-4598) Paginate stage page to avoid OOM with 100,000 tasks

[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228677#comment-14228677 ] Josh Rosen commented on SPARK-4598: --- I was able to reproduce this issue using the

[jira] [Commented] (SPARK-4598) Paginate stage page to avoid OOM with 100,000 tasks

[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228684#comment-14228684 ] Josh Rosen commented on SPARK-4598: --- Actually, it might be pretty hard to trim down the

[jira] [Commented] (SPARK-4598) Paginate stage page to avoid OOM with 100,000 tasks

[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228438#comment-14228438 ] Masayoshi TSUZUKI commented on SPARK-4598: -- Discussion about this problem seems

[jira] [Commented] (SPARK-4598) Paginate stage page to avoid OOM with 100,000 tasks

[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228453#comment-14228453 ] Masayoshi TSUZUKI commented on SPARK-4598: -- The similar problem was reported on

[jira] [Commented] (SPARK-4598) Paginate stage page to avoid OOM with 100,000 tasks

[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228508#comment-14228508 ] Josh Rosen commented on SPARK-4598: --- [~meiyoula], Do you have a sample job / workload

[jira] [Commented] (SPARK-4598) Paginate stage page to avoid OOM with 100,000 tasks

[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228646#comment-14228646 ] meiyoula commented on SPARK-4598: - [~joshrosen], I use the this two-day's github master

[jira] [Commented] (SPARK-4598) Paginate stage page to avoid OOM with 100,000 tasks

[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227842#comment-14227842 ] Patrick Wendell commented on SPARK-4598: Having sorting with pagination seems very

[jira] [Commented] (SPARK-4598) Paginate stage page to avoid OOM with 100,000 tasks

[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228043#comment-14228043 ] meiyoula commented on SPARK-4598: - Yearh, optimize the use of memory maybe can resolve the

[jira] [Commented] (SPARK-4598) Paginate stage page to avoid OOM with 100,000 tasks

[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14224810#comment-14224810 ] Patrick Wendell commented on SPARK-4598: It is a good idea to paginate this page.

[jira] [Commented] (SPARK-4598) Paginate stage page to avoid OOM with 100,000 tasks

[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225557#comment-14225557 ] WangTaoTheTonic commented on SPARK-4598: I think pagination is a better idea.

[jira] [Commented] (SPARK-4598) Paginate stage page to avoid OOM with 100,000 tasks

[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225566#comment-14225566 ] xukun commented on SPARK-4598: -- i think you can sort the task data first and then paginate