[jira] [Commented] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407714#comment-17407714 ] Adam Kennedy commented on SPARK-36446: -- [~tgraves] Yes, we are running with recovery enabled (in

[jira] [Comment Edited] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394905#comment-17394905 ] Adam Kennedy edited comment on SPARK-36446 at 8/6/21, 5:56 PM: --- The

[jira] [Comment Edited] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394905#comment-17394905 ] Adam Kennedy edited comment on SPARK-36446 at 8/6/21, 5:06 PM: --- The

[jira] [Comment Edited] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394905#comment-17394905 ] Adam Kennedy edited comment on SPARK-36446 at 8/6/21, 5:06 PM: --- The

[jira] [Commented] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394905#comment-17394905 ] Adam Kennedy commented on SPARK-36446: -- The problem was particularly amplified by the Executor

[jira] [Commented] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394873#comment-17394873 ] Adam Kennedy commented on SPARK-36446: -- Note: While I haven't investigated any other shuffle

[jira] [Updated] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-36446: - Summary: YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated

[jira] [Updated] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-36446: - Summary: YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated

[jira] [Created] (SPARK-36446) After dynamic deallocation YARN shuffle server restart crashes all jobs

Adam Kennedy created SPARK-36446: Summary: After dynamic deallocation YARN shuffle server restart crashes all jobs Key: SPARK-36446 URL: https://issues.apache.org/jira/browse/SPARK-36446 Project:

[jira] [Commented] (SPARK-21097) Dynamic allocation will preserve cached data

[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909342#comment-16909342 ] Adam Kennedy commented on SPARK-21097: -- Another supporting reason for supporting transfer of memory

[jira] [Updated] (SPARK-25889) Dynamic allocation load-aware ramp up

[ https://issues.apache.org/jira/browse/SPARK-25889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-25889: - Description: The time based exponential ramp up behavior for dynamic allocation is naive and

[jira] [Updated] (SPARK-25888) Service requests for persist() blocks via external service after dynamic deallocation

[ https://issues.apache.org/jira/browse/SPARK-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-25888: - Description: Large and highly multi-tenant Spark on YARN clusters with diverse job execution

[jira] [Updated] (SPARK-25888) Service requests for persist() blocks via external service after dynamic deallocation

[ https://issues.apache.org/jira/browse/SPARK-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-25888: - Issue Type: New Feature (was: Improvement) > Service requests for persist() blocks via

[jira] [Updated] (SPARK-25888) Service requests for persist() blocks via external service after dynamic deallocation

[ https://issues.apache.org/jira/browse/SPARK-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-25888: - Shepherd: DB Tsai > Service requests for persist() blocks via external service after dynamic >

[jira] [Created] (SPARK-25889) Dynamic allocation load-aware ramp up

Adam Kennedy created SPARK-25889: Summary: Dynamic allocation load-aware ramp up Key: SPARK-25889 URL: https://issues.apache.org/jira/browse/SPARK-25889 Project: Spark Issue Type: New

[jira] [Updated] (SPARK-25888) Service requests for persist() blocks via external service after dynamic deallocation

[ https://issues.apache.org/jira/browse/SPARK-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-25888: - Description: Large and highly multi-tenant Spark on YARN clusters with diverse job execution

[jira] [Updated] (SPARK-25888) Service requests for persist() blocks via external service after dynamic deallocation

[ https://issues.apache.org/jira/browse/SPARK-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-25888: - Description: Large and highly multi-tenant Spark on YARN clusters with diverse job execution

[jira] [Updated] (SPARK-25888) Service requests for persist() blocks via external service after dynamic deallocation

[ https://issues.apache.org/jira/browse/SPARK-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-25888: - Environment: (was: Large YARN cluster with 1,000 nodes, 50,000 cores and 250 users, with

[jira] [Updated] (SPARK-25888) Service requests for persist() blocks via external service after dynamic deallocation

[ https://issues.apache.org/jira/browse/SPARK-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-25888: - Environment: Large YARN cluster with 1,000 nodes, 50,000 cores and 250 users, with

[jira] [Updated] (SPARK-25888) Service requests for persist() blocks via external service after dynamic deallocation

[ https://issues.apache.org/jira/browse/SPARK-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-25888: - Summary: Service requests for persist() blocks via external service after dynamic deallocation

[jira] [Created] (SPARK-25888) Service requests for persist() block via external service after dynamic deallocation

Adam Kennedy created SPARK-25888: Summary: Service requests for persist() block via external service after dynamic deallocation Key: SPARK-25888 URL: https://issues.apache.org/jira/browse/SPARK-25888

[jira] [Commented] (SPARK-21155) Add (? running tasks) into Spark UI progress

[ https://issues.apache.org/jira/browse/SPARK-21155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222120#comment-16222120 ] Adam Kennedy commented on SPARK-21155: -- It can already get fairly crowded in that run area on large

[jira] [Commented] (SPARK-19112) add codec for ZStandard

[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113653#comment-16113653 ] Adam Kennedy commented on SPARK-19112: -- Will this be impacted by LEGAL-303? zstd-jni embeds zstd