[ https://issues.apache.org/jira/browse/SPARK-34337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279000#comment-17279000 ]
Holden Karau commented on SPARK-34337: -------------------------------------- Initially we should allow the user to configure a maximum amount of shuffle blocks to be stored. In the future we can try and use underlying FS info. > Reject disk blocks when out of disk space > ----------------------------------------- > > Key: SPARK-34337 > URL: https://issues.apache.org/jira/browse/SPARK-34337 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 3.2.0, 3.1.1, 3.1.2 > Reporter: Holden Karau > Priority: Major > > Now that we have the ability to store shuffle blocks on dis-aggregated > storage (when configured) we should add the option to reject storing blocks > locally on an executor at a certain disk pressure threshold. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org