Wei Guo created SPARK-42252: ------------------------------- Summary: Deprecate spark.shuffle.unsafe.file.output.buffer and add a new config Key: SPARK-42252 URL: https://issues.apache.org/jira/browse/SPARK-42252 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.0.0 Reporter: Wei Guo Fix For: 3.4.0
After Jira SPARK-28209 and PR [25007|[https://github.com/apache/spark/pull/25007]], the new shuffle writer api is proposed. All shuffle writers(BypassMergeSortShuffleWriter, SortShuffleWriter, UnsafeShuffleWriter) are based on LocalDiskShuffleMapOutputWriter to write local disk shuffle files. The config spark.shuffle.unsafe.file.output.buffer used in LocalDiskShuffleMapOutputWriter was only used in UnsafeShuffleWriter before. It's better to rename it and make it more suitable. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org