Wei Guo created SPARK-42252:
-------------------------------
Summary: Deprecate spark.shuffle.unsafe.file.output.buffer and add
a new config
Key: SPARK-42252
URL: https://issues.apache.org/jira/browse/SPARK-42252
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.0.0
Reporter: Wei Guo
Fix For: 3.4.0
After Jira SPARK-28209 and PR
[25007|[https://github.com/apache/spark/pull/25007]], the new shuffle writer
api is proposed. All shuffle writers(BypassMergeSortShuffleWriter,
SortShuffleWriter, UnsafeShuffleWriter) are based on
LocalDiskShuffleMapOutputWriter to write local disk shuffle files. The config
spark.shuffle.unsafe.file.output.buffer used in LocalDiskShuffleMapOutputWriter
was only used in UnsafeShuffleWriter before.
It's better to rename it and make it more suitable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
