This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ecc33d26e8fa [SPARK-50482][CORE] Deprecated no-op
`spark.shuffle.spill` config
ecc33d26e8fa is described below
commit ecc33d26e8fa5654cb19ce59c7d0155a8db8c139
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Dec 4 10:31:13 2024 +0900
[SPARK-50482][CORE] Deprecated no-op `spark.shuffle.spill` config
### What changes were proposed in this pull request?
This PR aims to deprecated no-op `spark.shuffle.spill` config in Apache
Spark 4.0.0.
- Previously, it complains only `spark.shuffle.spill=false`.
- We had better deprecate this configuration explicitly to simplify Apache
Spark 4+ configuration space.
### Why are the changes needed?
Since Apache Spark 1.6.0, `spark.shuffle.spill` has been ignored completely.
- https://github.com/apache/spark/pull/8831
### Does this PR introduce _any_ user-facing change?
There is no Spark shuffle behavior change. Only it will show additional
deprecation message when `spark.shuffle.spill=true` is used by users explicitly.
**BEFORE: Only spark.shuffle.spill=false**
```
24/12/03 10:37:23 WARN SortShuffleManager: spark.shuffle.spill was set to
false, but this configuration is ignored as of Spark 1.6+. Shuffle will
continue to spill to disk when necessary.
```
**AFTER: Both spark.shuffle.spill=false and spark.shuffle.spill=true**
```
24/12/03 10:37:47 WARN SparkConf: The configuration key
'spark.shuffle.spill' has been deprecated as of Spark 1.6 and may be removed in
the future. Not used anymore.
```
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #49047 from dongjoon-hyun/SPARK-50482.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
core/src/main/scala/org/apache/spark/SparkConf.scala | 1 +
.../scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala | 6 ------
2 files changed, 1 insertion(+), 6 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/SparkConf.scala
b/core/src/main/scala/org/apache/spark/SparkConf.scala
index cfb514913694..74abd5cbe0ae 100644
--- a/core/src/main/scala/org/apache/spark/SparkConf.scala
+++ b/core/src/main/scala/org/apache/spark/SparkConf.scala
@@ -608,6 +608,7 @@ private[spark] object SparkConf extends Logging {
"Please use spark.kryoserializer.buffer instead. The default value for
" +
"spark.kryoserializer.buffer.mb was previously specified as '0.064'.
Fractional values " +
"are no longer accepted. To specify the equivalent now, one may use
'64k'."),
+ DeprecatedConfig("spark.shuffle.spill", "1.6", "Not used anymore."),
DeprecatedConfig("spark.rpc", "2.0", "Not used anymore."),
DeprecatedConfig("spark.scheduler.executorTaskBlacklistTime", "2.1.0",
"Please use the new excludedOnFailure options,
spark.excludeOnFailure.*"),
diff --git
a/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala
b/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala
index efffda43695c..6902fb6d236d 100644
--- a/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala
+++ b/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala
@@ -74,12 +74,6 @@ private[spark] class SortShuffleManager(conf: SparkConf)
extends ShuffleManager
import SortShuffleManager._
- if (!conf.getBoolean("spark.shuffle.spill", true)) {
- logWarning(
- "spark.shuffle.spill was set to false, but this configuration is ignored
as of Spark 1.6+." +
- " Shuffle will continue to spill to disk when necessary.")
- }
-
/**
* A mapping from shuffle ids to the task ids of mappers producing output
for those shuffles.
*/
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
