This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new d944c0b17d67 [SPARK-57620][CORE] Remove redundant S3A Vectored IO
Hadoop configurations
d944c0b17d67 is described below
commit d944c0b17d6706d4e0e666942356948c815f855f
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Jun 22 19:13:48 2026 -0700
[SPARK-57620][CORE] Remove redundant S3A Vectored IO Hadoop configurations
### What changes were proposed in this pull request?
This PR aims to remove redundant `S3A Vectored IO` Hadoop configurations
from Apache Spark 4.3.0.
### Why are the changes needed?
Apache Spark 4.0.0 added these Hadoop configurations to aline with Apache
Hadoop 3.4.2.
- https://github.com/apache/spark/pull/49748
Since Apache Spark 4.1.0, this is not required because we are using the
latest Hadoop.
- #51127 (Apache Spark 4.1.0)
- #54448 (Apache Spark 4.2.0)
### Does this PR introduce _any_ user-facing change?
No behavior change because we used the default value of Apache Hadoop 3.4.2+
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #56678 from dongjoon-hyun/SPARK-57620.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit c5d5a303cea680d62d50848d7585267b894d2fcf)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/SparkContext.scala | 4 ----
1 file changed, 4 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index e00e34b89b84..0a701e1967bd 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -423,10 +423,6 @@ class SparkContext(config: SparkConf) extends Logging {
if (!_conf.contains("spark.app.name")) {
throw new SparkException("An application name must be set in your
configuration")
}
- // HADOOP-19229 Vector IO on cloud storage: increase threshold for range
merging
- // We can remove this after Apache Hadoop 3.4.2 releases
- conf.setIfMissing("spark.hadoop.fs.s3a.vectored.read.min.seek.size",
"128K")
- conf.setIfMissing("spark.hadoop.fs.s3a.vectored.read.max.merged.size",
"2M")
// This should be set as early as possible.
SparkContext.enableMagicCommitterIfNeeded(_conf)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
