This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9d6b9f7305f2 [SPARK-47529][DOCS] Use hadoop 3.4.0 in some docs
9d6b9f7305f2 is described below
commit 9d6b9f7305f208a8b66f5f390e063db86b5678cd
Author: panbingkun <[email protected]>
AuthorDate: Sat Mar 23 18:27:29 2024 -0700
[SPARK-47529][DOCS] Use hadoop 3.4.0 in some docs
### What changes were proposed in this pull request?
This PR aims to update `Hadoop` dependency in some docs.
### Why are the changes needed?
Currently Spark codebase master is using Apache Hadoop `3.4.0` by default.
### Does this PR introduce _any_ user-facing change?
No. This is a doc-only change.
### How was this patch tested?
N/A.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #45679 from panbingkun/minor_use_hadoop_3.4.
Authored-by: panbingkun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
assembly/README | 2 +-
docs/building-spark.md | 2 +-
docs/running-on-kubernetes.md | 2 +-
resource-managers/kubernetes/integration-tests/README.md | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/assembly/README b/assembly/README
index 3dde243d3e69..ad1305c5b4d5 100644
--- a/assembly/README
+++ b/assembly/README
@@ -9,4 +9,4 @@ This module is off by default. To activate it specify the
profile in the command
If you need to build an assembly for a different version of Hadoop the
hadoop-version system property needs to be set as in this example:
- -Dhadoop.version=3.3.6
+ -Dhadoop.version=3.4.0
diff --git a/docs/building-spark.md b/docs/building-spark.md
index 3d12b521c024..56efbc1a0110 100644
--- a/docs/building-spark.md
+++ b/docs/building-spark.md
@@ -79,7 +79,7 @@ from `hadoop.version`.
Example:
- ./build/mvn -Pyarn -Dhadoop.version=3.3.0 -DskipTests clean package
+ ./build/mvn -Pyarn -Dhadoop.version=3.4.0 -DskipTests clean package
## Building With Hive and JDBC Support
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 8c92bd9f8cb3..01e9d6382c18 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -236,7 +236,7 @@ A typical example of this using S3 is via passing the
following options:
```
...
---packages org.apache.hadoop:hadoop-aws:3.2.2
+--packages org.apache.hadoop:hadoop-aws:3.4.0
--conf spark.kubernetes.file.upload.path=s3a://<s3-bucket>/path
--conf spark.hadoop.fs.s3a.access.key=...
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
diff --git a/resource-managers/kubernetes/integration-tests/README.md
b/resource-managers/kubernetes/integration-tests/README.md
index c0d92d988b1a..f8070ec4ce93 100644
--- a/resource-managers/kubernetes/integration-tests/README.md
+++ b/resource-managers/kubernetes/integration-tests/README.md
@@ -130,7 +130,7 @@ properties to Maven. For example:
mvn integration-test -am -pl :spark-kubernetes-integration-tests_2.13 \
-Pkubernetes -Pkubernetes-integration-tests \
- -Phadoop-3 -Dhadoop.version=3.3.6 \
+ -Phadoop-3 -Dhadoop.version=3.4.0 \
-Dspark.kubernetes.test.sparkTgz=spark-4.0.0-SNAPSHOT-bin-example.tgz \
-Dspark.kubernetes.test.imageTag=sometag \
-Dspark.kubernetes.test.imageRepo=docker.io/somerepo \
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
