This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 9d6b9f7305f2 [SPARK-47529][DOCS] Use hadoop 3.4.0 in some docs 9d6b9f7305f2 is described below commit 9d6b9f7305f208a8b66f5f390e063db86b5678cd Author: panbingkun <panbing...@baidu.com> AuthorDate: Sat Mar 23 18:27:29 2024 -0700 [SPARK-47529][DOCS] Use hadoop 3.4.0 in some docs ### What changes were proposed in this pull request? This PR aims to update `Hadoop` dependency in some docs. ### Why are the changes needed? Currently Spark codebase master is using Apache Hadoop `3.4.0` by default. ### Does this PR introduce _any_ user-facing change? No. This is a doc-only change. ### How was this patch tested? N/A. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45679 from panbingkun/minor_use_hadoop_3.4. Authored-by: panbingkun <panbing...@baidu.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- assembly/README | 2 +- docs/building-spark.md | 2 +- docs/running-on-kubernetes.md | 2 +- resource-managers/kubernetes/integration-tests/README.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/assembly/README b/assembly/README index 3dde243d3e69..ad1305c5b4d5 100644 --- a/assembly/README +++ b/assembly/README @@ -9,4 +9,4 @@ This module is off by default. To activate it specify the profile in the command If you need to build an assembly for a different version of Hadoop the hadoop-version system property needs to be set as in this example: - -Dhadoop.version=3.3.6 + -Dhadoop.version=3.4.0 diff --git a/docs/building-spark.md b/docs/building-spark.md index 3d12b521c024..56efbc1a0110 100644 --- a/docs/building-spark.md +++ b/docs/building-spark.md @@ -79,7 +79,7 @@ from `hadoop.version`. Example: - ./build/mvn -Pyarn -Dhadoop.version=3.3.0 -DskipTests clean package + ./build/mvn -Pyarn -Dhadoop.version=3.4.0 -DskipTests clean package ## Building With Hive and JDBC Support diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md index 8c92bd9f8cb3..01e9d6382c18 100644 --- a/docs/running-on-kubernetes.md +++ b/docs/running-on-kubernetes.md @@ -236,7 +236,7 @@ A typical example of this using S3 is via passing the following options: ``` ... ---packages org.apache.hadoop:hadoop-aws:3.2.2 +--packages org.apache.hadoop:hadoop-aws:3.4.0 --conf spark.kubernetes.file.upload.path=s3a://<s3-bucket>/path --conf spark.hadoop.fs.s3a.access.key=... --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem diff --git a/resource-managers/kubernetes/integration-tests/README.md b/resource-managers/kubernetes/integration-tests/README.md index c0d92d988b1a..f8070ec4ce93 100644 --- a/resource-managers/kubernetes/integration-tests/README.md +++ b/resource-managers/kubernetes/integration-tests/README.md @@ -130,7 +130,7 @@ properties to Maven. For example: mvn integration-test -am -pl :spark-kubernetes-integration-tests_2.13 \ -Pkubernetes -Pkubernetes-integration-tests \ - -Phadoop-3 -Dhadoop.version=3.3.6 \ + -Phadoop-3 -Dhadoop.version=3.4.0 \ -Dspark.kubernetes.test.sparkTgz=spark-4.0.0-SNAPSHOT-bin-example.tgz \ -Dspark.kubernetes.test.imageTag=sometag \ -Dspark.kubernetes.test.imageRepo=docker.io/somerepo \ --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org