This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new f35df10 [SPARK-35758][DOCS] Update the document about building Spark
with Hadoop for Hadoop 2.x and 3.x
f35df10 is described below
commit f35df10e2bab01c492bf627c7b12ce076a5da01e
Author: Kousuke Saruta <[email protected]>
AuthorDate: Tue Jun 15 20:19:50 2021 +0900
[SPARK-35758][DOCS] Update the document about building Spark with Hadoop
for Hadoop 2.x and 3.x
### What changes were proposed in this pull request?
This PR updates the document about building Spark with Hadoop for Hadoop
3.x and Hadoop 3.2.
### Why are the changes needed?
The document says about how to build like as follows:
```
./build/mvn -Pyarn -Dhadoop.version=2.8.5 -DskipTests clean package
```
But this command fails because the default build settings are for Hadoop
3.x.
So, we need to modify the command example.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
I confirmed both of these commands successfully finished.
```
./build/mvn -Pyarn -Dhadoop.version=3.3.0 -DskipTests package
./build/mvn -Phadoop-2.7 -Pyarn -Dhadoop.version=2.8.5 -DskipTests package
```
I also built the document and confirmed the result.
This is before:
And this is after:
Closes #32917 from sarutak/fix-build-doc-with-hadoop.
Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit d54edf0bde33c0e93cf33cb41d6be13eb32b6848)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
docs/building-spark.md | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/docs/building-spark.md b/docs/building-spark.md
index 5106f2a..286b48e 100644
--- a/docs/building-spark.md
+++ b/docs/building-spark.md
@@ -77,7 +77,11 @@ from `hadoop.version`.
Example:
- ./build/mvn -Pyarn -Dhadoop.version=2.8.5 -DskipTests clean package
+ ./build/mvn -Pyarn -Dhadoop.version=3.3.0 -DskipTests clean package
+
+If you want to build with Hadoop 2.x, enable hadoop-2.7 profile:
+
+ ./build/mvn -Phadoop-2.7 -Pyarn -Dhadoop.version=2.8.5 -DskipTests clean
package
## Building With Hive and JDBC Support
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
