This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 228d12e [SPARK-36835][FOLLOWUP][BUILD][TEST-HADOOP2.7] Move
spark.yarn.isHadoopProvided to parent pom
228d12e is described below
commit 228d12e30ebc73bc4c72b5451489d088317b0099
Author: Chao Sun <[email protected]>
AuthorDate: Mon Sep 27 15:17:04 2021 +0800
[SPARK-36835][FOLLOWUP][BUILD][TEST-HADOOP2.7] Move
spark.yarn.isHadoopProvided to parent pom
### What changes were proposed in this pull request?
Move `spark.yarn.isHadoopProvided` to Spark parent pom, so that under
`resource-managers/yarn` we can make `hadoop-3.2` as the default profile.
### Why are the changes needed?
Currently under `resource-managers/yarn` there are 3 maven profiles :
`hadoop-provided`, `hadoop-2.7`, and `hadoop-3.2`, of which `hadoop-3.2` is
activated by default (via `activeByDefault`). The activation, however, doesn't
work when there is other explicitly activated profiles. In specific, if users
build Spark with `hadoop-provided`, maven will fail because it can't find
Hadoop 3.2 related dependencies, which are defined in the `hadoop-3.2` profile
section.
To fix the issue, this proposes to move the `hadoop-provided` section to
the parent pom. Currently this is only used to define a property
`spark.yarn.isHadoopProvided`, and it shouldn't matter where we define it.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Tested via running the command:
```
build/mvn clean package -DskipTests -B -Pmesos -Pyarn -Pkubernetes
-Pscala-2.12 -Phadoop-provided
```
which was failing before this PR but is succeeding with it.
Also checked active profiles with the command:
```
build/mvn -Pyarn -Phadoop-provided help:active-profiles
```
and it shows that `hadoop-3.2` is active for `spark-yarn` module now.
Closes #34110 from sunchao/SPARK-36835-followup2.
Authored-by: Chao Sun <[email protected]>
Signed-off-by: Gengliang Wang <[email protected]>
(cherry picked from commit f9efdeea8c1b7328d941b7e3d894a22afd3a42a2)
Signed-off-by: Gengliang Wang <[email protected]>
---
pom.xml | 4 ++++
resource-managers/yarn/pom.xml | 7 -------
2 files changed, 4 insertions(+), 7 deletions(-)
diff --git a/pom.xml b/pom.xml
index b33fbe9..d9c10ee 100644
--- a/pom.xml
+++ b/pom.xml
@@ -269,6 +269,7 @@
<hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
<hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
<hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+ <spark.yarn.isHadoopProvided>false</spark.yarn.isHadoopProvided>
<!--
Overridable test home. So that you can call individual pom files
directly without
@@ -3489,6 +3490,9 @@
-->
<profile>
<id>hadoop-provided</id>
+ <properties>
+ <spark.yarn.isHadoopProvided>true</spark.yarn.isHadoopProvided>
+ </properties>
</profile>
<profile>
<id>hive-provided</id>
diff --git a/resource-managers/yarn/pom.xml b/resource-managers/yarn/pom.xml
index da59900..cd5d408 100644
--- a/resource-managers/yarn/pom.xml
+++ b/resource-managers/yarn/pom.xml
@@ -30,17 +30,10 @@
<properties>
<sbt.project.name>yarn</sbt.project.name>
<jersey-1.version>1.19</jersey-1.version>
- <spark.yarn.isHadoopProvided>false</spark.yarn.isHadoopProvided>
</properties>
<profiles>
<profile>
- <id>hadoop-provided</id>
- <properties>
- <spark.yarn.isHadoopProvided>true</spark.yarn.isHadoopProvided>
- </properties>
- </profile>
- <profile>
<id>hadoop-2.7</id>
<dependencies>
<dependency>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
