LuciferYang commented on a change in pull request #30701:
URL: https://github.com/apache/spark/pull/30701#discussion_r710172178
##########
File path: core/pom.xml
##########
@@ -66,7 +66,13 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
Review comment:
@sunchao @dongjoon-hyun I find there are some bad case with using
variables here:
- if we run `mvn dependency:tree -pl resource-managers/yarn -Phadoop-2.7
-Pyarn -am`, the dependency-tree is correct
- if we run `mvn dependency:tree -pl resource-managers/yarn -Phadoop-2.7
-Pyarn`, the dependency-tree is wrong, the dependency of hadoop 3.3.1 exists in
the dependency tree, even though we have added the profile of - Phadoop2.7
So if we execute
```
mvn clean install -DskipTests -pl resource-managers/yarn -am -Phadoop-2.7
-Pyarn
mvn test -pl resource-managers/yarn -Phadoop-2.7 -Pyarn
-DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite
```
The test case `YarnClusterSuite` will failed as follows:
```
Discovery starting.
Discovery completed in 277 milliseconds.
Run starting. Expected test count is: 27
YarnClusterSuite:
*** RUN ABORTED ***
java.lang.NoClassDefFoundError:
org/apache/hadoop/shaded/com/google/inject/servlet/ServletModule
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
...
Cause: java.lang.ClassNotFoundException:
org.apache.hadoop.shaded.com.google.inject.servlet.ServletModule
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
