cxzl25 commented on code in PR #2073:
URL:
https://github.com/apache/incubator-celeborn/pull/2073#discussion_r1388845832
##########
project/CelebornBuild.scala:
##########
@@ -913,6 +913,31 @@ object MRClientProjects {
)
}
+ def mrIt: Project = {
+ val hadoopVersion = "3.3.6"
Review Comment:
Here are some dependency conflicts.
1. celeborn relies on guava 14, and hadoop-common for 3.2 relies on guava
27, so whichever version of guava is used here will result in
`NoSuchMethodError`.
<details>
<summary>NoSuchMethodError</summary>
### log
```
2023-11-10T11:28:08,905 ERROR [main]
org.apache.celeborn.mapreduce.v2.app.MRAppMasterWithCeleborn: Error starting
MRAppMaster
java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
~[hadoop-common-3.2.4.jar:?]
at
org.apache.celeborn.mapreduce.v2.app.MRAppMasterWithCeleborn.<init>(MRAppMasterWithCeleborn.java:73)
~[classes/:?]
at
org.apache.celeborn.mapreduce.v2.app.MRAppMasterWithCeleborn.main(MRAppMasterWithCeleborn.java:149)
~[classes/:?]
```
</details>
But 3.3 removes some guava dependencies and replaces them with a guava shade.
[[HADOOP-17098](https://issues.apache.org/jira/browse/HADOOP-17098)] Reduce
Guava dependency in Hadoop source code
[[HADOOP-17288](https://issues.apache.org/jira/browse/HADOOP-17288)] Use
shaded guava from thirdparty
2. Since MiniYARNCluster is started with YARN with some shaded packages, but
MR depends on `hadoop-mapreduce-client-app` which uses hadoop jar without
shaded, so here we have to put MR dependent jar at the top of MR task classpath
using `mapreduce.job.user. classpath.first=true` and `job.addFileToClassPath`
to put the MR dependent jar at the top of the MR task classpath.
<details>
<summary>NoSuchMethodError</summary>
### log
```
[info] Cause: java.lang.NoSuchMethodError:
org.apache.hadoop.yarn.webapp.WebApps$Builder.start(Lorg/apache/hadoop/yarn/webapp/WebApp;Lorg/apache/hadoop/shaded/org/eclipse/jetty/webapp/WebAppContext;)Lorg/apache/hadoop/yarn/webapp/WebApp;
[info] at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1389)
[info] at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1498)
[info] at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
[info] at
org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:376)
[info] at
org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
[info] at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
```
</details>
<details>
<summary>NoSuchMethodError</summary>
### log
```
[info] Cause: java.lang.NoSuchMethodError:
org.apache.hadoop.security.authentication.server.AuthenticationFilter.constructSecretProvider(Ljavax/servlet/ServletContext;Ljava/util/Properties;Z)Lorg/apache/hadoop/security/authentication/util/SignerSecretProvider;
[info] at
org.apache.hadoop.http.HttpServer2.constructSecretProvider(HttpServer2.java:799)
[info] at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:685)
[info] at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:129)
[info] at
org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:478)
[info] at
org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:372)
[info] at
org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:465)
[info] at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1389)
[info] at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1498)
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
