LuciferYang commented on PR #40847: URL: https://github.com/apache/spark/pull/40847#issuecomment-1520218583
@xkrogen @sunchao @pan3793 Synchronize my experimental results 1. Before building, we need to add the following content to `resource-managers/yarn/pom.xml` refer to https://github.com/apache/spark/pull/33160/files: ``` <profile> <id>no-shaded-hadoop-client</id> <dependencies> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-api</artifactId> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-common</artifactId> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-server-web-proxy</artifactId> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-client</artifactId> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-server-resourcemanager</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-server-tests</artifactId> <classifier>tests</classifier> <scope>test</scope> </dependency> </dependencies> </profile> ``` otherwise, the following compilation error will occurred with `-Pyarn`: ``` [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala:30: object MiniYARNCluster is not a member of package org.apache.hadoop.yarn.server [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala:65: not found: type MiniYARNCluster [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala:111: not found: type MiniYARNCluster [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:38: object resourcemanager is not a member of package org.apache.hadoop.yarn.server [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:39: object resourcemanager is not a member of package org.apache.hadoop.yarn.server [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:40: object resourcemanager is not a member of package org.apache.hadoop.yarn.server [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:41: object resourcemanager is not a member of package org.apache.hadoop.yarn.server [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:249: not found: type RMContext [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:251: not found: type RMApp [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:260: not found: type RMApplicationHistoryWriter [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:262: not found: type SystemMetricsPublisher [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:266: not found: type RMAppManager [ERROR] [Error] /${spark-source-dir}/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:271: not found: type ClientRMService ``` 2. With the above changeļ¼ master can build success with hadoop 3.1.x as following ``` build/mvn clean install -Dhadoop.version=3.1.4 -Dhadoop-client-api.artifact=hadoop-client -Dhadoop-client-runtime.artifact=hadoop-yarn-api -Dhadoop-client-minicluster.artifact=hadoop-client -DskipTests -Pno-shaded-hadoop-client -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive ``` Otherwise, cannot build `yarn` module with hadoop 3.1.x. 3. `hadoop-cloud` can't build with hadoop 3.1.x due to https://github.com/apache/spark/pull/40847#issuecomment-1518931685 **Overall, the current master cannot compile `yarn` and `hadoop-cloud` modules using hadoop 3.1.x without any changes, and all other modules are ok** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
