Zhang, Liye created SPARK-6689: ---------------------------------- Summary: MiniYarnCLuster still test failed with hadoop-2.2 Key: SPARK-6689 URL: https://issues.apache.org/jira/browse/SPARK-6689 Project: Spark Issue Type: Test Components: Tests, YARN Affects Versions: 1.3.0 Reporter: Zhang, Liye
when running unit test *YarnClusterSuite* with *hadoop-2.2*, exception will throw because *Timed out waiting for RM to come up*. Some previously related discussion can be traced in [spark-3710|https://issues.apache.org/jira/browse/SPARK-3710] ([PR2682|https://github.com/apache/spark/pull/2682]) and [spark-2778|https://issues.apache.org/jira/browse/SPARK-2778] ([PR2605|https://github.com/apache/spark/pull/2605]). With command *build/sbt -Pyarn -Phadoop-2.2 "test-only org.apache.spark.deploy.yarn.YarnClusterSuite"*, will get following exceptions: {noformat} [info] Exception encountered when attempting to run a suite with class name: org.apache.spark.deploy.yarn.YarnClusterSuite *** ABORTED *** (15 seconds, 799 milliseconds) [info] java.lang.IllegalStateException: Timed out waiting for RM to come up. [info] at org.apache.spark.deploy.yarn.YarnClusterSuite.beforeAll(YarnClusterSuite.scala:114) [info] at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187) [info] at org.apache.spark.deploy.yarn.YarnClusterSuite.beforeAll(YarnClusterSuite.scala:44) [info] at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253) [info] at org.apache.spark.deploy.yarn.YarnClusterSuite.run(YarnClusterSuite.scala:44) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671) [info] at sbt.ForkMain$Run$2.call(ForkMain.java:294) [info] at sbt.ForkMain$Run$2.call(ForkMain.java:284) [info] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [info] at java.lang.Thread.run(Thread.java:745) {noformat} And without *-Phadoop-2.2* or replace it with "*-Dhadoop.version*" (e.g. build/sbt -Pyarn "test-only org.apache.spark.deploy.yarn.YarnClusterSuite") more info will come out: {noformat} Exception in thread "Thread-7" java.lang.NoClassDefFoundError: org/mortbay/jetty/servlet/Context at org.apache.hadoop.yarn.webapp.WebApps.$for(WebApps.java:309) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:602) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:655) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper$2.run(MiniYARNCluster.java:219) Caused by: java.lang.ClassNotFoundException: org.mortbay.jetty.servlet.Context at java.net.URLClassLoader$1.run(URLClassLoader.java:372) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:360) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) [info] Resolving org.apache.hadoop#hadoop-yarn-server-common;2.2.0 ... Exception in thread "Thread-18" java.lang.NoClassDefFoundError: org/mortbay/jetty/servlet/Context at org.apache.hadoop.yarn.webapp.WebApps.$for(WebApps.java:309) at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:62) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:199) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.MiniYARNCluster$NodeManagerWrapper$1.run(MiniYARNCluster.java:337) Caused by: java.lang.ClassNotFoundException: org.mortbay.jetty.servlet.Context at java.net.URLClassLoader$1.run(URLClassLoader.java:372) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:360) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) [info] Resolving org.apache.hadoop#hadoop-yarn;2.2.0 ... [info] Exception encountered when attempting to run a suite with class name: org.apache.spark.deploy.yarn.YarnClusterSuite *** ABORTED *** (15 seconds, 799 milliseconds) [info] java.lang.IllegalStateException: Timed out waiting for RM to come up. [info] at org.apache.spark.deploy.yarn.YarnClusterSuite.beforeAll(YarnClusterSuite.scala:114) [info] at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187) [info] at org.apache.spark.deploy.yarn.YarnClusterSuite.beforeAll(YarnClusterSuite.scala:44) [info] at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253) [info] at org.apache.spark.deploy.yarn.YarnClusterSuite.run(YarnClusterSuite.scala:44) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671) [info] at sbt.ForkMain$Run$2.call(ForkMain.java:294) [info] at sbt.ForkMain$Run$2.call(ForkMain.java:284) [info] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [info] at java.lang.Thread.run(Thread.java:745) {noformat} If we add the following code into *yarn/pom.xml*, {code:xml} <dependency> <groupId>javax.ws.rs</groupId> <artifactId>jsr311-api</artifactId> <version>1.1.1</version> <scope>test</scope> </dependency> <dependency> <groupId>org.mortbay.jetty</groupId> <artifactId>jetty</artifactId> <version>6.1.26</version> <scope>test</scope> </dependency> {code} The error message would be : {noformat} Exception in thread "Thread-188" org.apache.hadoop.yarn.exceptions.YarnRuntimeException: NMWebapps failed to start. at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:76) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:199) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.MiniYARNCluster$NodeManagerWrapper$1.run(MiniYARNCluster.java:337) Caused by: org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:262) at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:62) ... 5 more Caused by: java.io.IOException: Problem in starting http server. Server handlers failed at org.apache.hadoop.http.HttpServer.start(HttpServer.java:700) at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:257) [info] Resolving org.apache.hadoop#hadoop-yarn-common;2.2.0 ... [info] Exception encountered when attempting to run a suite with class name: org.apache.spark.deploy.yarn.YarnClusterSuite *** ABORTED *** (3 seconds, 455 milliseconds) [info] org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException: NodeManager 0 failed to start [info] at org.apache.hadoop.yarn.server.MiniYARNCluster$NodeManagerWrapper.serviceStart(MiniYARNCluster.java:352) [info] at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) [info] at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) [info] at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) [info] at org.apache.spark.deploy.yarn.YarnClusterSuite.beforeAll(YarnClusterSuite.scala:94) [info] at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187) [info] at org.apache.spark.deploy.yarn.YarnClusterSuite.beforeAll(YarnClusterSuite.scala:44) [info] at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253) [info] at org.apache.spark.deploy.yarn.YarnClusterSuite.run(YarnClusterSuite.scala:44) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671) [info] at sbt.ForkMain$Run$2.call(ForkMain.java:294) [info] at sbt.ForkMain$Run$2.call(ForkMain.java:284) [info] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [info] at java.lang.Thread.run(Thread.java:745) [info] Cause: java.io.IOException: NodeManager 0 failed to start [info] at org.apache.hadoop.yarn.server.MiniYARNCluster$NodeManagerWrapper.serviceStart(MiniYARNCluster.java:348) [info] at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) [info] at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) [info] at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) [info] at org.apache.spark.deploy.yarn.YarnClusterSuite.beforeAll(YarnClusterSuite.scala:94) [info] at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187) [info] at org.apache.spark.deploy.yarn.YarnClusterSuite.beforeAll(YarnClusterSuite.scala:44) [info] at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253) [info] at org.apache.spark.deploy.yarn.YarnClusterSuite.run(YarnClusterSuite.scala:44) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671) [info] at sbt.ForkMain$Run$2.call(ForkMain.java:294) [info] at sbt.ForkMain$Run$2.call(ForkMain.java:284) [info] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [info] at java.lang.Thread.run(Thread.java:745) {noformat} *NOTE: This issue does not exists for hadoop-2.2+* -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org