Hi all, I get spark 1.0 snapshot code from git; and I compiled it using command: mvn -Pbigtop-dist -Dhadoop.version=2.3.0 -Dyarn.version=2.3.0 -DskipTests package -e
in cluster, I add [export SPARK_YARN_MODE=true] to spark-env.sh, and run HdfsTest examples; and I got error, any one got similar issue? SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/nosec/spark/lib/spark-examples_2.10-assembly-0.9.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/nosec/spark/assembly/target/scala-2.10/spark-assembly_2.10-1.0.0-SNAPSHOT-hadoop2.3.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory] 1 [main] INFO org.apache.spark.SecurityManager - SecurityManager, is authentication enabled: false are ui acls enabled: false users with view permissions: Set(hadoop) 587 [spark-akka.actor.default-dispatcher-2] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 674 [spark-akka.actor.default-dispatcher-2] INFO Remoting - Starting remoting 888 [spark-akka.actor.default-dispatcher-5] INFO Remoting - Remoting started; listening on addresses :[akka.tcp://sp...@vm-6.com:45972] 888 [spark-akka.actor.default-dispatcher-5] INFO Remoting - Remoting now listens on addresses: [akka.tcp://sp...@vm-6.com:45972] 900 [main] INFO org.apache.spark.SparkEnv - Registering MapOutputTracker 903 [main] INFO org.apache.spark.SparkEnv - Registering BlockManagerMaster 932 [main] INFO org.apache.spark.storage.DiskBlockManager - Created local directory at /opt/nosec/spark/local/spark-local-20140429013124-3e55 936 [main] INFO org.apache.spark.storage.MemoryStore - MemoryStore started with capacity 297.0 MB. 966 [main] INFO org.apache.spark.network.ConnectionManager - Bound socket to port 59708 with id = ConnectionManagerId(VM-6.com,59708) 971 [main] INFO org.apache.spark.storage.BlockManagerMaster - Trying to register BlockManager 973 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.storage.BlockManagerInfo - Registering block manager VM-6.com:59708 with 297.0 MB RAM 974 [main] INFO org.apache.spark.storage.BlockManagerMaster - Registered BlockManager 989 [main] INFO org.apache.spark.HttpServer - Starting HTTP Server 1888 [main] INFO org.eclipse.jetty.server.Server - jetty-7.x.y-SNAPSHOT 1909 [main] INFO org.eclipse.jetty.server.AbstractConnector - Started SocketConnector@0.0.0.0:58102 1910 [main] INFO org.apache.spark.broadcast.HttpBroadcast - Broadcast server started at http://9.91.11.28:58102 1918 [main] INFO org.apache.spark.HttpFileServer - HTTP File server directory is /tmp/spark-375beb29-df90-4b87-ab07-2a6855daf342 1918 [main] INFO org.apache.spark.HttpServer - Starting HTTP Server 1918 [main] INFO org.eclipse.jetty.server.Server - jetty-7.x.y-SNAPSHOT 1920 [main] INFO org.eclipse.jetty.server.AbstractConnector - Started SocketConnector@0.0.0.0:52144 2298 [main] INFO org.eclipse.jetty.server.Server - jetty-7.x.y-SNAPSHOT 2299 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/stages,null} 2301 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/stages/json,null} 2301 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/stages/stage,null} 2301 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/stages/stage/json,null} 2301 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/stages/pool,null} 2301 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/stages/pool/json,null} 2302 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/storage,null} 2302 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/storage/json,null} 2302 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/storage/rdd,null} 2302 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/storage/rdd/json,null} 2302 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/environment,null} 2302 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/environment/json,null} 2302 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/executors,null} 2302 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/executors/json,null} 2303 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/static,null} 2303 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/,null} 2303 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/stages/stage/kill,null} 2303 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.ServletContextHandler{/metrics/json,null} 2313 [main] INFO org.eclipse.jetty.server.AbstractConnector - Started SelectChannelConnector@0.0.0.0:4040 2314 [main] INFO org.apache.spark.ui.SparkUI - Started SparkUI at http://VM-6.com:4040 Exception in thread "main" java.lang.ExceptionInInitializerError at org.apache.spark.SparkContext.addJar(SparkContext.scala:894) at org.apache.spark.SparkContext$$anonfun$5.apply(SparkContext.scala:234) at org.apache.spark.SparkContext$$anonfun$5.apply(SparkContext.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34) at org.apache.spark.SparkContext.<init>(SparkContext.scala:234) at org.apache.spark.SparkContext.<init>(SparkContext.scala:110) at org.apache.spark.examples.HdfsTest$.main(HdfsTest.scala:24) at org.apache.spark.examples.HdfsTest.main(HdfsTest.scala) Caused by: org.apache.spark.SparkException: Unable to load YARN support at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:91) at org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:86) at org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala) ... 9 more Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.YarnSparkHadoopUtil at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:186) at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:87) ... 11 more Thanks. Peter Shi