I've build a recent version of Spark (commit
23af00f9e0e5108f62cdb9629e3eb4e54bbaa321).
My Hadoop version is -
*SPARK_HADOOP_VERSION=0.20.2-cdh3u6*
I've a very simple Standalone app that I want to run on my cluster. The
simple.sbt for that app looks like.
name := "Simple Project"
version := "1.0"
scalaVersion := "2.9.3"
libraryDependencies += "org.apache.spark" %% "spark-core" %
"0.8.1-incubating"
libraryDependencies += "org.apache.hadoop" % "hadoop-client" %
"0.20.2-cdh3u6"
resolvers ++= Seq("Akka Repository" at "http://repo.akka.io/releases/","Spray
Repository" at "http://repo.spray.cc/";)
I can do a sbt package successfully. However when I do a sbt run I get the
following exception. I guess the spark-core version above is wrong. How do
I make it point to the local build I've or should be revert back to the
0.8.1-incubating ?
[error] (*run-main) org.apache.hadoop.ipc.RPC$VersionMismatch: Protocol
org.apache.hadoop.hdfs.protocol.ClientProtocol version mismatch. (client =
61, server = 63)*
org.apache.hadoop.ipc.RPC$VersionMismatch: Protocol
org.apache.hadoop.hdfs.protocol.ClientProtocol version mismatch. (client =
61, server = 63)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:401)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:176)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:141)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:201)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:199)
at scala.Option.getOrElse(Option.scala:108)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:199)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:26)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:201)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:199)
at scala.Option.getOrElse(Option.scala:108)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:199)
at org.apache.spark.rdd.RDD.take(RDD.scala:789)
at SimpleApp$.main(SimpleApp.scala:12)
at SimpleApp.main(SimpleApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
[trace] Stack trace suppressed: run last compile:run for the full output.
14/02/04 20:52:28 INFO network.ConnectionManager: Selector thread was
interrupted!
