Any suggestions from anyone?
Thanks
Bijoy
On Tue, Jun 10, 2014 at 11:46 PM, bijoy deb <bijoy.comput...@gmail.com>
wrote:
> Hi all,
>
> I have build Shark-0.9.1 using sbt using the below command:
>
> *SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.6.0 sbt/sbt assembly*
>
> My Hadoop cluster is also having version 2.0.0-mr1-cdh4.6.0.
>
> But when I try to execute the below command from Spark shell,which reads a
> file from HDFS, I get the "IPC version mismatch- IPC version 7 on server
> versus IPC version 4" on client error on org.apache.hadoop.hdfs.DFSClient
> class.
>
>
>
>
>
>
>
>
>
>
>
> *scala> val s = sc.textFile("hdfs://host:port/test.txt")scala>
> s.count()14/06/10 23:42:59 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable 14/06/10 23:42:59 WARN snappy.LoadSnappy: Snappy native library
> not loadedorg.apache.hadoop.ipc.RemoteException: Server IPC version 7
> cannot communicate with client version 4 at
> org.apache.hadoop.ipc.Client.call(Client.java:1070) at
> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at
> com.sun.proxy.$Proxy9.getProtocolVersion(Unknown Source) at
> org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at
> org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)*
> at
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:176)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
>
>
> Apparently this error is because of version mismatch of the hadoop-hdfs
> jar between client (one referred by Spark) and server(hadoop cluster).But
> what I don't understand is why is this mismatch (since I had built Spark
> with the correct Hadoop version).
>
> Any suggestions would be highly appreciated.
>
> Thanks
> Bijoy
>
