Not working as I am still getting the exception org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
when using setting SPARK_HADOOP_VERSION=2.0.0-cdh4.2.0 The only option I have now is compile spark 0.8 branch in my CDH4 env. The prebuilt spark binaries does not work in my CDH4 env. Thnaks, Hussam From: Jarada, Hussam Sent: Thursday, November 21, 2013 11:33 AM To: [email protected] Subject: RE: setting SPARK_HADOOP_VERSION Dell - Internal Use - Confidential So my setting should be SPARK_HADOOP_VERSION=2.0.0-cdh4.2.0 When using prebuilt spark binary from the download site. Let me try it cause I am still getting this exception when trying to run with setting 2.0.0-cdh4.4.0 or 2.0.0-mr1-cdh4.4.0 exception 'org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4' From: Patrick Wendell [mailto:[email protected]] Sent: Thursday, November 21, 2013 11:29 AM To: [email protected]<mailto:[email protected]> Subject: Re: setting SPARK_HADOOP_VERSION It was 4.2.0 - we've kept it this way for the last few releases for consistency. On Thu, Nov 21, 2013 at 11:19 AM, Pierre Borckmans <[email protected]<mailto:[email protected]>> wrote: You are right about the typo. I did a copy paste from the website which presents this mistake. I forgot about this even though I submitted the correction to GutHub last week ;) Message sent from a mobile service - excuse typos and abbreviations Le 21 nov. 2013 à 20:16, <[email protected]<mailto:[email protected]>> a écrit : Thank u Pierre. I think u had typo, as it should be cdh and not chd Anyone know the prebuilt spark binaries for CDH4 which hadoop version was built against? http://spark.incubator.apache.org/downloads.html was it 2.0.0-cdh4.4.0 or 2.0.0-mr1-cdh4.4.0 ? Thanks, Hussam From: Pierre Borckmans [mailto:[email protected]] Sent: Thursday, November 21, 2013 10:33 AM To: [email protected]<mailto:[email protected]> Subject: Re: setting SPARK_HADOOP_VERSION Given your cdh4 distribution, you should use 2.0.0-mr1-chd4.4.0 That is if you don't want to use YARN Otherwise you should use: 2.0.0-chd4.4.0 I think you were misreading the table as the column you mentioned is for the HDP distribution. HTH Pierre Sent from a mobile, excuse typos and abbreviations Message sent from a mobile service - excuse typos and abbreviations Le 21 nov. 2013 à 18:42, <[email protected]<mailto:[email protected]>> a écrit : Hi, Can someone advise please on the following setting of SPARK_HADOOP_VERSION? I have spark 0.8 pre compiled binary of CDH4 downloaded from spark site, running in CDH4 env. Running hadoop version shows [spark@kserv06 ~]$ hadoop version Hadoop 2.0.0-cdh4.4.0 Subversion file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.4.0/src/hadoop-common-project/hadoop-common<file:///\\data\1\jenkins\workspace\generic-package-rhel64-6-0\topdir\BUILD\hadoop-2.0.0-cdh4.4.0\src\hadoop-common-project\hadoop-common> -r c0eba6cd38c984557e96a16ccd7356b7de835e79 Compiled by jenkins on Tue Sep 3 19:33:17 PDT 2013 >From source with checksum ac7e170aa709b3ace13dc5f775487180 This command was run using /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop/hadoop-common-2.0.0-cdh4.4.0.jar [spark@kserv06 ~]$ So per checking http://spark.incubator.apache.org/docs/latest/hadoop-third-party-distributions.html SPARK_HADOOP_VERSION should be 1.2.0 ... is that correct? Not sure if it's related but I am getting this exception when invoking javaRDD.saveAsTextFile exception 'org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4' Thanks, Hussam
