You are right about the typo. I did a copy paste from the website which presents this mistake. I forgot about this even though I submitted the correction to GutHub last week ;)
Message sent from a mobile service - excuse typos and abbreviations > Le 21 nov. 2013 à 20:16, <[email protected]> a écrit : > > Thank u Pierre. I think u had typo, as it should be cdh and not chd > > Anyone know the prebuilt spark binaries for CDH4 which hadoop version was > built against? > > http://spark.incubator.apache.org/downloads.html > > was it 2.0.0-cdh4.4.0 or 2.0.0-mr1-cdh4.4.0 ? > > Thanks, > Hussam > From: Pierre Borckmans [mailto:[email protected]] > Sent: Thursday, November 21, 2013 10:33 AM > To: [email protected] > Subject: Re: setting SPARK_HADOOP_VERSION > > Given your cdh4 distribution, you should use > 2.0.0-mr1-chd4.4.0 > > That is if you don't want to use YARN > > Otherwise you should use: > 2.0.0-chd4.4.0 > > I think you were misreading the table as the column you mentioned is for the > HDP distribution. > > HTH > > Pierre > > Sent from a mobile, excuse typos and abbreviations > > Message sent from a mobile service - excuse typos and abbreviations > > Le 21 nov. 2013 à 18:42, <[email protected]> a écrit : > > Hi, > > Can someone advise please on the following setting of SPARK_HADOOP_VERSION? > > I have spark 0.8 pre compiled binary of CDH4 downloaded from spark site, > running in CDH4 env. > > Running hadoop version shows > > [spark@kserv06 ~]$ hadoop version > Hadoop 2.0.0-cdh4.4.0 > Subversion > file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.4.0/src/hadoop-common-project/hadoop-common > -r c0eba6cd38c984557e96a16ccd7356b7de835e79 > Compiled by jenkins on Tue Sep 3 19:33:17 PDT 2013 > From source with checksum ac7e170aa709b3ace13dc5f775487180 > This command was run using > /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop/hadoop-common-2.0.0-cdh4.4.0.jar > [spark@kserv06 ~]$ > > So per checking > http://spark.incubator.apache.org/docs/latest/hadoop-third-party-distributions.html > > SPARK_HADOOP_VERSION should be 1.2.0 … is that correct? > > Not sure if it’s related but I am getting this exception when invoking > javaRDD.saveAsTextFile > > exception 'org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot > communicate with client version 4' > > > Thanks, > Hussam
