Pat - Can u please be more specific on which hadoop version the prebuilt spark
binaries CDH4 used?
is it 2.0.0-cdh4.2.0 or 2.0.0-mr1-cdh4.2.0
I am blocked here cause all my efforts to build spark 0.8.0 with different
spark hadoop settings did not resolve my exception when invoking
javaRDD.saveastextfile
org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate
with client version 4
at org.apache.hadoop.ipc.Client.call(Client.java:1113)
same code works if I run it under hadoop 1.2.1 but not under CDH4 which has
Hadoop 2.0.0-cdh4.4.0
Thanks,
Hussam
From: Patrick Wendell [mailto:[email protected]]
Sent: Thursday, November 21, 2013 11:29 AM
To: [email protected]
Subject: Re: setting SPARK_HADOOP_VERSION
It was 4.2.0 - we've kept it this way for the last few releases for consistency.
On Thu, Nov 21, 2013 at 11:19 AM, Pierre Borckmans
<[email protected]<mailto:[email protected]>>
wrote:
You are right about the typo. I did a copy paste from the website which
presents this mistake.
I forgot about this even though I submitted the correction to GutHub last week
;)
Message sent from a mobile service - excuse typos and abbreviations
Le 21 nov. 2013 à 20:16,
<[email protected]<mailto:[email protected]>> a écrit :
Thank u Pierre. I think u had typo, as it should be cdh and not chd
Anyone know the prebuilt spark binaries for CDH4 which hadoop version was built
against?
http://spark.incubator.apache.org/downloads.html
was it 2.0.0-cdh4.4.0 or 2.0.0-mr1-cdh4.4.0 ?
Thanks,
Hussam
From: Pierre Borckmans [mailto:[email protected]]
Sent: Thursday, November 21, 2013 10:33 AM
To: [email protected]<mailto:[email protected]>
Subject: Re: setting SPARK_HADOOP_VERSION
Given your cdh4 distribution, you should use
2.0.0-mr1-chd4.4.0
That is if you don't want to use YARN
Otherwise you should use:
2.0.0-chd4.4.0
I think you were misreading the table as the column you mentioned is for the
HDP distribution.
HTH
Pierre
Sent from a mobile, excuse typos and abbreviations
Message sent from a mobile service - excuse typos and abbreviations
Le 21 nov. 2013 à 18:42,
<[email protected]<mailto:[email protected]>> a écrit :
Hi,
Can someone advise please on the following setting of SPARK_HADOOP_VERSION?
I have spark 0.8 pre compiled binary of CDH4 downloaded from spark site,
running in CDH4 env.
Running hadoop version shows
[spark@kserv06 ~]$ hadoop version
Hadoop 2.0.0-cdh4.4.0
Subversion
file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.4.0/src/hadoop-common-project/hadoop-common<file:///\\data\1\jenkins\workspace\generic-package-rhel64-6-0\topdir\BUILD\hadoop-2.0.0-cdh4.4.0\src\hadoop-common-project\hadoop-common>
-r c0eba6cd38c984557e96a16ccd7356b7de835e79
Compiled by jenkins on Tue Sep 3 19:33:17 PDT 2013
>From source with checksum ac7e170aa709b3ace13dc5f775487180
This command was run using
/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop/hadoop-common-2.0.0-cdh4.4.0.jar
[spark@kserv06 ~]$
So per checking
http://spark.incubator.apache.org/docs/latest/hadoop-third-party-distributions.html
SPARK_HADOOP_VERSION should be 1.2.0 ... is that correct?
Not sure if it's related but I am getting this exception when invoking
javaRDD.saveAsTextFile
exception 'org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot
communicate with client version 4'
Thanks,
Hussam
