Saisai Shao; Raj Adyanthaya; spark users
Subject: Re: Is Apache Spark-2.2.1 compatible with Hadoop-3.0.0
My current best guess is that Spark does not fully support Hadoop 3.x because
https://issues.apache.org/jira/browse/SPARK-18673 (updates to Hive shims for
Hadoop 3.x) has not been resolved. There
My current best guess is that Spark does *not* fully support Hadoop 3.x
because https://issues.apache.org/jira/browse/SPARK-18673 (updates to Hive
shims for Hadoop 3.x) has not been resolved. There are also likely to be
transitive dependency conflicts which will need to be resolved.
On Mon, Jan
yes , spark download page does mention that 2.2.1 is for 'hadoop-2.7 and
later', but my confusion is because spark was released on 1st dec and
hadoop-3 stable version released on 13th Dec. And to my similar question
on stackoverflow.com
AFAIK, there's no large scale test for Hadoop 3.0 in the community. So it
is not clear whether it is supported or not (or has some issues). I think
in the download page "Pre-Built for Apache Hadoop 2.7 and later" mostly
means that it supports Hadoop 2.7+ (2.8...), but not 3.0 (IIUC).
Thanks
Jerry
Hi Akshay
On the Spark Download page when you select Spark 2.2.1 it gives you an
option to select package type. In that, there is an option to select
"Pre-Built for Apache Hadoop 2.7 and later". I am assuming it means that it
does support Hadoop 3.0.
http://spark.apache.org/downloads.html
hello Users,
I need to know whether we can run latest spark on latest hadoop version
i.e., spark-2.2.1 released on 1st dec and hadoop-3.0.0 released on 13th dec.
thanks.
