My current best guess is that Spark does *not* fully support Hadoop 3.x because https://issues.apache.org/jira/browse/SPARK-18673 (updates to Hive shims for Hadoop 3.x) has not been resolved. There are also likely to be transitive dependency conflicts which will need to be resolved.
On Mon, Jan 8, 2018 at 8:52 AM akshay naidu <akshaynaid...@gmail.com> wrote: > yes , spark download page does mention that 2.2.1 is for 'hadoop-2.7 and > later', but my confusion is because spark was released on 1st dec and > hadoop-3 stable version released on 13th Dec. And to my similar question > on stackoverflow.com > <https://stackoverflow.com/questions/47920005/how-is-hadoop-3-0-0-s-compatibility-with-older-versions-of-hive-pig-sqoop-and> > , Mr. jacek-laskowski > <https://stackoverflow.com/users/1305344/jacek-laskowski> replied that > spark-2.2.1 doesn't support hadoop-3. so I am just looking for more clarity > on this doubt before moving on to upgrades. > > Thanks all for help. > > Akshay. > > On Mon, Jan 8, 2018 at 8:47 AM, Saisai Shao <sai.sai.s...@gmail.com> > wrote: > >> AFAIK, there's no large scale test for Hadoop 3.0 in the community. So it >> is not clear whether it is supported or not (or has some issues). I think >> in the download page "Pre-Built for Apache Hadoop 2.7 and later" mostly >> means that it supports Hadoop 2.7+ (2.8...), but not 3.0 (IIUC). >> >> Thanks >> Jerry >> >> 2018-01-08 4:50 GMT+08:00 Raj Adyanthaya <raj...@gmail.com>: >> >>> Hi Akshay >>> >>> On the Spark Download page when you select Spark 2.2.1 it gives you an >>> option to select package type. In that, there is an option to select >>> "Pre-Built for Apache Hadoop 2.7 and later". I am assuming it means that it >>> does support Hadoop 3.0. >>> >>> http://spark.apache.org/downloads.html >>> >>> Thanks, >>> Raj A. >>> >>> On Sat, Jan 6, 2018 at 8:23 PM, akshay naidu <akshaynaid...@gmail.com> >>> wrote: >>> >>>> hello Users, >>>> I need to know whether we can run latest spark on latest hadoop >>>> version i.e., spark-2.2.1 released on 1st dec and hadoop-3.0.0 released on >>>> 13th dec. >>>> thanks. >>>> >>> >>> >> >
