Re: Is Apache Spark-2.2.1 compatible with Hadoop-3.0.0

[Felix Cheung]

And Hadoop-3.x is not part of the release and sign off for 2.2.1.

Maybe we could update the website to avoid any confusion with "later".


From: Josh Rosen <joshro...@databricks.com>
Sent: Monday, January 8, 2018 10:17:14 AM
To: akshay naidu
Cc: Saisai Shao; Raj Adyanthaya; spark users
Subject: Re: Is Apache Spark-2.2.1 compatible with Hadoop-3.0.0

My current best guess is that Spark does not fully support Hadoop 3.x because 
https://issues.apache.org/jira/browse/SPARK-18673 (updates to Hive shims for 
Hadoop 3.x) has not been resolved. There are also likely to be transitive 
dependency conflicts which will need to be resolved.

On Mon, Jan 8, 2018 at 8:52 AM akshay naidu 
<akshaynaid...@gmail.com<mailto:akshaynaid...@gmail.com>> wrote:
yes , spark download page does mention that 2.2.1 is for 'hadoop-2.7 and 
later', but my confusion is because spark was released on 1st dec and hadoop-3 
stable version released on 13th Dec. And  to my similar question on 
stackoverflow.com<https://stackoverflow.com/questions/47920005/how-is-hadoop-3-0-0-s-compatibility-with-older-versions-of-hive-pig-sqoop-and>
 , Mr. jacek-laskowski<https://stackoverflow.com/users/1305344/jacek-laskowski> 
replied that spark-2.2.1 doesn't support hadoop-3. so I am just looking for 
more clarity on this doubt before moving on to upgrades.

Thanks all for help.

Akshay.

On Mon, Jan 8, 2018 at 8:47 AM, Saisai Shao 
<sai.sai.s...@gmail.com<mailto:sai.sai.s...@gmail.com>> wrote:
AFAIK, there's no large scale test for Hadoop 3.0 in the community. So it is 
not clear whether it is supported or not (or has some issues). I think in the 
download page "Pre-Built for Apache Hadoop 2.7 and later" mostly means that it 
supports Hadoop 2.7+ (2.8...), but not 3.0 (IIUC).

Thanks
Jerry

2018-01-08 4:50 GMT+08:00 Raj Adyanthaya 
<raj...@gmail.com<mailto:raj...@gmail.com>>:
Hi Akshay

On the Spark Download page when you select Spark 2.2.1 it gives you an option 
to select package type. In that, there is an option to select  "Pre-Built for 
Apache Hadoop 2.7 and later". I am assuming it means that it does support 
Hadoop 3.0.

http://spark.apache.org/downloads.html

Thanks,
Raj A.

On Sat, Jan 6, 2018 at 8:23 PM, akshay naidu 
<akshaynaid...@gmail.com<mailto:akshaynaid...@gmail.com>> wrote:
hello Users,
I need to know whether we can run latest spark on  latest hadoop version i.e., 
spark-2.2.1 released on 1st dec and hadoop-3.0.0 released on 13th dec.
thanks.

Re: Is Apache Spark-2.2.1 compatible with Hadoop-3.0.0

[Josh Rosen]

My current best guess is that Spark does *not* fully support Hadoop 3.x
because https://issues.apache.org/jira/browse/SPARK-18673 (updates to Hive
shims for Hadoop 3.x) has not been resolved. There are also likely to be
transitive dependency conflicts which will need to be resolved.

On Mon, Jan 8, 2018 at 8:52 AM akshay naidu  wrote:

> yes , spark download page does mention that 2.2.1 is for 'hadoop-2.7 and
> later', but my confusion is because spark was released on 1st dec and
> hadoop-3 stable version released on 13th Dec. And  to my similar question
> on stackoverflow.com
> 
> , Mr. jacek-laskowski
>  replied that
> spark-2.2.1 doesn't support hadoop-3. so I am just looking for more clarity
> on this doubt before moving on to upgrades.
>
> Thanks all for help.
>
> Akshay.
>
> On Mon, Jan 8, 2018 at 8:47 AM, Saisai Shao 
> wrote:
>
>> AFAIK, there's no large scale test for Hadoop 3.0 in the community. So it
>> is not clear whether it is supported or not (or has some issues). I think
>> in the download page "Pre-Built for Apache Hadoop 2.7 and later" mostly
>> means that it supports Hadoop 2.7+ (2.8...), but not 3.0 (IIUC).
>>
>> Thanks
>> Jerry
>>
>> 2018-01-08 4:50 GMT+08:00 Raj Adyanthaya :
>>
>>> Hi Akshay
>>>
>>> On the Spark Download page when you select Spark 2.2.1 it gives you an
>>> option to select package type. In that, there is an option to select
>>> "Pre-Built for Apache Hadoop 2.7 and later". I am assuming it means that it
>>> does support Hadoop 3.0.
>>>
>>> http://spark.apache.org/downloads.html
>>>
>>> Thanks,
>>> Raj A.
>>>
>>> On Sat, Jan 6, 2018 at 8:23 PM, akshay naidu 
>>> wrote:
>>>
 hello Users,
 I need to know whether we can run latest spark on  latest hadoop
 version i.e., spark-2.2.1 released on 1st dec and hadoop-3.0.0 released on
 13th dec.
 thanks.

>>>
>>>
>>
>

Re: Is Apache Spark-2.2.1 compatible with Hadoop-3.0.0

[akshay naidu]

yes , spark download page does mention that 2.2.1 is for 'hadoop-2.7 and
later', but my confusion is because spark was released on 1st dec and
hadoop-3 stable version released on 13th Dec. And  to my similar question
on stackoverflow.com

, Mr. jacek-laskowski
 replied that
spark-2.2.1 doesn't support hadoop-3. so I am just looking for more clarity
on this doubt before moving on to upgrades.

Thanks all for help.
Akshay.

On Mon, Jan 8, 2018 at 8:47 AM, Saisai Shao  wrote:

> AFAIK, there's no large scale test for Hadoop 3.0 in the community. So it
> is not clear whether it is supported or not (or has some issues). I think
> in the download page "Pre-Built for Apache Hadoop 2.7 and later" mostly
> means that it supports Hadoop 2.7+ (2.8...), but not 3.0 (IIUC).
>
> Thanks
> Jerry
>
> 2018-01-08 4:50 GMT+08:00 Raj Adyanthaya :
>
>> Hi Akshay
>>
>> On the Spark Download page when you select Spark 2.2.1 it gives you an
>> option to select package type. In that, there is an option to select
>> "Pre-Built for Apache Hadoop 2.7 and later". I am assuming it means that it
>> does support Hadoop 3.0.
>>
>> http://spark.apache.org/downloads.html
>>
>> Thanks,
>> Raj A.
>>
>> On Sat, Jan 6, 2018 at 8:23 PM, akshay naidu 
>> wrote:
>>
>>> hello Users,
>>> I need to know whether we can run latest spark on  latest hadoop version
>>> i.e., spark-2.2.1 released on 1st dec and hadoop-3.0.0 released on 13th dec.
>>> thanks.
>>>
>>
>>
>

Re: Is Apache Spark-2.2.1 compatible with Hadoop-3.0.0

[Saisai Shao]

AFAIK, there's no large scale test for Hadoop 3.0 in the community. So it
is not clear whether it is supported or not (or has some issues). I think
in the download page "Pre-Built for Apache Hadoop 2.7 and later" mostly
means that it supports Hadoop 2.7+ (2.8...), but not 3.0 (IIUC).

Thanks
Jerry

2018-01-08 4:50 GMT+08:00 Raj Adyanthaya :

> Hi Akshay
>
> On the Spark Download page when you select Spark 2.2.1 it gives you an
> option to select package type. In that, there is an option to select
> "Pre-Built for Apache Hadoop 2.7 and later". I am assuming it means that it
> does support Hadoop 3.0.
>
> http://spark.apache.org/downloads.html
>
> Thanks,
> Raj A.
>
> On Sat, Jan 6, 2018 at 8:23 PM, akshay naidu 
> wrote:
>
>> hello Users,
>> I need to know whether we can run latest spark on  latest hadoop version
>> i.e., spark-2.2.1 released on 1st dec and hadoop-3.0.0 released on 13th dec.
>> thanks.
>>
>
>

Re: Is Apache Spark-2.2.1 compatible with Hadoop-3.0.0

[Raj Adyanthaya]

Hi Akshay

On the Spark Download page when you select Spark 2.2.1 it gives you an
option to select package type. In that, there is an option to select
"Pre-Built for Apache Hadoop 2.7 and later". I am assuming it means that it
does support Hadoop 3.0.

http://spark.apache.org/downloads.html

Thanks,
Raj A.

On Sat, Jan 6, 2018 at 8:23 PM, akshay naidu 
wrote:

> hello Users,
> I need to know whether we can run latest spark on  latest hadoop version
> i.e., spark-2.2.1 released on 1st dec and hadoop-3.0.0 released on 13th dec.
> thanks.
>

Is Apache Spark-2.2.1 compatible with Hadoop-3.0.0

[akshay naidu]

hello Users,
I need to know whether we can run latest spark on  latest hadoop version
i.e., spark-2.2.1 released on 1st dec and hadoop-3.0.0 released on 13th dec.
thanks.