Re: HDP 2.3 support for Spark 1.5.x

[Krishna Sankar]

Thanks Guys. Yep, now I would install 1.5.1 over HDP 2.3, if that works.
Cheers


On Mon, Sep 28, 2015 at 9:47 AM, Ted Yu  wrote:

> Krishna:
> If you want to query ORC files, see the following JIRA:
>
> [SPARK-10623] [SQL] Fixes ORC predicate push-down
>
> which is in the 1.5.1. release.
>
> FYI
>
> On Mon, Sep 28, 2015 at 9:42 AM, Fabien Martin 
> wrote:
>
>> Hi Krishna,
>>
>>- Take a lokk at 
>> *http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
>>
>> *
>>- Or you can specify your 1.5.x jar as the Spark one using something
>>like :
>>
>> --conf
>> "spark.yarn.jar=hdfs://master:8020/spark-assembly-1.5.0-hadoop2.6.0.jar"
>>
>> The main drawback is :
>>
>> *Known Issues*
>>
>> *Spark YARN ATS integration does not work in this tech preview. You will
>> not see the history of Spark jobs in the Jobs server after a job is
>> finished.*
>>
>> 2015-09-23 1:31 GMT+02:00 Zhan Zhang :
>>
>>> Hi Krishna,
>>>
>>> For the time being, you can download from upstream, and it should be
>>> running OK for HDP2.3.  For hdp specific problem, you can ask in
>>> Hortonworks forum.
>>>
>>> Thanks.
>>>
>>> Zhan Zhang
>>>
>>> On Sep 22, 2015, at 3:42 PM, Krishna Sankar  wrote:
>>>
>>> Guys,
>>>
>>>- We have HDP 2.3 installed just now. It comes with Spark 1.3.x. The
>>>current wisdom is that it will support the 1.4.x train (which is good, 
>>> need
>>>DataFrame et al).
>>>- What is the plan to support Spark 1.5.x ? Can we install 1.5.0 on
>>>HDP 2.3 ? Or will Spark 1.5.x support be in HDP 2.3.x and if so ~when ?
>>>
>>> Cheers & Thanks
>>> 
>>>
>>>
>>>
>>
>

Re: HDP 2.3 support for Spark 1.5.x

[Zhan Zhang]

Hi Krishna,

For the time being, you can download from upstream, and it should be running OK 
for HDP2.3.  For hdp specific problem, you can ask in Hortonworks forum.

Thanks.

Zhan Zhang

On Sep 22, 2015, at 3:42 PM, Krishna Sankar 
> wrote:

Guys,

  *   We have HDP 2.3 installed just now. It comes with Spark 1.3.x. The 
current wisdom is that it will support the 1.4.x train (which is good, need 
DataFrame et al).
  *   What is the plan to support Spark 1.5.x ? Can we install 1.5.0 on HDP 2.3 
? Or will Spark 1.5.x support be in HDP 2.3.x and if so ~when ?

Cheers & Thanks