Re: [discuss] ending support for Java 7 in Spark 2.0

[Reynold Xin]

Hi Sean,

See http://www.oracle.com/technetwork/java/eol-135779.html

Java 7 hasn't EOLed yet. If you look at support you can get from Oracle,
it's actually goes to 2019. And you can even get more support after that.

Spark has always maintained great backward compatibility with other
systems, way beyond what vendors typically support. For example, we
supported Hadoop 1.x all the way until Spark 1.6 (basically the last
release), while all the vendors have dropped support for them already.

Putting my Databricks hat on we actually only support Java 8, but I think
it would be great to still support Java 7 in the upstream release for some
larger deployments. I like the idea of deprecating or at least strongly
encouraging people to update.

On Tuesday, April 5, 2016, Sean Owen  wrote:

> Following
> https://github.com/apache/spark/pull/12165#issuecomment-205791222
> I'd like to make a point about process and then answer points below.
>
> We have this funny system where anyone can propose a change, and any
> of a few people can veto a change unilaterally. The latter rarely
> comes up. 9 changes out of 10 nobody disagrees on; sometimes a
> committer will say 'no' to a change and nobody else with that bit
> disagrees.
>
> Sometimes it matters and here I see, what, 4 out of 5 people including
> committers supporting a particular change. A veto to oppose that is
> pretty drastic. It's not something to use because you or customers
> prefer a certain outcome. This reads like you're informing people
> you've changed your mind and that's the decision, when it can't work
> that way. I saw this happen to a lesser extent in the thread about
> Scala 2.10.
>
> It doesn't mean majority rules here either, but can I suggest you
> instead counter-propose an outcome that the people here voting in
> favor of what you're vetoing would probably also buy into? I bet
> everyone's willing to give wide accommodation to your concerns. It's
> probably not hard, like: let's plan to not support Java 7 in Spark
> 2.1.0. (Then we can debate the logic of that.)
>
> On Mon, Apr 4, 2016 at 6:28 AM, Reynold Xin  > wrote:
> > some technology companies, are still using Java 7. One thing is that up
> > until this date, users still can't install openjdk 8 on Ubuntu by
> default. I
> > see that as an indication that it is too early to drop Java 7.
>
> I have Java 8 on my Ubuntu instance, and installed it directly via apt-get.
> http://openjdk.java.net/install/
>
>
> > Looking at the timeline, JDK release a major new version roughly every 3
> > years. We dropped Java 6 support one year ago, so from a timeline point
> of
> > view we would be very aggressive here if we were to drop Java 7 support
> in
> > Spark 2.0.
>
> The metric is really (IMHO) when the JDK goes EOL. Java 6 was EOL in
> Feb 2013, so supporting it into Spark 1.x was probably too long. Java
> 7 was EOL in April 2015. It's not really somehow every ~3 years.
>
>
> > Note that not dropping Java 7 support now doesn't mean we have to support
> > Java 7 throughout Spark 2.x. We dropped Java 6 support in Spark 1.5, even
> > though Spark 1.0 started with Java 6.
>
> Whatever arguments one has about preventing people from updating to
> the latest and greatest then apply to a *minor* release, which is
> worse. Java 6 support was probably overdue for removal at 1.0;
> better-late-than-never, not necessarily the right time to do it.
>
>
> > In terms of testing, Josh has actually improved our test infra so now we
> > would run the Java 8 tests: https://github.com/apache/spark/pull/12073
>
> Excellent, but, orthogonal.
>
> Even if I personally don't see the merit in these arguments compared
> to the counter-arguments, retaining Java 7 support now wouldn't be a
> terrible outcome. I'd like to see better process and a more reasonable
> compromise result though.
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Sean Owen]

Following https://github.com/apache/spark/pull/12165#issuecomment-205791222
I'd like to make a point about process and then answer points below.

We have this funny system where anyone can propose a change, and any
of a few people can veto a change unilaterally. The latter rarely
comes up. 9 changes out of 10 nobody disagrees on; sometimes a
committer will say 'no' to a change and nobody else with that bit
disagrees.

Sometimes it matters and here I see, what, 4 out of 5 people including
committers supporting a particular change. A veto to oppose that is
pretty drastic. It's not something to use because you or customers
prefer a certain outcome. This reads like you're informing people
you've changed your mind and that's the decision, when it can't work
that way. I saw this happen to a lesser extent in the thread about
Scala 2.10.

It doesn't mean majority rules here either, but can I suggest you
instead counter-propose an outcome that the people here voting in
favor of what you're vetoing would probably also buy into? I bet
everyone's willing to give wide accommodation to your concerns. It's
probably not hard, like: let's plan to not support Java 7 in Spark
2.1.0. (Then we can debate the logic of that.)

On Mon, Apr 4, 2016 at 6:28 AM, Reynold Xin  wrote:
> some technology companies, are still using Java 7. One thing is that up
> until this date, users still can't install openjdk 8 on Ubuntu by default. I
> see that as an indication that it is too early to drop Java 7.

I have Java 8 on my Ubuntu instance, and installed it directly via apt-get.
http://openjdk.java.net/install/


> Looking at the timeline, JDK release a major new version roughly every 3
> years. We dropped Java 6 support one year ago, so from a timeline point of
> view we would be very aggressive here if we were to drop Java 7 support in
> Spark 2.0.

The metric is really (IMHO) when the JDK goes EOL. Java 6 was EOL in
Feb 2013, so supporting it into Spark 1.x was probably too long. Java
7 was EOL in April 2015. It's not really somehow every ~3 years.


> Note that not dropping Java 7 support now doesn't mean we have to support
> Java 7 throughout Spark 2.x. We dropped Java 6 support in Spark 1.5, even
> though Spark 1.0 started with Java 6.

Whatever arguments one has about preventing people from updating to
the latest and greatest then apply to a *minor* release, which is
worse. Java 6 support was probably overdue for removal at 1.0;
better-late-than-never, not necessarily the right time to do it.


> In terms of testing, Josh has actually improved our test infra so now we
> would run the Java 8 tests: https://github.com/apache/spark/pull/12073

Excellent, but, orthogonal.

Even if I personally don't see the merit in these arguments compared
to the counter-arguments, retaining Java 7 support now wouldn't be a
terrible outcome. I'd like to see better process and a more reasonable
compromise result though.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Karlis Zigurs]

Curveball: Is there a need to use lambdas quite yet?

On Mon, Apr 4, 2016 at 10:58 PM, Ofir Manor  wrote:
> I think that a backup plan could be to announce that JDK7 is deprecated in
> Spark 2.0 and support for it will be fully removed in Spark 2.1. This gives
> admins enough warning to install JDK8 along side their "main" JDK (or fully
> migrate to it), while allowing the project to merge JDK8-specific changes to
> trunk right after the 2.0 release.
>
> However, I personally think it is better to drop JDK7 now. I'm sure that
> both the community and the distributors (Databricks, Cloudera, Hortonworks,
> MapR, IBM etc) will all rush to help their customers migrate their
> environment to support Spark 2.0, so I think any backlash won't be dramatic
> or lasting.
>
> Just my two cents,
>
> Ofir Manor
>
> Co-Founder & CTO | Equalum
>
> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>
>
> On Mon, Apr 4, 2016 at 6:48 PM, Luciano Resende 
> wrote:
>>
>> Reynold,
>>
>> Considering the performance improvements you mentioned in your original
>> e-mail and also considering that few other big data projects have already or
>> are in progress of abandoning JDK 7, I think it would benefit Spark if we go
>> with JDK 8.0 only.
>>
>> Are there users that will be less aggressive ? Yes, but those would most
>> likely be in more stable releases like 1.6.x.
>>
>> On Sun, Apr 3, 2016 at 10:28 PM, Reynold Xin  wrote:
>>>
>>> Since my original email, I've talked to a lot more users and looked at
>>> what various environments support. It is true that a lot of enterprises, and
>>> even some technology companies, are still using Java 7. One thing is that up
>>> until this date, users still can't install openjdk 8 on Ubuntu by default. I
>>> see that as an indication that it is too early to drop Java 7.
>>>
>>> Looking at the timeline, JDK release a major new version roughly every 3
>>> years. We dropped Java 6 support one year ago, so from a timeline point of
>>> view we would be very aggressive here if we were to drop Java 7 support in
>>> Spark 2.0.
>>>
>>> Note that not dropping Java 7 support now doesn't mean we have to support
>>> Java 7 throughout Spark 2.x. We dropped Java 6 support in Spark 1.5, even
>>> though Spark 1.0 started with Java 6.
>>>
>>> In terms of testing, Josh has actually improved our test infra so now we
>>> would run the Java 8 tests: https://github.com/apache/spark/pull/12073
>>>
>>>
>>>
>>>
>>> On Thu, Mar 24, 2016 at 8:51 PM, Liwei Lin  wrote:

 Arguments are really convincing; new Dataset API as well as performance

 improvements is exiting, so I'm personally +1 on moving onto Java8.



 However, I'm afraid Tencent is one of "the organizations stuck with
 Java7"

 -- our IT Infra division wouldn't upgrade to Java7 until Java8 is out,
 and

 wouldn't upgrade to Java8 until Java9 is out.


 So:

 (non-binding) +1 on dropping scala 2.10 support

 (non-binding)  -1 on dropping Java 7 support

   * as long as we figure out a practical way to run
 Spark with

 JDK8 on JDK7 clusters, this -1 would then
 definitely be +1


 Thanks !


 On Fri, Mar 25, 2016 at 10:28 AM, Koert Kuipers 
 wrote:
>
> i think that logic is reasonable, but then the same should also apply
> to scala 2.10, which is also unmaintained/unsupported at this point
> (basically has been since march 2015 except for one hotfix due to a 
> license
> incompatibility)
>
> who wants to support scala 2.10 three years after they did the last
> maintenance release?
>
>
> On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan 
> wrote:
>>
>> Removing compatibility (with jdk, etc) can be done with a major
>> release- given that 7 has been EOLed a while back and is now 
>> unsupported, we
>> have to decide if we drop support for it in 2.0 or 3.0 (2+ years from 
>> now).
>>
>> Given the functionality & performance benefits of going to jdk8,
>> future enhancements relevant in 2.x timeframe ( scala, dependencies) 
>> which
>> requires it, and simplicity wrt code, test & support it looks like a good
>> checkpoint to drop jdk7 support.
>>
>> As already mentioned in the thread, existing yarn clusters are
>> unaffected if they want to continue running jdk7 and yet use spark2 
>> (install
>> jdk8 on all nodes and use it via JAVA_HOME, or worst case distribute 
>> jdk8 as
>> archive - suboptimal).
>> I am unsure about mesos (standalone might be easier upgrade I guess
>> ?).
>>
>>
>> Proposal is for 1.6x line to continue to be supported with critical
>> fixes; newer features will require 2.x and so jdk8
>>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Ofir Manor]

I think that a backup plan could be to announce that JDK7 is deprecated in
Spark 2.0 and support for it will be fully removed in Spark 2.1. This gives
admins enough warning to install JDK8 along side their "main" JDK (or fully
migrate to it), while allowing the project to merge JDK8-specific changes
to trunk right after the 2.0 release.

However, I personally think it is better to drop JDK7 now. I'm sure that
both the community and the distributors (Databricks, Cloudera, Hortonworks,
MapR, IBM etc) will all rush to help their customers migrate their
environment to support Spark 2.0, so I think any backlash won't be dramatic
or lasting.

Just my two cents,

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Mon, Apr 4, 2016 at 6:48 PM, Luciano Resende 
wrote:

> Reynold,
>
> Considering the performance improvements you mentioned in your original
> e-mail and also considering that few other big data projects have already
> or are in progress of abandoning JDK 7, I think it would benefit Spark if
> we go with JDK 8.0 only.
>
> Are there users that will be less aggressive ? Yes, but those would most
> likely be in more stable releases like 1.6.x.
>
> On Sun, Apr 3, 2016 at 10:28 PM, Reynold Xin  wrote:
>
>> Since my original email, I've talked to a lot more users and looked at
>> what various environments support. It is true that a lot of enterprises,
>> and even some technology companies, are still using Java 7. One thing is
>> that up until this date, users still can't install openjdk 8 on Ubuntu by
>> default. I see that as an indication that it is too early to drop Java 7.
>>
>> Looking at the timeline, JDK release a major new version roughly every 3
>> years. We dropped Java 6 support one year ago, so from a timeline point of
>> view we would be very aggressive here if we were to drop Java 7 support in
>> Spark 2.0.
>>
>> Note that not dropping Java 7 support now doesn't mean we have to support
>> Java 7 throughout Spark 2.x. We dropped Java 6 support in Spark 1.5, even
>> though Spark 1.0 started with Java 6.
>>
>> In terms of testing, Josh has actually improved our test infra so now we
>> would run the Java 8 tests: https://github.com/apache/spark/pull/12073
>>
>>
>>
>>
>> On Thu, Mar 24, 2016 at 8:51 PM, Liwei Lin  wrote:
>>
>>> Arguments are really convincing; new Dataset API as well as performance
>>>
>>> improvements is exiting, so I'm personally +1 on moving onto Java8.
>>>
>>>
>>>
>>> However, I'm afraid Tencent is one of "the organizations stuck with
>>> Java7"
>>>
>>> -- our IT Infra division wouldn't upgrade to Java7 until Java8 is out,
>>> and
>>>
>>> wouldn't upgrade to Java8 until Java9 is out.
>>>
>>>
>>> So:
>>>
>>> (non-binding) +1 on dropping scala 2.10 support
>>>
>>> (non-binding)  -1 on dropping Java 7 support
>>>
>>>   * as long as we figure out a practical way to run
>>> Spark with
>>>
>>> JDK8 on JDK7 clusters, this -1 would then
>>> definitely be +1
>>>
>>>
>>> Thanks !
>>>
>>> On Fri, Mar 25, 2016 at 10:28 AM, Koert Kuipers 
>>> wrote:
>>>
 i think that logic is reasonable, but then the same should also apply
 to scala 2.10, which is also unmaintained/unsupported at this point
 (basically has been since march 2015 except for one hotfix due to a license
 incompatibility)

 who wants to support scala 2.10 three years after they did the last
 maintenance release?


 On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan 
 wrote:

> Removing compatibility (with jdk, etc) can be done with a major
> release- given that 7 has been EOLed a while back and is now unsupported,
> we have to decide if we drop support for it in 2.0 or 3.0 (2+ years from
> now).
>
> Given the functionality & performance benefits of going to jdk8,
> future enhancements relevant in 2.x timeframe ( scala, dependencies) which
> requires it, and simplicity wrt code, test & support it looks like a good
> checkpoint to drop jdk7 support.
>
> As already mentioned in the thread, existing yarn clusters are
> unaffected if they want to continue running jdk7 and yet use
> spark2 (install jdk8 on all nodes and use it via JAVA_HOME, or worst case
> distribute jdk8 as archive - suboptimal).
> I am unsure about mesos (standalone might be easier upgrade I guess ?).
>
>
> Proposal is for 1.6x line to continue to be supported with critical
> fixes; newer features will require 2.x and so jdk8
>
> Regards
> Mridul
>
>
> On Thursday, March 24, 2016, Marcelo Vanzin 
> wrote:
>
>> On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin 
>> wrote:
>> > If you want to go down that route, you should also ask somebody who
>> has had
>> > experience 

Re: [discuss] ending support for Java 7 in Spark 2.0

[Luciano Resende]

Reynold,

Considering the performance improvements you mentioned in your original
e-mail and also considering that few other big data projects have already
or are in progress of abandoning JDK 7, I think it would benefit Spark if
we go with JDK 8.0 only.

Are there users that will be less aggressive ? Yes, but those would most
likely be in more stable releases like 1.6.x.

On Sun, Apr 3, 2016 at 10:28 PM, Reynold Xin  wrote:

> Since my original email, I've talked to a lot more users and looked at
> what various environments support. It is true that a lot of enterprises,
> and even some technology companies, are still using Java 7. One thing is
> that up until this date, users still can't install openjdk 8 on Ubuntu by
> default. I see that as an indication that it is too early to drop Java 7.
>
> Looking at the timeline, JDK release a major new version roughly every 3
> years. We dropped Java 6 support one year ago, so from a timeline point of
> view we would be very aggressive here if we were to drop Java 7 support in
> Spark 2.0.
>
> Note that not dropping Java 7 support now doesn't mean we have to support
> Java 7 throughout Spark 2.x. We dropped Java 6 support in Spark 1.5, even
> though Spark 1.0 started with Java 6.
>
> In terms of testing, Josh has actually improved our test infra so now we
> would run the Java 8 tests: https://github.com/apache/spark/pull/12073
>
>
>
>
> On Thu, Mar 24, 2016 at 8:51 PM, Liwei Lin  wrote:
>
>> Arguments are really convincing; new Dataset API as well as performance
>>
>> improvements is exiting, so I'm personally +1 on moving onto Java8.
>>
>>
>>
>> However, I'm afraid Tencent is one of "the organizations stuck with
>> Java7"
>>
>> -- our IT Infra division wouldn't upgrade to Java7 until Java8 is out, and
>>
>> wouldn't upgrade to Java8 until Java9 is out.
>>
>>
>> So:
>>
>> (non-binding) +1 on dropping scala 2.10 support
>>
>> (non-binding)  -1 on dropping Java 7 support
>>
>>   * as long as we figure out a practical way to run
>> Spark with
>>
>> JDK8 on JDK7 clusters, this -1 would then
>> definitely be +1
>>
>>
>> Thanks !
>>
>> On Fri, Mar 25, 2016 at 10:28 AM, Koert Kuipers 
>> wrote:
>>
>>> i think that logic is reasonable, but then the same should also apply to
>>> scala 2.10, which is also unmaintained/unsupported at this point (basically
>>> has been since march 2015 except for one hotfix due to a license
>>> incompatibility)
>>>
>>> who wants to support scala 2.10 three years after they did the last
>>> maintenance release?
>>>
>>>
>>> On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan 
>>> wrote:
>>>
 Removing compatibility (with jdk, etc) can be done with a major
 release- given that 7 has been EOLed a while back and is now unsupported,
 we have to decide if we drop support for it in 2.0 or 3.0 (2+ years from
 now).

 Given the functionality & performance benefits of going to jdk8, future
 enhancements relevant in 2.x timeframe ( scala, dependencies) which
 requires it, and simplicity wrt code, test & support it looks like a good
 checkpoint to drop jdk7 support.

 As already mentioned in the thread, existing yarn clusters are
 unaffected if they want to continue running jdk7 and yet use
 spark2 (install jdk8 on all nodes and use it via JAVA_HOME, or worst case
 distribute jdk8 as archive - suboptimal).
 I am unsure about mesos (standalone might be easier upgrade I guess ?).


 Proposal is for 1.6x line to continue to be supported with critical
 fixes; newer features will require 2.x and so jdk8

 Regards
 Mridul


 On Thursday, March 24, 2016, Marcelo Vanzin 
 wrote:

> On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin 
> wrote:
> > If you want to go down that route, you should also ask somebody who
> has had
> > experience managing a large organization's applications and try to
> update
> > Scala version.
>
> I understand both sides. But if you look at what I've been asking
> since the beginning, it's all about the cost and benefits of dropping
> support for java 1.7.
>
> The biggest argument in your original e-mail is about testing. And the
> testing cost is much bigger for supporting scala 2.10 than it is for
> supporting java 1.7. If you read one of my earlier replies, it should
> be even possible to just do everything in a single job - compile for
> java 7 and still be able to test things in 1.8, including lambdas,
> which seems to be the main thing you were worried about.
>
>
> > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin 
> wrote:
> >>
> >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin 
> wrote:
> >> > Actually it's *way* harder to 

Re: [discuss] ending support for Java 7 in Spark 2.0

[Xuefeng Wu]

Many open source projects are aggressive, such as Oracle JDK and Ubuntu, But 
they provide stable commercial supporting.


In other words, the enterprises doesn't drop JDK7, might aslo do not drop Spark 
1.x to adopt Spark 2.x early version.





On Sun, Apr 3, 2016 at 10:29 PM -0700, "Reynold Xin"  
wrote:










Since my original email, I've talked to a lot more users and looked at what 
various environments support. It is true that a lot of enterprises, and even 
some technology companies, are still using Java 7. One thing is that up until 
this date, users still can't install openjdk 8 on Ubuntu by default. I see that 
as an indication that it is too early to drop Java 7.

Looking at the timeline, JDK release a major new version roughly every 3 years. 
We dropped Java 6 support one year ago, so from a timeline point of view we 
would be very aggressive here if we were to drop Java 7 support in Spark 2.0.
Note that not dropping Java 7 support now doesn't mean we have to support Java 
7 throughout Spark 2.x. We dropped Java 6 support in Spark 1.5, even though 
Spark 1.0 started with Java 6.
In terms of testing, Josh has actually improved our test infra so now we would 
run the Java 8 tests: https://github.com/apache/spark/pull/12073



On Thu, Mar 24, 2016 at 8:51 PM, Liwei Lin  wrote:


Arguments are really convincing; new Dataset API as well as performance


improvements is exiting, so I'm personally +1 on moving onto Java8.

 

However, I'm afraid Tencent is one of "the organizations stuck with Java7"

-- our IT Infra division wouldn't upgrade to Java7 until Java8 is out, and

wouldn't upgrade to Java8 until Java9 is out.




So: 

(non-binding) +1 on dropping scala 2.10 support

(non-binding)  -1 on dropping Java 7 support

                      * as long as we figure out a practical way to run Spark 
with

                        JDK8 on JDK7 clusters, this -1 would then definitely be 
+1




Thanks !
On Fri, Mar 25, 2016 at 10:28 AM, Koert Kuipers  wrote:
i think that logic is reasonable, but then the same should also apply to scala 
2.10, which is also unmaintained/unsupported at this point (basically has been 
since march 2015 except for one hotfix due to a license incompatibility)

who wants to support scala 2.10 three years after they did the last maintenance 
release?


On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan  wrote:
Removing compatibility (with jdk, etc) can be done with a major release- given 
that 7 has been EOLed a while back and is now unsupported, we have to decide if 
we drop support for it in 2.0 or 3.0 (2+ years from now).
Given the functionality & performance benefits of going to jdk8, future 
enhancements relevant in 2.x timeframe ( scala, dependencies) which requires 
it, and simplicity wrt code, test & support it looks like a good checkpoint to 
drop jdk7 support.
As already mentioned in the thread, existing yarn clusters are unaffected if 
they want to continue running jdk7 and yet use spark2 (install jdk8 on all 
nodes and use it via JAVA_HOME, or worst case distribute jdk8 as archive - 
suboptimal).I am unsure about mesos (standalone might be easier upgrade I guess 
?).

Proposal is for 1.6x line to continue to be supported with critical fixes; 
newer features will require 2.x and so jdk8
Regards Mridul 

On Thursday, March 24, 2016, Marcelo Vanzin  wrote:
On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin  wrote:

> If you want to go down that route, you should also ask somebody who has had

> experience managing a large organization's applications and try to update

> Scala version.



I understand both sides. But if you look at what I've been asking

since the beginning, it's all about the cost and benefits of dropping

support for java 1.7.



The biggest argument in your original e-mail is about testing. And the

testing cost is much bigger for supporting scala 2.10 than it is for

supporting java 1.7. If you read one of my earlier replies, it should

be even possible to just do everything in a single job - compile for

java 7 and still be able to test things in 1.8, including lambdas,

which seems to be the main thing you were worried about.





> On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin  wrote:

>>

>> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin  wrote:

>> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than

>> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are

>> > not

>> > binary compatible, whereas JVM 7 and 8 are binary compatible except

>> > certain

>> > esoteric cases.

>>

>> True, but ask anyone who manages a large cluster how long it would

>> take them to upgrade the jdk across their cluster and validate all

>> their applications and everything... binary compatibility is a tiny

>> drop in that bucket.

>>

>> --

>> Marcelo

>

>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Reynold Xin]

Since my original email, I've talked to a lot more users and looked at what
various environments support. It is true that a lot of enterprises, and
even some technology companies, are still using Java 7. One thing is that
up until this date, users still can't install openjdk 8 on Ubuntu by
default. I see that as an indication that it is too early to drop Java 7.

Looking at the timeline, JDK release a major new version roughly every 3
years. We dropped Java 6 support one year ago, so from a timeline point of
view we would be very aggressive here if we were to drop Java 7 support in
Spark 2.0.

Note that not dropping Java 7 support now doesn't mean we have to support
Java 7 throughout Spark 2.x. We dropped Java 6 support in Spark 1.5, even
though Spark 1.0 started with Java 6.

In terms of testing, Josh has actually improved our test infra so now we
would run the Java 8 tests: https://github.com/apache/spark/pull/12073




On Thu, Mar 24, 2016 at 8:51 PM, Liwei Lin  wrote:

> Arguments are really convincing; new Dataset API as well as performance
>
> improvements is exiting, so I'm personally +1 on moving onto Java8.
>
>
>
> However, I'm afraid Tencent is one of "the organizations stuck with Java7"
>
> -- our IT Infra division wouldn't upgrade to Java7 until Java8 is out, and
>
> wouldn't upgrade to Java8 until Java9 is out.
>
>
> So:
>
> (non-binding) +1 on dropping scala 2.10 support
>
> (non-binding)  -1 on dropping Java 7 support
>
>   * as long as we figure out a practical way to run
> Spark with
>
> JDK8 on JDK7 clusters, this -1 would then
> definitely be +1
>
>
> Thanks !
>
> On Fri, Mar 25, 2016 at 10:28 AM, Koert Kuipers  wrote:
>
>> i think that logic is reasonable, but then the same should also apply to
>> scala 2.10, which is also unmaintained/unsupported at this point (basically
>> has been since march 2015 except for one hotfix due to a license
>> incompatibility)
>>
>> who wants to support scala 2.10 three years after they did the last
>> maintenance release?
>>
>>
>> On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan 
>> wrote:
>>
>>> Removing compatibility (with jdk, etc) can be done with a major release-
>>> given that 7 has been EOLed a while back and is now unsupported, we have to
>>> decide if we drop support for it in 2.0 or 3.0 (2+ years from now).
>>>
>>> Given the functionality & performance benefits of going to jdk8, future
>>> enhancements relevant in 2.x timeframe ( scala, dependencies) which
>>> requires it, and simplicity wrt code, test & support it looks like a good
>>> checkpoint to drop jdk7 support.
>>>
>>> As already mentioned in the thread, existing yarn clusters are
>>> unaffected if they want to continue running jdk7 and yet use
>>> spark2 (install jdk8 on all nodes and use it via JAVA_HOME, or worst case
>>> distribute jdk8 as archive - suboptimal).
>>> I am unsure about mesos (standalone might be easier upgrade I guess ?).
>>>
>>>
>>> Proposal is for 1.6x line to continue to be supported with critical
>>> fixes; newer features will require 2.x and so jdk8
>>>
>>> Regards
>>> Mridul
>>>
>>>
>>> On Thursday, March 24, 2016, Marcelo Vanzin  wrote:
>>>
 On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin 
 wrote:
 > If you want to go down that route, you should also ask somebody who
 has had
 > experience managing a large organization's applications and try to
 update
 > Scala version.

 I understand both sides. But if you look at what I've been asking
 since the beginning, it's all about the cost and benefits of dropping
 support for java 1.7.

 The biggest argument in your original e-mail is about testing. And the
 testing cost is much bigger for supporting scala 2.10 than it is for
 supporting java 1.7. If you read one of my earlier replies, it should
 be even possible to just do everything in a single job - compile for
 java 7 and still be able to test things in 1.8, including lambdas,
 which seems to be the main thing you were worried about.


 > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin 
 wrote:
 >>
 >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin 
 wrote:
 >> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
 >> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11
 are
 >> > not
 >> > binary compatible, whereas JVM 7 and 8 are binary compatible except
 >> > certain
 >> > esoteric cases.
 >>
 >> True, but ask anyone who manages a large cluster how long it would
 >> take them to upgrade the jdk across their cluster and validate all
 >> their applications and everything... binary compatibility is a tiny
 >> drop in that bucket.
 >>
 >> --
 >> Marcelo
 >
 >



 --
 

RE: [discuss] ending support for Java 7 in Spark 2.0

[Raymond Honderdors]

Maybe the question should be how far back should spark be compatible?


There is nothings stopping people to run spark 1.6.x with jdk 7 or scala 2.10 
or Hadoop <2.6
But if they want spark 2.x they should consider a migration to jdk8 and scala 
2.11

Or am I getting it all wrong?


Raymond Honderdors
Team Lead Analytics BI
Business Intelligence Developer
raymond.honderd...@sizmek.com<mailto:raymond.honderd...@sizmek.com>
T +972.7325.3569
Herzliya

From: Tom Graves [mailto:tgraves...@yahoo.com.INVALID]
Sent: Wednesday, March 30, 2016 4:46 PM
To: Steve Loughran <ste...@hortonworks.com>
Cc: Reynold Xin <r...@databricks.com>; Koert Kuipers <ko...@tresata.com>; 
Kostas Sakellis <kos...@cloudera.com>; Marcelo Vanzin <van...@cloudera.com>; 
dev@spark.apache.org
Subject: Re: [discuss] ending support for Java 7 in Spark 2.0

Steve, those are good points, I had forgotten Hadoop had those issues.We 
run with jdk 8, hadoop is built for jdk7 compatibility, we are running hadoop 
2.7 on our clusters and by the time Spark 2.0 is out I would expected a mix of 
Hadoop 2.7 and 2.8.  We also don't use spnego.

I didn't quite follow what you were saying with the hadoop services being on 
jdk7.  Are you saying building spark with say hadoop 2.8 libraries but your 
hadoop cluster is running hadoop 2.6 or less? If so I would agree that isn't a 
good idea.

Personally and from Yahoo point I'm still fine with going to jdk8 but I could 
see where other people are on older versions of Hadoop where it might be a 
problem.

Tom

On Wednesday, March 30, 2016 5:42 AM, Steve Loughran 
<ste...@hortonworks.com<mailto:ste...@hortonworks.com>> wrote:


Can I note that if Spark 2.0 is going to be Java 8+ only, then that means 
Hadoop 2.6.x should be the minimum Hadoop version.

https://issues.apache.org/jira/browse/HADOOP-11090

Where things get complicated, is that situation of: Hadoop services on Java 7, 
Spark on Java 8 in its own JVM

I'm not sure that you could get away with having the newer version of the 
Hadoop classes in the spark assembly/lib dir, without coming up against 
incompatibilities with the Hadoop JNI libraries. These are currently backwards 
compatible, but trying to link up Hadoop 2.7 against a Hadoop 2.6 hadoop lib 
will generate an UnsatisfiedLinkException. Meaning: the whole cluster's hadoop 
libs have to be in sync, or at least the main cluster release in a version of 
hadoop 2.x >= the spark bundled edition.

Ignoring that detail,

Hadoop 2.6.1+
Guava >= 15? 17?

 I think the outcome of Hadoop < 2.6 and JDK >= 8 is "undefined"; all bug 
reports will be met with a "please upgrade, re-open if the problem is still 
there".

Kerberos is  a particular troublespot here : You need Hadoop 2.6.1+ for 
Kerberos to work in Java 8 and recent versions of Java 7 (HADOOP-10786)

Note also that HADOOP-11628 is in 2.8 only. SPNEGO + CNAMES. I'll see about 
pulling that into 2.7.x, though I'm reluctant to go near 2.6 just to keep that 
extra stable.


Thomas: you've got the big clusters, what versions of Hadoop will they be on by 
the time you look at Spark 2.0?

-Steve

Re: [discuss] ending support for Java 7 in Spark 2.0

[Tom Graves]

Steve, those are good points, I had forgotten Hadoop had those issues.    We 
run with jdk 8, hadoop is built for jdk7 compatibility, we are running hadoop 
2.7 on our clusters and by the time Spark 2.0 is out I would expected a mix of 
Hadoop 2.7 and 2.8.  We also don't use spnego.
I didn't quite follow what you were saying with the hadoop services being on 
jdk7.  Are you saying building spark with say hadoop 2.8 libraries but your 
hadoop cluster is running hadoop 2.6 or less? If so I would agree that isn't a 
good idea.
Personally and from Yahoo point I'm still fine with going to jdk8 but I could 
see where other people are on older versions of Hadoop where it might be a 
problem.
Tom 

On Wednesday, March 30, 2016 5:42 AM, Steve Loughran 
 wrote:
 

 
Can I note that if Spark 2.0 is going to be Java 8+ only, then that means 
Hadoop 2.6.x should be the minimum Hadoop version.
https://issues.apache.org/jira/browse/HADOOP-11090
Where things get complicated, is that situation of: Hadoop services on Java 7, 
Spark on Java 8 in its own JVM
I'm not sure that you could get away with having the newer version of the 
Hadoop classes in the spark assembly/lib dir, without coming up against 
incompatibilities with the Hadoop JNI libraries. These are currently backwards 
compatible, but trying to link up Hadoop 2.7 against a Hadoop 2.6 hadoop lib 
will generate an UnsatisfiedLinkException. Meaning: the whole cluster's hadoop 
libs have to be in sync, or at least the main cluster release in a version of 
hadoop 2.x >= the spark bundled edition.
Ignoring that detail, 
Hadoop 2.6.1+Guava >= 15? 17?
 I think the outcome of Hadoop < 2.6 and JDK >= 8 is "undefined"; all bug 
reports will be met with a "please upgrade, re-open if the problem is still 
there". 
Kerberos is  a particular troublespot here : You need Hadoop 2.6.1+ for 
Kerberos to work in Java 8 and recent versions of Java 7 (HADOOP-10786)
Note also that HADOOP-11628 is in 2.8 only. SPNEGO + CNAMES. I'll see about 
pulling that into 2.7.x, though I'm reluctant to go near 2.6 just to keep that 
extra stable.

Thomas: you've got the big clusters, what versions of Hadoop will they be on by 
the time you look at Spark 2.0?
-Steve


 

  

Re: [discuss] ending support for Java 7 in Spark 2.0

[Steve Loughran]

Can I note that if Spark 2.0 is going to be Java 8+ only, then that means 
Hadoop 2.6.x should be the minimum Hadoop version.

https://issues.apache.org/jira/browse/HADOOP-11090

Where things get complicated, is that situation of: Hadoop services on Java 7, 
Spark on Java 8 in its own JVM

I'm not sure that you could get away with having the newer version of the 
Hadoop classes in the spark assembly/lib dir, without coming up against 
incompatibilities with the Hadoop JNI libraries. These are currently backwards 
compatible, but trying to link up Hadoop 2.7 against a Hadoop 2.6 hadoop lib 
will generate an UnsatisfiedLinkException. Meaning: the whole cluster's hadoop 
libs have to be in sync, or at least the main cluster release in a version of 
hadoop 2.x >= the spark bundled edition.

Ignoring that detail,

Hadoop 2.6.1+
Guava >= 15? 17?

 I think the outcome of Hadoop < 2.6 and JDK >= 8 is "undefined"; all bug 
reports will be met with a "please upgrade, re-open if the problem is still 
there".

Kerberos is  a particular troublespot here : You need Hadoop 2.6.1+ for 
Kerberos to work in Java 8 and recent versions of Java 7 (HADOOP-10786)

Note also that HADOOP-11628 is in 2.8 only. SPNEGO + CNAMES. I'll see about 
pulling that into 2.7.x, though I'm reluctant to go near 2.6 just to keep that 
extra stable.


Thomas: you've got the big clusters, what versions of Hadoop will they be on by 
the time you look at Spark 2.0?

-Steve

Re: [discuss] ending support for Java 7 in Spark 2.0

[Tom Graves]

+1.
Tom 

On Tuesday, March 29, 2016 1:17 PM, Reynold Xin  wrote:
 

 They work.

On Tue, Mar 29, 2016 at 10:01 AM, Koert Kuipers  wrote:

if scala prior to sbt 2.10.4 didn't support java 8, does that mean that 3rd 
party scala libraries compiled with a scala version < 2.10.4 might not work on 
java 8?


On Mon, Mar 28, 2016 at 7:06 PM, Kostas Sakellis  wrote:

Also, +1 on dropping jdk7 in Spark 2.0. 
Kostas
On Mon, Mar 28, 2016 at 2:01 PM, Marcelo Vanzin  wrote:

Finally got some internal feedback on this, and we're ok with
requiring people to deploy jdk8 for 2.0, so +1 too.

On Mon, Mar 28, 2016 at 1:15 PM, Luciano Resende  wrote:
> +1, I also checked with few projects inside IBM that consume Spark and they
> seem to be ok with the direction of droping JDK 7.
>
> On Mon, Mar 28, 2016 at 11:24 AM, Michael Gummelt 
> wrote:
>>
>> +1 from Mesosphere
>>
>> On Mon, Mar 28, 2016 at 5:12 AM, Steve Loughran 
>> wrote:
>>>
>>>
>>> > On 25 Mar 2016, at 01:59, Mridul Muralidharan  wrote:
>>> >
>>> > Removing compatibility (with jdk, etc) can be done with a major
>>> > release- given that 7 has been EOLed a while back and is now unsupported, 
>>> > we
>>> > have to decide if we drop support for it in 2.0 or 3.0 (2+ years from 
>>> > now).
>>> >
>>> > Given the functionality & performance benefits of going to jdk8, future
>>> > enhancements relevant in 2.x timeframe ( scala, dependencies) which 
>>> > requires
>>> > it, and simplicity wrt code, test & support it looks like a good 
>>> > checkpoint
>>> > to drop jdk7 support.
>>> >
>>> > As already mentioned in the thread, existing yarn clusters are
>>> > unaffected if they want to continue running jdk7 and yet use spark2 
>>> > (install
>>> > jdk8 on all nodes and use it via JAVA_HOME, or worst case distribute jdk8 
>>> > as
>>> > archive - suboptimal).
>>>
>>> you wouldn't want to dist it as an archive; it's not just the binaries,
>>> it's the install phase. And you'd better remember to put the JCE jar in on
>>> top of the JDK for kerberos to work.
>>>
>>> setting up environment vars to point to JDK8 in the launched
>>> app/container avoids that. Yes, the ops team do need to install java, but if
>>> you offer them the choice of "installing a centrally managed Java" and
>>> "having my code try and install it", they should go for the managed option.
>>>
>>> One thing to consider for 2.0 is to make it easier to set up those env
>>> vars for both python and java. And, as the techniques for mixing JDK
>>> versions is clearly not that well known, documenting it.
>>>
>>> (FWIW I've done code which even uploads it's own hadoop-* JAR, but what
>>> gets you is changes in the hadoop-native libs; you do need to get the PATH
>>> var spot on)
>>>
>>>
>>> > I am unsure about mesos (standalone might be easier upgrade I guess ?).
>>> >
>>> >
>>> > Proposal is for 1.6x line to continue to be supported with critical
>>> > fixes; newer features will require 2.x and so jdk8
>>> >
>>> > Regards
>>> > Mridul
>>> >
>>> >
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: dev-h...@spark.apache.org
>>>
>>
>>
>>
>> --
>> Michael Gummelt
>> Software Engineer
>> Mesosphere
>
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/



--
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org









  

Re: [discuss] ending support for Java 7 in Spark 2.0

[Reynold Xin]

They work.


On Tue, Mar 29, 2016 at 10:01 AM, Koert Kuipers  wrote:

> if scala prior to sbt 2.10.4 didn't support java 8, does that mean that
> 3rd party scala libraries compiled with a scala version < 2.10.4 might not
> work on java 8?
>
>
> On Mon, Mar 28, 2016 at 7:06 PM, Kostas Sakellis 
> wrote:
>
>> Also, +1 on dropping jdk7 in Spark 2.0.
>>
>> Kostas
>>
>> On Mon, Mar 28, 2016 at 2:01 PM, Marcelo Vanzin 
>> wrote:
>>
>>> Finally got some internal feedback on this, and we're ok with
>>> requiring people to deploy jdk8 for 2.0, so +1 too.
>>>
>>> On Mon, Mar 28, 2016 at 1:15 PM, Luciano Resende 
>>> wrote:
>>> > +1, I also checked with few projects inside IBM that consume Spark and
>>> they
>>> > seem to be ok with the direction of droping JDK 7.
>>> >
>>> > On Mon, Mar 28, 2016 at 11:24 AM, Michael Gummelt <
>>> mgumm...@mesosphere.io>
>>> > wrote:
>>> >>
>>> >> +1 from Mesosphere
>>> >>
>>> >> On Mon, Mar 28, 2016 at 5:12 AM, Steve Loughran <
>>> ste...@hortonworks.com>
>>> >> wrote:
>>> >>>
>>> >>>
>>> >>> > On 25 Mar 2016, at 01:59, Mridul Muralidharan 
>>> wrote:
>>> >>> >
>>> >>> > Removing compatibility (with jdk, etc) can be done with a major
>>> >>> > release- given that 7 has been EOLed a while back and is now
>>> unsupported, we
>>> >>> > have to decide if we drop support for it in 2.0 or 3.0 (2+ years
>>> from now).
>>> >>> >
>>> >>> > Given the functionality & performance benefits of going to jdk8,
>>> future
>>> >>> > enhancements relevant in 2.x timeframe ( scala, dependencies)
>>> which requires
>>> >>> > it, and simplicity wrt code, test & support it looks like a good
>>> checkpoint
>>> >>> > to drop jdk7 support.
>>> >>> >
>>> >>> > As already mentioned in the thread, existing yarn clusters are
>>> >>> > unaffected if they want to continue running jdk7 and yet use
>>> spark2 (install
>>> >>> > jdk8 on all nodes and use it via JAVA_HOME, or worst case
>>> distribute jdk8 as
>>> >>> > archive - suboptimal).
>>> >>>
>>> >>> you wouldn't want to dist it as an archive; it's not just the
>>> binaries,
>>> >>> it's the install phase. And you'd better remember to put the JCE jar
>>> in on
>>> >>> top of the JDK for kerberos to work.
>>> >>>
>>> >>> setting up environment vars to point to JDK8 in the launched
>>> >>> app/container avoids that. Yes, the ops team do need to install
>>> java, but if
>>> >>> you offer them the choice of "installing a centrally managed Java"
>>> and
>>> >>> "having my code try and install it", they should go for the managed
>>> option.
>>> >>>
>>> >>> One thing to consider for 2.0 is to make it easier to set up those
>>> env
>>> >>> vars for both python and java. And, as the techniques for mixing JDK
>>> >>> versions is clearly not that well known, documenting it.
>>> >>>
>>> >>> (FWIW I've done code which even uploads it's own hadoop-* JAR, but
>>> what
>>> >>> gets you is changes in the hadoop-native libs; you do need to get
>>> the PATH
>>> >>> var spot on)
>>> >>>
>>> >>>
>>> >>> > I am unsure about mesos (standalone might be easier upgrade I
>>> guess ?).
>>> >>> >
>>> >>> >
>>> >>> > Proposal is for 1.6x line to continue to be supported with critical
>>> >>> > fixes; newer features will require 2.x and so jdk8
>>> >>> >
>>> >>> > Regards
>>> >>> > Mridul
>>> >>> >
>>> >>> >
>>> >>>
>>> >>>
>>> >>> -
>>> >>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>>> >>> For additional commands, e-mail: dev-h...@spark.apache.org
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Michael Gummelt
>>> >> Software Engineer
>>> >> Mesosphere
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Luciano Resende
>>> > http://twitter.com/lresende1975
>>> > http://lresende.blogspot.com/
>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: dev-h...@spark.apache.org
>>>
>>>
>>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Koert Kuipers]

if scala prior to sbt 2.10.4 didn't support java 8, does that mean that 3rd
party scala libraries compiled with a scala version < 2.10.4 might not work
on java 8?


On Mon, Mar 28, 2016 at 7:06 PM, Kostas Sakellis 
wrote:

> Also, +1 on dropping jdk7 in Spark 2.0.
>
> Kostas
>
> On Mon, Mar 28, 2016 at 2:01 PM, Marcelo Vanzin 
> wrote:
>
>> Finally got some internal feedback on this, and we're ok with
>> requiring people to deploy jdk8 for 2.0, so +1 too.
>>
>> On Mon, Mar 28, 2016 at 1:15 PM, Luciano Resende 
>> wrote:
>> > +1, I also checked with few projects inside IBM that consume Spark and
>> they
>> > seem to be ok with the direction of droping JDK 7.
>> >
>> > On Mon, Mar 28, 2016 at 11:24 AM, Michael Gummelt <
>> mgumm...@mesosphere.io>
>> > wrote:
>> >>
>> >> +1 from Mesosphere
>> >>
>> >> On Mon, Mar 28, 2016 at 5:12 AM, Steve Loughran <
>> ste...@hortonworks.com>
>> >> wrote:
>> >>>
>> >>>
>> >>> > On 25 Mar 2016, at 01:59, Mridul Muralidharan 
>> wrote:
>> >>> >
>> >>> > Removing compatibility (with jdk, etc) can be done with a major
>> >>> > release- given that 7 has been EOLed a while back and is now
>> unsupported, we
>> >>> > have to decide if we drop support for it in 2.0 or 3.0 (2+ years
>> from now).
>> >>> >
>> >>> > Given the functionality & performance benefits of going to jdk8,
>> future
>> >>> > enhancements relevant in 2.x timeframe ( scala, dependencies) which
>> requires
>> >>> > it, and simplicity wrt code, test & support it looks like a good
>> checkpoint
>> >>> > to drop jdk7 support.
>> >>> >
>> >>> > As already mentioned in the thread, existing yarn clusters are
>> >>> > unaffected if they want to continue running jdk7 and yet use spark2
>> (install
>> >>> > jdk8 on all nodes and use it via JAVA_HOME, or worst case
>> distribute jdk8 as
>> >>> > archive - suboptimal).
>> >>>
>> >>> you wouldn't want to dist it as an archive; it's not just the
>> binaries,
>> >>> it's the install phase. And you'd better remember to put the JCE jar
>> in on
>> >>> top of the JDK for kerberos to work.
>> >>>
>> >>> setting up environment vars to point to JDK8 in the launched
>> >>> app/container avoids that. Yes, the ops team do need to install java,
>> but if
>> >>> you offer them the choice of "installing a centrally managed Java" and
>> >>> "having my code try and install it", they should go for the managed
>> option.
>> >>>
>> >>> One thing to consider for 2.0 is to make it easier to set up those env
>> >>> vars for both python and java. And, as the techniques for mixing JDK
>> >>> versions is clearly not that well known, documenting it.
>> >>>
>> >>> (FWIW I've done code which even uploads it's own hadoop-* JAR, but
>> what
>> >>> gets you is changes in the hadoop-native libs; you do need to get the
>> PATH
>> >>> var spot on)
>> >>>
>> >>>
>> >>> > I am unsure about mesos (standalone might be easier upgrade I guess
>> ?).
>> >>> >
>> >>> >
>> >>> > Proposal is for 1.6x line to continue to be supported with critical
>> >>> > fixes; newer features will require 2.x and so jdk8
>> >>> >
>> >>> > Regards
>> >>> > Mridul
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>> -
>> >>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> >>> For additional commands, e-mail: dev-h...@spark.apache.org
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Michael Gummelt
>> >> Software Engineer
>> >> Mesosphere
>> >
>> >
>> >
>> >
>> > --
>> > Luciano Resende
>> > http://twitter.com/lresende1975
>> > http://lresende.blogspot.com/
>>
>>
>>
>> --
>> Marcelo
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Kostas Sakellis]

Also, +1 on dropping jdk7 in Spark 2.0.

Kostas

On Mon, Mar 28, 2016 at 2:01 PM, Marcelo Vanzin  wrote:

> Finally got some internal feedback on this, and we're ok with
> requiring people to deploy jdk8 for 2.0, so +1 too.
>
> On Mon, Mar 28, 2016 at 1:15 PM, Luciano Resende 
> wrote:
> > +1, I also checked with few projects inside IBM that consume Spark and
> they
> > seem to be ok with the direction of droping JDK 7.
> >
> > On Mon, Mar 28, 2016 at 11:24 AM, Michael Gummelt <
> mgumm...@mesosphere.io>
> > wrote:
> >>
> >> +1 from Mesosphere
> >>
> >> On Mon, Mar 28, 2016 at 5:12 AM, Steve Loughran  >
> >> wrote:
> >>>
> >>>
> >>> > On 25 Mar 2016, at 01:59, Mridul Muralidharan 
> wrote:
> >>> >
> >>> > Removing compatibility (with jdk, etc) can be done with a major
> >>> > release- given that 7 has been EOLed a while back and is now
> unsupported, we
> >>> > have to decide if we drop support for it in 2.0 or 3.0 (2+ years
> from now).
> >>> >
> >>> > Given the functionality & performance benefits of going to jdk8,
> future
> >>> > enhancements relevant in 2.x timeframe ( scala, dependencies) which
> requires
> >>> > it, and simplicity wrt code, test & support it looks like a good
> checkpoint
> >>> > to drop jdk7 support.
> >>> >
> >>> > As already mentioned in the thread, existing yarn clusters are
> >>> > unaffected if they want to continue running jdk7 and yet use spark2
> (install
> >>> > jdk8 on all nodes and use it via JAVA_HOME, or worst case distribute
> jdk8 as
> >>> > archive - suboptimal).
> >>>
> >>> you wouldn't want to dist it as an archive; it's not just the binaries,
> >>> it's the install phase. And you'd better remember to put the JCE jar
> in on
> >>> top of the JDK for kerberos to work.
> >>>
> >>> setting up environment vars to point to JDK8 in the launched
> >>> app/container avoids that. Yes, the ops team do need to install java,
> but if
> >>> you offer them the choice of "installing a centrally managed Java" and
> >>> "having my code try and install it", they should go for the managed
> option.
> >>>
> >>> One thing to consider for 2.0 is to make it easier to set up those env
> >>> vars for both python and java. And, as the techniques for mixing JDK
> >>> versions is clearly not that well known, documenting it.
> >>>
> >>> (FWIW I've done code which even uploads it's own hadoop-* JAR, but what
> >>> gets you is changes in the hadoop-native libs; you do need to get the
> PATH
> >>> var spot on)
> >>>
> >>>
> >>> > I am unsure about mesos (standalone might be easier upgrade I guess
> ?).
> >>> >
> >>> >
> >>> > Proposal is for 1.6x line to continue to be supported with critical
> >>> > fixes; newer features will require 2.x and so jdk8
> >>> >
> >>> > Regards
> >>> > Mridul
> >>> >
> >>> >
> >>>
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> >>> For additional commands, e-mail: dev-h...@spark.apache.org
> >>>
> >>
> >>
> >>
> >> --
> >> Michael Gummelt
> >> Software Engineer
> >> Mesosphere
> >
> >
> >
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
>
>
>
> --
> Marcelo
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Marcelo Vanzin]

Finally got some internal feedback on this, and we're ok with
requiring people to deploy jdk8 for 2.0, so +1 too.

On Mon, Mar 28, 2016 at 1:15 PM, Luciano Resende  wrote:
> +1, I also checked with few projects inside IBM that consume Spark and they
> seem to be ok with the direction of droping JDK 7.
>
> On Mon, Mar 28, 2016 at 11:24 AM, Michael Gummelt 
> wrote:
>>
>> +1 from Mesosphere
>>
>> On Mon, Mar 28, 2016 at 5:12 AM, Steve Loughran 
>> wrote:
>>>
>>>
>>> > On 25 Mar 2016, at 01:59, Mridul Muralidharan  wrote:
>>> >
>>> > Removing compatibility (with jdk, etc) can be done with a major
>>> > release- given that 7 has been EOLed a while back and is now unsupported, 
>>> > we
>>> > have to decide if we drop support for it in 2.0 or 3.0 (2+ years from 
>>> > now).
>>> >
>>> > Given the functionality & performance benefits of going to jdk8, future
>>> > enhancements relevant in 2.x timeframe ( scala, dependencies) which 
>>> > requires
>>> > it, and simplicity wrt code, test & support it looks like a good 
>>> > checkpoint
>>> > to drop jdk7 support.
>>> >
>>> > As already mentioned in the thread, existing yarn clusters are
>>> > unaffected if they want to continue running jdk7 and yet use spark2 
>>> > (install
>>> > jdk8 on all nodes and use it via JAVA_HOME, or worst case distribute jdk8 
>>> > as
>>> > archive - suboptimal).
>>>
>>> you wouldn't want to dist it as an archive; it's not just the binaries,
>>> it's the install phase. And you'd better remember to put the JCE jar in on
>>> top of the JDK for kerberos to work.
>>>
>>> setting up environment vars to point to JDK8 in the launched
>>> app/container avoids that. Yes, the ops team do need to install java, but if
>>> you offer them the choice of "installing a centrally managed Java" and
>>> "having my code try and install it", they should go for the managed option.
>>>
>>> One thing to consider for 2.0 is to make it easier to set up those env
>>> vars for both python and java. And, as the techniques for mixing JDK
>>> versions is clearly not that well known, documenting it.
>>>
>>> (FWIW I've done code which even uploads it's own hadoop-* JAR, but what
>>> gets you is changes in the hadoop-native libs; you do need to get the PATH
>>> var spot on)
>>>
>>>
>>> > I am unsure about mesos (standalone might be easier upgrade I guess ?).
>>> >
>>> >
>>> > Proposal is for 1.6x line to continue to be supported with critical
>>> > fixes; newer features will require 2.x and so jdk8
>>> >
>>> > Regards
>>> > Mridul
>>> >
>>> >
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: dev-h...@spark.apache.org
>>>
>>
>>
>>
>> --
>> Michael Gummelt
>> Software Engineer
>> Mesosphere
>
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/



-- 
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Luciano Resende]

+1, I also checked with few projects inside IBM that consume Spark and they
seem to be ok with the direction of droping JDK 7.

On Mon, Mar 28, 2016 at 11:24 AM, Michael Gummelt 
wrote:

> +1 from Mesosphere
>
> On Mon, Mar 28, 2016 at 5:12 AM, Steve Loughran 
> wrote:
>
>>
>> > On 25 Mar 2016, at 01:59, Mridul Muralidharan  wrote:
>> >
>> > Removing compatibility (with jdk, etc) can be done with a major
>> release- given that 7 has been EOLed a while back and is now unsupported,
>> we have to decide if we drop support for it in 2.0 or 3.0 (2+ years from
>> now).
>> >
>> > Given the functionality & performance benefits of going to jdk8, future
>> enhancements relevant in 2.x timeframe ( scala, dependencies) which
>> requires it, and simplicity wrt code, test & support it looks like a good
>> checkpoint to drop jdk7 support.
>> >
>> > As already mentioned in the thread, existing yarn clusters are
>> unaffected if they want to continue running jdk7 and yet use spark2
>> (install jdk8 on all nodes and use it via JAVA_HOME, or worst case
>> distribute jdk8 as archive - suboptimal).
>>
>> you wouldn't want to dist it as an archive; it's not just the binaries,
>> it's the install phase. And you'd better remember to put the JCE jar in on
>> top of the JDK for kerberos to work.
>>
>> setting up environment vars to point to JDK8 in the launched
>> app/container avoids that. Yes, the ops team do need to install java, but
>> if you offer them the choice of "installing a centrally managed Java" and
>> "having my code try and install it", they should go for the managed option.
>>
>> One thing to consider for 2.0 is to make it easier to set up those env
>> vars for both python and java. And, as the techniques for mixing JDK
>> versions is clearly not that well known, documenting it.
>>
>> (FWIW I've done code which even uploads it's own hadoop-* JAR, but what
>> gets you is changes in the hadoop-native libs; you do need to get the PATH
>> var spot on)
>>
>>
>> > I am unsure about mesos (standalone might be easier upgrade I guess ?).
>> >
>> >
>> > Proposal is for 1.6x line to continue to be supported with critical
>> fixes; newer features will require 2.x and so jdk8
>> >
>> > Regards
>> > Mridul
>> >
>> >
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>>
>
>
> --
> Michael Gummelt
> Software Engineer
> Mesosphere
>



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [discuss] ending support for Java 7 in Spark 2.0

[Michael Gummelt]

+1 from Mesosphere

On Mon, Mar 28, 2016 at 5:12 AM, Steve Loughran 
wrote:

>
> > On 25 Mar 2016, at 01:59, Mridul Muralidharan  wrote:
> >
> > Removing compatibility (with jdk, etc) can be done with a major release-
> given that 7 has been EOLed a while back and is now unsupported, we have to
> decide if we drop support for it in 2.0 or 3.0 (2+ years from now).
> >
> > Given the functionality & performance benefits of going to jdk8, future
> enhancements relevant in 2.x timeframe ( scala, dependencies) which
> requires it, and simplicity wrt code, test & support it looks like a good
> checkpoint to drop jdk7 support.
> >
> > As already mentioned in the thread, existing yarn clusters are
> unaffected if they want to continue running jdk7 and yet use spark2
> (install jdk8 on all nodes and use it via JAVA_HOME, or worst case
> distribute jdk8 as archive - suboptimal).
>
> you wouldn't want to dist it as an archive; it's not just the binaries,
> it's the install phase. And you'd better remember to put the JCE jar in on
> top of the JDK for kerberos to work.
>
> setting up environment vars to point to JDK8 in the launched app/container
> avoids that. Yes, the ops team do need to install java, but if you offer
> them the choice of "installing a centrally managed Java" and "having my
> code try and install it", they should go for the managed option.
>
> One thing to consider for 2.0 is to make it easier to set up those env
> vars for both python and java. And, as the techniques for mixing JDK
> versions is clearly not that well known, documenting it.
>
> (FWIW I've done code which even uploads it's own hadoop-* JAR, but what
> gets you is changes in the hadoop-native libs; you do need to get the PATH
> var spot on)
>
>
> > I am unsure about mesos (standalone might be easier upgrade I guess ?).
> >
> >
> > Proposal is for 1.6x line to continue to be supported with critical
> fixes; newer features will require 2.x and so jdk8
> >
> > Regards
> > Mridul
> >
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>


-- 
Michael Gummelt
Software Engineer
Mesosphere

Re: [discuss] ending support for Java 7 in Spark 2.0

[Steve Loughran]

> On 25 Mar 2016, at 01:59, Mridul Muralidharan  wrote:
> 
> Removing compatibility (with jdk, etc) can be done with a major release- 
> given that 7 has been EOLed a while back and is now unsupported, we have to 
> decide if we drop support for it in 2.0 or 3.0 (2+ years from now).
> 
> Given the functionality & performance benefits of going to jdk8, future 
> enhancements relevant in 2.x timeframe ( scala, dependencies) which requires 
> it, and simplicity wrt code, test & support it looks like a good checkpoint 
> to drop jdk7 support.
> 
> As already mentioned in the thread, existing yarn clusters are unaffected if 
> they want to continue running jdk7 and yet use spark2 (install jdk8 on all 
> nodes and use it via JAVA_HOME, or worst case distribute jdk8 as archive - 
> suboptimal).

you wouldn't want to dist it as an archive; it's not just the binaries, it's 
the install phase. And you'd better remember to put the JCE jar in on top of 
the JDK for kerberos to work.

setting up environment vars to point to JDK8 in the launched app/container 
avoids that. Yes, the ops team do need to install java, but if you offer them 
the choice of "installing a centrally managed Java" and "having my code try and 
install it", they should go for the managed option.

One thing to consider for 2.0 is to make it easier to set up those env vars for 
both python and java. And, as the techniques for mixing JDK versions is clearly 
not that well known, documenting it. 

(FWIW I've done code which even uploads it's own hadoop-* JAR, but what gets 
you is changes in the hadoop-native libs; you do need to get the PATH var spot 
on)


> I am unsure about mesos (standalone might be easier upgrade I guess ?).
> 
> 
> Proposal is for 1.6x line to continue to be supported with critical fixes; 
> newer features will require 2.x and so jdk8
> 
> Regards 
> Mridul 
> 
> 


-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Koert Kuipers]

i asked around a little, and the general trend at our clients seems to be
that they plan to upgrade the clusters to java 8 within the year.

so with that in mind i wish this was a little later (i would have preferred
a java-8-only spark at the end of year). but since a major spark version
only comes around every so often, i guess it makes sense to make the jump
now. so:
+1 on dropping java 7
+1 on dropping scala 2.10

i would especially like to point out (as others have before me) that nobody
has come in and said they actually need scala 2.10 support, so that seems
like the easiest choice to me of all.

On Fri, Mar 25, 2016 at 10:03 AM, Andrew Ray  wrote:

> +1 on removing Java 7 and Scala 2.10 support.
>
> It looks to be entirely possible to support Java 8 containers in a YARN
> cluster otherwise running Java 7 (example code for alt JAVA_HOME
> https://issues.apache.org/jira/secure/attachment/12671739/YARN-1964.patch)
> so really there should be no big problem. Even if that somehow doesn't work
> I'm still +1 as the benefits are so large.
>
> I'd also like to point out that it is completely trivial to have multiple
> versions of Spark running concurrently on a YARN cluster. At my previous
> (extremely large) employer we had almost every release since 1.0 installed,
> with the latest being default and production apps pinned to a specific
> version. So if you want to keep using some Scala 2.10 only library or just
> don't want to migrate to Java 8, feel free to continue using Spark 1.x for
> those applications.
>
> IMHO we need to move on from EOL stuff to make room for the future (Java
> 9, Scala 2.12) and Spark 2.0 is the only chance we are going to have to do
> so for a long time.
>
> --Andrew
>
> On Thu, Mar 24, 2016 at 10:55 PM, Mridul Muralidharan 
> wrote:
>
>>
>> I do agree w.r.t scala 2.10 as well; similar arguments apply (though
>> there is a nuanced diff - source compatibility for scala vs binary
>> compatibility wrt Java)
>> Was there a proposal which did not go through ? Not sure if I missed it.
>>
>> Regards
>> Mridul
>>
>>
>> On Thursday, March 24, 2016, Koert Kuipers  wrote:
>>
>>> i think that logic is reasonable, but then the same should also apply to
>>> scala 2.10, which is also unmaintained/unsupported at this point (basically
>>> has been since march 2015 except for one hotfix due to a license
>>> incompatibility)
>>>
>>> who wants to support scala 2.10 three years after they did the last
>>> maintenance release?
>>>
>>>
>>> On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan 
>>> wrote:
>>>
 Removing compatibility (with jdk, etc) can be done with a major
 release- given that 7 has been EOLed a while back and is now unsupported,
 we have to decide if we drop support for it in 2.0 or 3.0 (2+ years from
 now).

 Given the functionality & performance benefits of going to jdk8, future
 enhancements relevant in 2.x timeframe ( scala, dependencies) which
 requires it, and simplicity wrt code, test & support it looks like a good
 checkpoint to drop jdk7 support.

 As already mentioned in the thread, existing yarn clusters are
 unaffected if they want to continue running jdk7 and yet use
 spark2 (install jdk8 on all nodes and use it via JAVA_HOME, or worst case
 distribute jdk8 as archive - suboptimal).
 I am unsure about mesos (standalone might be easier upgrade I guess ?).


 Proposal is for 1.6x line to continue to be supported with critical
 fixes; newer features will require 2.x and so jdk8

 Regards
 Mridul


 On Thursday, March 24, 2016, Marcelo Vanzin 
 wrote:

> On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin 
> wrote:
> > If you want to go down that route, you should also ask somebody who
> has had
> > experience managing a large organization's applications and try to
> update
> > Scala version.
>
> I understand both sides. But if you look at what I've been asking
> since the beginning, it's all about the cost and benefits of dropping
> support for java 1.7.
>
> The biggest argument in your original e-mail is about testing. And the
> testing cost is much bigger for supporting scala 2.10 than it is for
> supporting java 1.7. If you read one of my earlier replies, it should
> be even possible to just do everything in a single job - compile for
> java 7 and still be able to test things in 1.8, including lambdas,
> which seems to be the main thing you were worried about.
>
>
> > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin 
> wrote:
> >>
> >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin 
> wrote:
> >> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11,
> than
> >> > upgrading the JVM runtime from 7 

Re: [discuss] ending support for Java 7 in Spark 2.0

[Andrew Ray]

+1 on removing Java 7 and Scala 2.10 support.

It looks to be entirely possible to support Java 8 containers in a YARN
cluster otherwise running Java 7 (example code for alt JAVA_HOME
https://issues.apache.org/jira/secure/attachment/12671739/YARN-1964.patch)
so really there should be no big problem. Even if that somehow doesn't work
I'm still +1 as the benefits are so large.

I'd also like to point out that it is completely trivial to have multiple
versions of Spark running concurrently on a YARN cluster. At my previous
(extremely large) employer we had almost every release since 1.0 installed,
with the latest being default and production apps pinned to a specific
version. So if you want to keep using some Scala 2.10 only library or just
don't want to migrate to Java 8, feel free to continue using Spark 1.x for
those applications.

IMHO we need to move on from EOL stuff to make room for the future (Java 9,
Scala 2.12) and Spark 2.0 is the only chance we are going to have to do so
for a long time.

--Andrew

On Thu, Mar 24, 2016 at 10:55 PM, Mridul Muralidharan 
wrote:

>
> I do agree w.r.t scala 2.10 as well; similar arguments apply (though there
> is a nuanced diff - source compatibility for scala vs binary compatibility
> wrt Java)
> Was there a proposal which did not go through ? Not sure if I missed it.
>
> Regards
> Mridul
>
>
> On Thursday, March 24, 2016, Koert Kuipers  wrote:
>
>> i think that logic is reasonable, but then the same should also apply to
>> scala 2.10, which is also unmaintained/unsupported at this point (basically
>> has been since march 2015 except for one hotfix due to a license
>> incompatibility)
>>
>> who wants to support scala 2.10 three years after they did the last
>> maintenance release?
>>
>>
>> On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan 
>> wrote:
>>
>>> Removing compatibility (with jdk, etc) can be done with a major release-
>>> given that 7 has been EOLed a while back and is now unsupported, we have to
>>> decide if we drop support for it in 2.0 or 3.0 (2+ years from now).
>>>
>>> Given the functionality & performance benefits of going to jdk8, future
>>> enhancements relevant in 2.x timeframe ( scala, dependencies) which
>>> requires it, and simplicity wrt code, test & support it looks like a good
>>> checkpoint to drop jdk7 support.
>>>
>>> As already mentioned in the thread, existing yarn clusters are
>>> unaffected if they want to continue running jdk7 and yet use
>>> spark2 (install jdk8 on all nodes and use it via JAVA_HOME, or worst case
>>> distribute jdk8 as archive - suboptimal).
>>> I am unsure about mesos (standalone might be easier upgrade I guess ?).
>>>
>>>
>>> Proposal is for 1.6x line to continue to be supported with critical
>>> fixes; newer features will require 2.x and so jdk8
>>>
>>> Regards
>>> Mridul
>>>
>>>
>>> On Thursday, March 24, 2016, Marcelo Vanzin  wrote:
>>>
 On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin 
 wrote:
 > If you want to go down that route, you should also ask somebody who
 has had
 > experience managing a large organization's applications and try to
 update
 > Scala version.

 I understand both sides. But if you look at what I've been asking
 since the beginning, it's all about the cost and benefits of dropping
 support for java 1.7.

 The biggest argument in your original e-mail is about testing. And the
 testing cost is much bigger for supporting scala 2.10 than it is for
 supporting java 1.7. If you read one of my earlier replies, it should
 be even possible to just do everything in a single job - compile for
 java 7 and still be able to test things in 1.8, including lambdas,
 which seems to be the main thing you were worried about.


 > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin 
 wrote:
 >>
 >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin 
 wrote:
 >> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
 >> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11
 are
 >> > not
 >> > binary compatible, whereas JVM 7 and 8 are binary compatible except
 >> > certain
 >> > esoteric cases.
 >>
 >> True, but ask anyone who manages a large cluster how long it would
 >> take them to upgrade the jdk across their cluster and validate all
 >> their applications and everything... binary compatibility is a tiny
 >> drop in that bucket.
 >>
 >> --
 >> Marcelo
 >
 >



 --
 Marcelo

 -
 To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
 For additional commands, e-mail: dev-h...@spark.apache.org


>>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Mridul Muralidharan]

I do agree w.r.t scala 2.10 as well; similar arguments apply (though there
is a nuanced diff - source compatibility for scala vs binary compatibility
wrt Java)
Was there a proposal which did not go through ? Not sure if I missed it.

Regards
Mridul

On Thursday, March 24, 2016, Koert Kuipers  wrote:

> i think that logic is reasonable, but then the same should also apply to
> scala 2.10, which is also unmaintained/unsupported at this point (basically
> has been since march 2015 except for one hotfix due to a license
> incompatibility)
>
> who wants to support scala 2.10 three years after they did the last
> maintenance release?
>
>
> On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan  > wrote:
>
>> Removing compatibility (with jdk, etc) can be done with a major release-
>> given that 7 has been EOLed a while back and is now unsupported, we have to
>> decide if we drop support for it in 2.0 or 3.0 (2+ years from now).
>>
>> Given the functionality & performance benefits of going to jdk8, future
>> enhancements relevant in 2.x timeframe ( scala, dependencies) which
>> requires it, and simplicity wrt code, test & support it looks like a good
>> checkpoint to drop jdk7 support.
>>
>> As already mentioned in the thread, existing yarn clusters are unaffected
>> if they want to continue running jdk7 and yet use spark2 (install jdk8 on
>> all nodes and use it via JAVA_HOME, or worst case distribute jdk8 as
>> archive - suboptimal).
>> I am unsure about mesos (standalone might be easier upgrade I guess ?).
>>
>>
>> Proposal is for 1.6x line to continue to be supported with critical
>> fixes; newer features will require 2.x and so jdk8
>>
>> Regards
>> Mridul
>>
>>
>> On Thursday, March 24, 2016, Marcelo Vanzin > > wrote:
>>
>>> On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin 
>>> wrote:
>>> > If you want to go down that route, you should also ask somebody who
>>> has had
>>> > experience managing a large organization's applications and try to
>>> update
>>> > Scala version.
>>>
>>> I understand both sides. But if you look at what I've been asking
>>> since the beginning, it's all about the cost and benefits of dropping
>>> support for java 1.7.
>>>
>>> The biggest argument in your original e-mail is about testing. And the
>>> testing cost is much bigger for supporting scala 2.10 than it is for
>>> supporting java 1.7. If you read one of my earlier replies, it should
>>> be even possible to just do everything in a single job - compile for
>>> java 7 and still be able to test things in 1.8, including lambdas,
>>> which seems to be the main thing you were worried about.
>>>
>>>
>>> > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin 
>>> wrote:
>>> >>
>>> >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin 
>>> wrote:
>>> >> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
>>> >> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11
>>> are
>>> >> > not
>>> >> > binary compatible, whereas JVM 7 and 8 are binary compatible except
>>> >> > certain
>>> >> > esoteric cases.
>>> >>
>>> >> True, but ask anyone who manages a large cluster how long it would
>>> >> take them to upgrade the jdk across their cluster and validate all
>>> >> their applications and everything... binary compatibility is a tiny
>>> >> drop in that bucket.
>>> >>
>>> >> --
>>> >> Marcelo
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: dev-h...@spark.apache.org
>>>
>>>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Mridul Muralidharan]

I do agree w.r.t scala 2.10 as well; similar arguments apply (though there
is a nuanced diff - source compatibility for scala vs binary compatibility
wrt Java)
Was there a proposal which did not go through ? Not sure if I missed it.

Regards
Mridul

On Thursday, March 24, 2016, Koert Kuipers  wrote:

> i think that logic is reasonable, but then the same should also apply to
> scala 2.10, which is also unmaintained/unsupported at this point (basically
> has been since march 2015 except for one hotfix due to a license
> incompatibility)
>
> who wants to support scala 2.10 three years after they did the last
> maintenance release?
>
>
> On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan  > wrote:
>
>> Removing compatibility (with jdk, etc) can be done with a major release-
>> given that 7 has been EOLed a while back and is now unsupported, we have to
>> decide if we drop support for it in 2.0 or 3.0 (2+ years from now).
>>
>> Given the functionality & performance benefits of going to jdk8, future
>> enhancements relevant in 2.x timeframe ( scala, dependencies) which
>> requires it, and simplicity wrt code, test & support it looks like a good
>> checkpoint to drop jdk7 support.
>>
>> As already mentioned in the thread, existing yarn clusters are unaffected
>> if they want to continue running jdk7 and yet use spark2 (install jdk8 on
>> all nodes and use it via JAVA_HOME, or worst case distribute jdk8 as
>> archive - suboptimal).
>> I am unsure about mesos (standalone might be easier upgrade I guess ?).
>>
>>
>> Proposal is for 1.6x line to continue to be supported with critical
>> fixes; newer features will require 2.x and so jdk8
>>
>> Regards
>> Mridul
>>
>>
>> On Thursday, March 24, 2016, Marcelo Vanzin > > wrote:
>>
>>> On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin 
>>> wrote:
>>> > If you want to go down that route, you should also ask somebody who
>>> has had
>>> > experience managing a large organization's applications and try to
>>> update
>>> > Scala version.
>>>
>>> I understand both sides. But if you look at what I've been asking
>>> since the beginning, it's all about the cost and benefits of dropping
>>> support for java 1.7.
>>>
>>> The biggest argument in your original e-mail is about testing. And the
>>> testing cost is much bigger for supporting scala 2.10 than it is for
>>> supporting java 1.7. If you read one of my earlier replies, it should
>>> be even possible to just do everything in a single job - compile for
>>> java 7 and still be able to test things in 1.8, including lambdas,
>>> which seems to be the main thing you were worried about.
>>>
>>>
>>> > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin 
>>> wrote:
>>> >>
>>> >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin 
>>> wrote:
>>> >> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
>>> >> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11
>>> are
>>> >> > not
>>> >> > binary compatible, whereas JVM 7 and 8 are binary compatible except
>>> >> > certain
>>> >> > esoteric cases.
>>> >>
>>> >> True, but ask anyone who manages a large cluster how long it would
>>> >> take them to upgrade the jdk across their cluster and validate all
>>> >> their applications and everything... binary compatibility is a tiny
>>> >> drop in that bucket.
>>> >>
>>> >> --
>>> >> Marcelo
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: dev-h...@spark.apache.org
>>>
>>>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Koert Kuipers]

i think that logic is reasonable, but then the same should also apply to
scala 2.10, which is also unmaintained/unsupported at this point (basically
has been since march 2015 except for one hotfix due to a license
incompatibility)

who wants to support scala 2.10 three years after they did the last
maintenance release?


On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan 
wrote:

> Removing compatibility (with jdk, etc) can be done with a major release-
> given that 7 has been EOLed a while back and is now unsupported, we have to
> decide if we drop support for it in 2.0 or 3.0 (2+ years from now).
>
> Given the functionality & performance benefits of going to jdk8, future
> enhancements relevant in 2.x timeframe ( scala, dependencies) which
> requires it, and simplicity wrt code, test & support it looks like a good
> checkpoint to drop jdk7 support.
>
> As already mentioned in the thread, existing yarn clusters are unaffected
> if they want to continue running jdk7 and yet use spark2 (install jdk8 on
> all nodes and use it via JAVA_HOME, or worst case distribute jdk8 as
> archive - suboptimal).
> I am unsure about mesos (standalone might be easier upgrade I guess ?).
>
>
> Proposal is for 1.6x line to continue to be supported with critical fixes; 
> newer
> features will require 2.x and so jdk8
>
> Regards
> Mridul
>
>
> On Thursday, March 24, 2016, Marcelo Vanzin  wrote:
>
>> On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin  wrote:
>> > If you want to go down that route, you should also ask somebody who has
>> had
>> > experience managing a large organization's applications and try to
>> update
>> > Scala version.
>>
>> I understand both sides. But if you look at what I've been asking
>> since the beginning, it's all about the cost and benefits of dropping
>> support for java 1.7.
>>
>> The biggest argument in your original e-mail is about testing. And the
>> testing cost is much bigger for supporting scala 2.10 than it is for
>> supporting java 1.7. If you read one of my earlier replies, it should
>> be even possible to just do everything in a single job - compile for
>> java 7 and still be able to test things in 1.8, including lambdas,
>> which seems to be the main thing you were worried about.
>>
>>
>> > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin 
>> wrote:
>> >>
>> >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin 
>> wrote:
>> >> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
>> >> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11
>> are
>> >> > not
>> >> > binary compatible, whereas JVM 7 and 8 are binary compatible except
>> >> > certain
>> >> > esoteric cases.
>> >>
>> >> True, but ask anyone who manages a large cluster how long it would
>> >> take them to upgrade the jdk across their cluster and validate all
>> >> their applications and everything... binary compatibility is a tiny
>> >> drop in that bucket.
>> >>
>> >> --
>> >> Marcelo
>> >
>> >
>>
>>
>>
>> --
>> Marcelo
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Mridul Muralidharan]

Removing compatibility (with jdk, etc) can be done with a major release-
given that 7 has been EOLed a while back and is now unsupported, we have to
decide if we drop support for it in 2.0 or 3.0 (2+ years from now).

Given the functionality & performance benefits of going to jdk8, future
enhancements relevant in 2.x timeframe ( scala, dependencies) which
requires it, and simplicity wrt code, test & support it looks like a good
checkpoint to drop jdk7 support.

As already mentioned in the thread, existing yarn clusters are unaffected
if they want to continue running jdk7 and yet use spark2 (install jdk8 on
all nodes and use it via JAVA_HOME, or worst case distribute jdk8 as
archive - suboptimal).
I am unsure about mesos (standalone might be easier upgrade I guess ?).


Proposal is for 1.6x line to continue to be supported with critical
fixes; newer
features will require 2.x and so jdk8

Regards
Mridul


On Thursday, March 24, 2016, Marcelo Vanzin  wrote:

> On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin  > wrote:
> > If you want to go down that route, you should also ask somebody who has
> had
> > experience managing a large organization's applications and try to update
> > Scala version.
>
> I understand both sides. But if you look at what I've been asking
> since the beginning, it's all about the cost and benefits of dropping
> support for java 1.7.
>
> The biggest argument in your original e-mail is about testing. And the
> testing cost is much bigger for supporting scala 2.10 than it is for
> supporting java 1.7. If you read one of my earlier replies, it should
> be even possible to just do everything in a single job - compile for
> java 7 and still be able to test things in 1.8, including lambdas,
> which seems to be the main thing you were worried about.
>
>
> > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin  > wrote:
> >>
> >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin  > wrote:
> >> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
> >> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are
> >> > not
> >> > binary compatible, whereas JVM 7 and 8 are binary compatible except
> >> > certain
> >> > esoteric cases.
> >>
> >> True, but ask anyone who manages a large cluster how long it would
> >> take them to upgrade the jdk across their cluster and validate all
> >> their applications and everything... binary compatibility is a tiny
> >> drop in that bucket.
> >>
> >> --
> >> Marcelo
> >
> >
>
>
>
> --
> Marcelo
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org 
> For additional commands, e-mail: dev-h...@spark.apache.org 
>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Koert Kuipers]

i think marcelo also pointed this out before. its very interesting to hear,
i was not aware of that until today. it would mean we would only have to
convince a group/client with a cluster to install jdk8 on the nodes,
without actually transitioning to it, if i understand it correctly. that
would definitely lower the hurdle by a lot.

On Thu, Mar 24, 2016 at 9:36 PM, Mridul Muralidharan 
wrote:

>
> Container Java version can be different from yarn Java version : we run
> jobs with jdk8 on jdk7 cluster without issues.
>
> Regards
> Mridul
>
>
> On Thursday, March 24, 2016, Koert Kuipers  wrote:
>
>> i guess what i am saying is that in a yarn world the only hard
>> restrictions left are the the containers you run in, which means the hadoop
>> version, java version and python version (if you use python).
>>
>>
>> On Thu, Mar 24, 2016 at 12:39 PM, Koert Kuipers 
>> wrote:
>>
>>> The group will not upgrade to spark 2.0 themselves, but they are mostly
>>> fine with vendors like us deploying our application via yarn with whatever
>>> spark version we choose (and bundle, so they do not install it separately,
>>> they might not even be aware of what spark version we use). This all works
>>> because spark does not need to be on the cluster nodes, just on the one
>>> machine where our application gets launched. Having yarn is pretty awesome
>>> in this respect.
>>>
>>> On Thu, Mar 24, 2016 at 12:25 PM, Sean Owen  wrote:
>>>
 (PS CDH5 runs fine with Java 8, but I understand your more general
 point.)

 This is a familiar context indeed, but in that context, would a group
 not wanting to update to Java 8 want to manually put Spark 2.0 into
 the mix? That is, if this is a context where the cluster is
 purposefully some stable mix of components, would you be updating just
 one?

 You make a good point about Scala being more library than
 infrastructure component. So it can be updated on a per-app basis. On
 the one hand it's harder to handle different Scala versions from the
 framework side, it's less hard on the deployment side.

 On Thu, Mar 24, 2016 at 4:27 PM, Koert Kuipers 
 wrote:
 > i think the arguments are convincing, but it also makes me wonder if
 i live
 > in some kind of alternate universe... we deploy on customers
 clusters, where
 > the OS, python version, java version and hadoop distro are not chosen
 by us.
 > so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply
 have
 > access to a single proxy machine and launch through yarn. asking them
 to
 > upgrade java is pretty much out of the question or a 6+ month ordeal.
 of the
 > 10 client clusters i can think of on the top of my head all of them
 are on
 > java 7, none are on java 8. so by doing this you would make spark 2
 > basically unusable for us (unless most of them have plans of
 upgrading in
 > near term to java 8, i will ask around and report back...).
 >
 > on a side note, its particularly interesting to me that spark 2 chose
 to
 > continue support for scala 2.10, because even for us in our very
 constricted
 > client environments the scala version is something we can easily
 upgrade (we
 > just deploy a custom build of spark for the relevant scala version and
 > hadoop distro). and because scala is not a dependency of any hadoop
 distro
 > (so not on classpath, which i am very happy about) we can use
 whatever scala
 > version we like. also i found the upgrade path from scala 2.10 to
 2.11 to be
 > very easy, so i have a hard time understanding why anyone would stay
 on
 > scala 2.10. and finally with scala 2.12 around the corner you really
 dont
 > want to be supporting 3 versions. so clearly i am missing something
 here.
 >
 >
 >
 > On Thu, Mar 24, 2016 at 8:52 AM, Jean-Baptiste Onofré <
 j...@nanthrax.net>
 > wrote:
 >>
 >> +1 to support Java 8 (and future) *only* in Spark 2.0, and end
 support of
 >> Java 7. It makes sense.
 >>
 >> Regards
 >> JB
 >>
 >>
 >> On 03/24/2016 08:27 AM, Reynold Xin wrote:
 >>>
 >>> About a year ago we decided to drop Java 6 support in Spark 1.5. I
 am
 >>> wondering if we should also just drop Java 7 support in Spark 2.0
 (i.e.
 >>> Spark 2.0 would require Java 8 to run).
 >>>
 >>> Oracle ended public updates for JDK 7 in one year ago (Apr 2015),
 and
 >>> removed public downloads for JDK 7 in July 2015. In the past I've
 >>> actually been against dropping Java 8, but today I ran into an issue
 >>> with the new Dataset API not working well with Java 8 lambdas, and
 that
 >>> changed my opinion on this.
 >>>
 >>> I've been thinking more about this issue today and 

Re: [discuss] ending support for Java 7 in Spark 2.0

[Mridul Muralidharan]

Container Java version can be different from yarn Java version : we run
jobs with jdk8 on jdk7 cluster without issues.

Regards
Mridul

On Thursday, March 24, 2016, Koert Kuipers  wrote:

> i guess what i am saying is that in a yarn world the only hard
> restrictions left are the the containers you run in, which means the hadoop
> version, java version and python version (if you use python).
>
>
> On Thu, Mar 24, 2016 at 12:39 PM, Koert Kuipers  > wrote:
>
>> The group will not upgrade to spark 2.0 themselves, but they are mostly
>> fine with vendors like us deploying our application via yarn with whatever
>> spark version we choose (and bundle, so they do not install it separately,
>> they might not even be aware of what spark version we use). This all works
>> because spark does not need to be on the cluster nodes, just on the one
>> machine where our application gets launched. Having yarn is pretty awesome
>> in this respect.
>>
>> On Thu, Mar 24, 2016 at 12:25 PM, Sean Owen > > wrote:
>>
>>> (PS CDH5 runs fine with Java 8, but I understand your more general
>>> point.)
>>>
>>> This is a familiar context indeed, but in that context, would a group
>>> not wanting to update to Java 8 want to manually put Spark 2.0 into
>>> the mix? That is, if this is a context where the cluster is
>>> purposefully some stable mix of components, would you be updating just
>>> one?
>>>
>>> You make a good point about Scala being more library than
>>> infrastructure component. So it can be updated on a per-app basis. On
>>> the one hand it's harder to handle different Scala versions from the
>>> framework side, it's less hard on the deployment side.
>>>
>>> On Thu, Mar 24, 2016 at 4:27 PM, Koert Kuipers >> > wrote:
>>> > i think the arguments are convincing, but it also makes me wonder if i
>>> live
>>> > in some kind of alternate universe... we deploy on customers clusters,
>>> where
>>> > the OS, python version, java version and hadoop distro are not chosen
>>> by us.
>>> > so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply
>>> have
>>> > access to a single proxy machine and launch through yarn. asking them
>>> to
>>> > upgrade java is pretty much out of the question or a 6+ month ordeal.
>>> of the
>>> > 10 client clusters i can think of on the top of my head all of them
>>> are on
>>> > java 7, none are on java 8. so by doing this you would make spark 2
>>> > basically unusable for us (unless most of them have plans of upgrading
>>> in
>>> > near term to java 8, i will ask around and report back...).
>>> >
>>> > on a side note, its particularly interesting to me that spark 2 chose
>>> to
>>> > continue support for scala 2.10, because even for us in our very
>>> constricted
>>> > client environments the scala version is something we can easily
>>> upgrade (we
>>> > just deploy a custom build of spark for the relevant scala version and
>>> > hadoop distro). and because scala is not a dependency of any hadoop
>>> distro
>>> > (so not on classpath, which i am very happy about) we can use whatever
>>> scala
>>> > version we like. also i found the upgrade path from scala 2.10 to 2.11
>>> to be
>>> > very easy, so i have a hard time understanding why anyone would stay on
>>> > scala 2.10. and finally with scala 2.12 around the corner you really
>>> dont
>>> > want to be supporting 3 versions. so clearly i am missing something
>>> here.
>>> >
>>> >
>>> >
>>> > On Thu, Mar 24, 2016 at 8:52 AM, Jean-Baptiste Onofré >> >
>>> > wrote:
>>> >>
>>> >> +1 to support Java 8 (and future) *only* in Spark 2.0, and end
>>> support of
>>> >> Java 7. It makes sense.
>>> >>
>>> >> Regards
>>> >> JB
>>> >>
>>> >>
>>> >> On 03/24/2016 08:27 AM, Reynold Xin wrote:
>>> >>>
>>> >>> About a year ago we decided to drop Java 6 support in Spark 1.5. I am
>>> >>> wondering if we should also just drop Java 7 support in Spark 2.0
>>> (i.e.
>>> >>> Spark 2.0 would require Java 8 to run).
>>> >>>
>>> >>> Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
>>> >>> removed public downloads for JDK 7 in July 2015. In the past I've
>>> >>> actually been against dropping Java 8, but today I ran into an issue
>>> >>> with the new Dataset API not working well with Java 8 lambdas, and
>>> that
>>> >>> changed my opinion on this.
>>> >>>
>>> >>> I've been thinking more about this issue today and also talked with a
>>> >>> lot people offline to gather feedback, and I actually think the pros
>>> >>> outweighs the cons, for the following reasons (in some rough order of
>>> >>> importance):
>>> >>>
>>> >>> 1. It is complicated to test how well Spark APIs work for Java
>>> lambdas
>>> >>> if we support Java 7. Jenkins machines need to 

Re: [discuss] ending support for Java 7 in Spark 2.0

[Koert Kuipers]

the good news is, that from an shared infrastructure perspective, most
places have zero scala, so the upgrade is actually very easy. i can see how
it would be different for say twitter

On Thu, Mar 24, 2016 at 7:50 PM, Reynold Xin  wrote:

> If you want to go down that route, you should also ask somebody who has
> had experience managing a large organization's applications and try to
> update Scala version.
>
>
> On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin 
> wrote:
>
>> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin  wrote:
>> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
>> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are
>> not
>> > binary compatible, whereas JVM 7 and 8 are binary compatible except
>> certain
>> > esoteric cases.
>>
>> True, but ask anyone who manages a large cluster how long it would
>> take them to upgrade the jdk across their cluster and validate all
>> their applications and everything... binary compatibility is a tiny
>> drop in that bucket.
>>
>> --
>> Marcelo
>>
>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Michael Armbrust]

On Thu, Mar 24, 2016 at 4:54 PM, Mark Hamstra 
 wrote:

> It's a pain in the ass.  Especially if some of your transitive
> dependencies never upgraded from 2.10 to 2.11.
>

Yeah, I'm going to have to agree here.  It is not as bad as it was in the
2.9 days, but its still non-trivial due to the eco-system part of it.  For
this reason I think that it is premature to drop support for 2.10.x.

Re: [discuss] ending support for Java 7 in Spark 2.0

[Mark Hamstra]

There aren't many such libraries, but there are a few.  When faced with one
of those dependencies that still doesn't go beyond 2.10, you essentially
have the choice of taking on the maintenance burden to bring the library up
to date, or you do what is potentially a fairly larger refactoring to use
an alternative but well-maintained library.

On Thu, Mar 24, 2016 at 4:53 PM, Kostas Sakellis 
wrote:

> In addition, with Spark 2.0, we are throwing away binary compatibility
> anyways so user applications will have to be recompiled.
>
> The only argument I can see is for libraries that have already been built
> on Scala 2.10 that are no longer being maintained. How big of an issue do
> we think that is?
>
> Kostas
>
> On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin 
> wrote:
>
>> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin  wrote:
>> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
>> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are
>> not
>> > binary compatible, whereas JVM 7 and 8 are binary compatible except
>> certain
>> > esoteric cases.
>>
>> True, but ask anyone who manages a large cluster how long it would
>> take them to upgrade the jdk across their cluster and validate all
>> their applications and everything... binary compatibility is a tiny
>> drop in that bucket.
>>
>> --
>> Marcelo
>>
>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Marcelo Vanzin]

On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin  wrote:
> If you want to go down that route, you should also ask somebody who has had
> experience managing a large organization's applications and try to update
> Scala version.

I understand both sides. But if you look at what I've been asking
since the beginning, it's all about the cost and benefits of dropping
support for java 1.7.

The biggest argument in your original e-mail is about testing. And the
testing cost is much bigger for supporting scala 2.10 than it is for
supporting java 1.7. If you read one of my earlier replies, it should
be even possible to just do everything in a single job - compile for
java 7 and still be able to test things in 1.8, including lambdas,
which seems to be the main thing you were worried about.


> On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin  wrote:
>>
>> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin  wrote:
>> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
>> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are
>> > not
>> > binary compatible, whereas JVM 7 and 8 are binary compatible except
>> > certain
>> > esoteric cases.
>>
>> True, but ask anyone who manages a large cluster how long it would
>> take them to upgrade the jdk across their cluster and validate all
>> their applications and everything... binary compatibility is a tiny
>> drop in that bucket.
>>
>> --
>> Marcelo
>
>



-- 
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Mark Hamstra]

It's a pain in the ass.  Especially if some of your transitive dependencies
never upgraded from 2.10 to 2.11.

On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin  wrote:

> If you want to go down that route, you should also ask somebody who has
> had experience managing a large organization's applications and try to
> update Scala version.
>
>
> On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin 
> wrote:
>
>> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin  wrote:
>> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
>> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are
>> not
>> > binary compatible, whereas JVM 7 and 8 are binary compatible except
>> certain
>> > esoteric cases.
>>
>> True, but ask anyone who manages a large cluster how long it would
>> take them to upgrade the jdk across their cluster and validate all
>> their applications and everything... binary compatibility is a tiny
>> drop in that bucket.
>>
>> --
>> Marcelo
>>
>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Kostas Sakellis]

In addition, with Spark 2.0, we are throwing away binary compatibility
anyways so user applications will have to be recompiled.

The only argument I can see is for libraries that have already been built
on Scala 2.10 that are no longer being maintained. How big of an issue do
we think that is?

Kostas

On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin  wrote:

> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin  wrote:
> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are
> not
> > binary compatible, whereas JVM 7 and 8 are binary compatible except
> certain
> > esoteric cases.
>
> True, but ask anyone who manages a large cluster how long it would
> take them to upgrade the jdk across their cluster and validate all
> their applications and everything... binary compatibility is a tiny
> drop in that bucket.
>
> --
> Marcelo
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Reynold Xin]

If you want to go down that route, you should also ask somebody who has had
experience managing a large organization's applications and try to update
Scala version.


On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin  wrote:

> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin  wrote:
> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are
> not
> > binary compatible, whereas JVM 7 and 8 are binary compatible except
> certain
> > esoteric cases.
>
> True, but ask anyone who manages a large cluster how long it would
> take them to upgrade the jdk across their cluster and validate all
> their applications and everything... binary compatibility is a tiny
> drop in that bucket.
>
> --
> Marcelo
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Marcelo Vanzin]

On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin  wrote:
> Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
> upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are not
> binary compatible, whereas JVM 7 and 8 are binary compatible except certain
> esoteric cases.

True, but ask anyone who manages a large cluster how long it would
take them to upgrade the jdk across their cluster and validate all
their applications and everything... binary compatibility is a tiny
drop in that bucket.

-- 
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Reynold Xin]

Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are not
binary compatible, whereas JVM 7 and 8 are binary compatible except certain
esoteric cases.


On Thu, Mar 24, 2016 at 4:44 PM, Kostas Sakellis 
wrote:

> If an argument here is the ongoing build/maintenance burden I think we
> should seriously consider dropping scala 2.10 in Spark 2.0. Supporting
> scala 2.10 is bigger build/infrastructure burden than supporting jdk7 since
> you actually have to build different artifacts and test them whereas you
> can target Spark onto 1.7 and just test it on JDK8.
>
> In addition, as others pointed out, it seems like a bigger pain to drop
> support for a JDK than scala version. So if we are considering dropping
> java 7, which is a breaking change on the infra side, now is also a good
> time to drop Scala 2.10 support.
>
> Kostas
>
> P.S. I haven't heard anyone on this thread fight for Scala 2.10 support.
>
> On Thu, Mar 24, 2016 at 2:46 PM, Marcelo Vanzin 
> wrote:
>
>> On Thu, Mar 24, 2016 at 2:41 PM, Jakob Odersky  wrote:
>> > You can, but since it's going to be a maintainability issue I would
>> > argue it is in fact a problem.
>>
>> Every thing you choose to support generates a maintenance burden.
>> Support 3 versions of Scala would be a huge maintenance burden, for
>> example, as is supporting 2 versions of the JDK. Just note that,
>> technically, we do support 2 versions of the jdk today; we just don't
>> do a lot of automated testing on jdk 8 (PRs are all built with jdk 7
>> AFAIK).
>>
>> So at the end it's a compromise. How many users will be affected by
>> your choices? That's the question that I think is the most important.
>> If switching to java 8-only means a bunch of users won't be able to
>> upgrade, it means that Spark 2.0 will get less use than 1.x and will
>> take longer to gain traction. That has other ramifications - such as
>> less use means less issues might be found and the overall quality may
>> suffer in the beginning of this transition.
>>
>> --
>> Marcelo
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Kostas Sakellis]

If an argument here is the ongoing build/maintenance burden I think we
should seriously consider dropping scala 2.10 in Spark 2.0. Supporting
scala 2.10 is bigger build/infrastructure burden than supporting jdk7 since
you actually have to build different artifacts and test them whereas you
can target Spark onto 1.7 and just test it on JDK8.

In addition, as others pointed out, it seems like a bigger pain to drop
support for a JDK than scala version. So if we are considering dropping
java 7, which is a breaking change on the infra side, now is also a good
time to drop Scala 2.10 support.

Kostas

P.S. I haven't heard anyone on this thread fight for Scala 2.10 support.

On Thu, Mar 24, 2016 at 2:46 PM, Marcelo Vanzin  wrote:

> On Thu, Mar 24, 2016 at 2:41 PM, Jakob Odersky  wrote:
> > You can, but since it's going to be a maintainability issue I would
> > argue it is in fact a problem.
>
> Every thing you choose to support generates a maintenance burden.
> Support 3 versions of Scala would be a huge maintenance burden, for
> example, as is supporting 2 versions of the JDK. Just note that,
> technically, we do support 2 versions of the jdk today; we just don't
> do a lot of automated testing on jdk 8 (PRs are all built with jdk 7
> AFAIK).
>
> So at the end it's a compromise. How many users will be affected by
> your choices? That's the question that I think is the most important.
> If switching to java 8-only means a bunch of users won't be able to
> upgrade, it means that Spark 2.0 will get less use than 1.x and will
> take longer to gain traction. That has other ramifications - such as
> less use means less issues might be found and the overall quality may
> suffer in the beginning of this transition.
>
> --
> Marcelo
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Marcelo Vanzin]

On Thu, Mar 24, 2016 at 2:41 PM, Jakob Odersky  wrote:
> You can, but since it's going to be a maintainability issue I would
> argue it is in fact a problem.

Every thing you choose to support generates a maintenance burden.
Support 3 versions of Scala would be a huge maintenance burden, for
example, as is supporting 2 versions of the JDK. Just note that,
technically, we do support 2 versions of the jdk today; we just don't
do a lot of automated testing on jdk 8 (PRs are all built with jdk 7
AFAIK).

So at the end it's a compromise. How many users will be affected by
your choices? That's the question that I think is the most important.
If switching to java 8-only means a bunch of users won't be able to
upgrade, it means that Spark 2.0 will get less use than 1.x and will
take longer to gain traction. That has other ramifications - such as
less use means less issues might be found and the overall quality may
suffer in the beginning of this transition.

-- 
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Jakob Odersky]

I mean from the perspective of someone developing Spark, it makes
things more complicated. It's just my point of view, people that
actually support Spark deployments may have a different opinion ;)

On Thu, Mar 24, 2016 at 2:41 PM, Jakob Odersky  wrote:
> You can, but since it's going to be a maintainability issue I would
> argue it is in fact a problem.
>
> On Thu, Mar 24, 2016 at 2:34 PM, Marcelo Vanzin  wrote:
>> Hi Jakob,
>>
>> On Thu, Mar 24, 2016 at 2:29 PM, Jakob Odersky  wrote:
>>> Reynold's 3rd point is particularly strong in my opinion. Supporting
>>> Consider what would happen if Spark 2.0 doesn't require Java 8 and
>>> hence not support Scala 2.12. Will it be stuck on an older version
>>> until 3.0 is out?
>>
>> That's a false choice. You can support 2.10 (or 2.11) on Java 7 and
>> 2.12 on Java 8.
>>
>> I'm not saying it's a great idea, just that what you're suggesting
>> isn't really a problem.
>>
>> --
>> Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Jakob Odersky]

You can, but since it's going to be a maintainability issue I would
argue it is in fact a problem.

On Thu, Mar 24, 2016 at 2:34 PM, Marcelo Vanzin  wrote:
> Hi Jakob,
>
> On Thu, Mar 24, 2016 at 2:29 PM, Jakob Odersky  wrote:
>> Reynold's 3rd point is particularly strong in my opinion. Supporting
>> Consider what would happen if Spark 2.0 doesn't require Java 8 and
>> hence not support Scala 2.12. Will it be stuck on an older version
>> until 3.0 is out?
>
> That's a false choice. You can support 2.10 (or 2.11) on Java 7 and
> 2.12 on Java 8.
>
> I'm not saying it's a great idea, just that what you're suggesting
> isn't really a problem.
>
> --
> Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Marcelo Vanzin]

Hi Jakob,

On Thu, Mar 24, 2016 at 2:29 PM, Jakob Odersky  wrote:
> Reynold's 3rd point is particularly strong in my opinion. Supporting
> Consider what would happen if Spark 2.0 doesn't require Java 8 and
> hence not support Scala 2.12. Will it be stuck on an older version
> until 3.0 is out?

That's a false choice. You can support 2.10 (or 2.11) on Java 7 and
2.12 on Java 8.

I'm not saying it's a great idea, just that what you're suggesting
isn't really a problem.

-- 
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Romi Kuntsman]

+1 for Java 8 only

I think it will make it easier to make a unified API for Java and Scala,
instead of the wrappers of Java over Scala.
On Mar 24, 2016 11:46 AM, "Stephen Boesch"  wrote:

> +1 for java8 only   +1 for 2.11+ only .At this point scala libraries
> supporting only 2.10 are typically less active and/or poorly maintained.
> That trend will only continue when considering the lifespan of spark 2.X.
>
> 2016-03-24 11:32 GMT-07:00 Steve Loughran :
>
>>
>> On 24 Mar 2016, at 15:27, Koert Kuipers  wrote:
>>
>> i think the arguments are convincing, but it also makes me wonder if i
>> live in some kind of alternate universe... we deploy on customers clusters,
>> where the OS, python version, java version and hadoop distro are not chosen
>> by us. so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply
>> have access to a single proxy machine and launch through yarn. asking them
>> to upgrade java is pretty much out of the question or a 6+ month ordeal. of
>> the 10 client clusters i can think of on the top of my head all of them are
>> on java 7, none are on java 8. so by doing this you would make spark 2
>> basically unusable for us (unless most of them have plans of upgrading in
>> near term to java 8, i will ask around and report back...).
>>
>>
>>
>> It's not actually mandatory for the process executing in the Yarn cluster
>> to run with the same JVM as the rest of the Hadoop stack; all that is
>> needed is for the environment variables to set up the JAVA_HOME and PATH.
>> Switching JVMs not something which YARN makes it easy to do, but it may be
>> possible, especially if Spark itself provides some hooks, so you don't have
>> to manually lay with setting things up. That may be something which could
>> significantly ease adoption of Spark 2 in YARN clusters. Same for Python.
>>
>> This is something I could probably help others to address
>>
>>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Jakob Odersky]

Reynold's 3rd point is particularly strong in my opinion. Supporting
Scala 2.12 will require Java 8 anyway, and introducing such a change
is probably best done in a major release.
Consider what would happen if Spark 2.0 doesn't require Java 8 and
hence not support Scala 2.12. Will it be stuck on an older version
until 3.0 is out? Will it be introduced in a minor release?
I think 2.0 is the best time for such a change.

On Thu, Mar 24, 2016 at 11:46 AM, Stephen Boesch  wrote:
> +1 for java8 only   +1 for 2.11+ only .At this point scala libraries
> supporting only 2.10 are typically less active and/or poorly maintained.
> That trend will only continue when considering the lifespan of spark 2.X.
>
> 2016-03-24 11:32 GMT-07:00 Steve Loughran :
>>
>>
>> On 24 Mar 2016, at 15:27, Koert Kuipers  wrote:
>>
>> i think the arguments are convincing, but it also makes me wonder if i
>> live in some kind of alternate universe... we deploy on customers clusters,
>> where the OS, python version, java version and hadoop distro are not chosen
>> by us. so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply
>> have access to a single proxy machine and launch through yarn. asking them
>> to upgrade java is pretty much out of the question or a 6+ month ordeal. of
>> the 10 client clusters i can think of on the top of my head all of them are
>> on java 7, none are on java 8. so by doing this you would make spark 2
>> basically unusable for us (unless most of them have plans of upgrading in
>> near term to java 8, i will ask around and report back...).
>>
>>
>>
>> It's not actually mandatory for the process executing in the Yarn cluster
>> to run with the same JVM as the rest of the Hadoop stack; all that is needed
>> is for the environment variables to set up the JAVA_HOME and PATH. Switching
>> JVMs not something which YARN makes it easy to do, but it may be possible,
>> especially if Spark itself provides some hooks, so you don't have to
>> manually lay with setting things up. That may be something which could
>> significantly ease adoption of Spark 2 in YARN clusters. Same for Python.
>>
>> This is something I could probably help others to address
>>
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Andrew Ash]

Spark 2.x has to be the time for Java 8.

I'd rather increase JVM major version on a Spark major version than on a
Spark minor version, and I'd rather Spark do that upgrade for the 2.x
series than the 3.x series (~2yr from now based on the lifetime of Spark
1.x).  If we wait until the next opportunity for a breaking change to Spark
(3.x) we might be upgrading to Java 9 at that point rather than Java 8.

If Spark users need Java 7 they are free to continue using the 1.x series,
the same way that folks who need Java 6 are free to continue using 1.4

On Thu, Mar 24, 2016 at 11:46 AM, Stephen Boesch  wrote:

> +1 for java8 only   +1 for 2.11+ only .At this point scala libraries
> supporting only 2.10 are typically less active and/or poorly maintained.
> That trend will only continue when considering the lifespan of spark 2.X.
>
> 2016-03-24 11:32 GMT-07:00 Steve Loughran :
>
>>
>> On 24 Mar 2016, at 15:27, Koert Kuipers  wrote:
>>
>> i think the arguments are convincing, but it also makes me wonder if i
>> live in some kind of alternate universe... we deploy on customers clusters,
>> where the OS, python version, java version and hadoop distro are not chosen
>> by us. so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply
>> have access to a single proxy machine and launch through yarn. asking them
>> to upgrade java is pretty much out of the question or a 6+ month ordeal. of
>> the 10 client clusters i can think of on the top of my head all of them are
>> on java 7, none are on java 8. so by doing this you would make spark 2
>> basically unusable for us (unless most of them have plans of upgrading in
>> near term to java 8, i will ask around and report back...).
>>
>>
>>
>> It's not actually mandatory for the process executing in the Yarn cluster
>> to run with the same JVM as the rest of the Hadoop stack; all that is
>> needed is for the environment variables to set up the JAVA_HOME and PATH.
>> Switching JVMs not something which YARN makes it easy to do, but it may be
>> possible, especially if Spark itself provides some hooks, so you don't have
>> to manually lay with setting things up. That may be something which could
>> significantly ease adoption of Spark 2 in YARN clusters. Same for Python.
>>
>> This is something I could probably help others to address
>>
>>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Stephen Boesch]

+1 for java8 only   +1 for 2.11+ only .At this point scala libraries
supporting only 2.10 are typically less active and/or poorly maintained.
That trend will only continue when considering the lifespan of spark 2.X.

2016-03-24 11:32 GMT-07:00 Steve Loughran :

>
> On 24 Mar 2016, at 15:27, Koert Kuipers  wrote:
>
> i think the arguments are convincing, but it also makes me wonder if i
> live in some kind of alternate universe... we deploy on customers clusters,
> where the OS, python version, java version and hadoop distro are not chosen
> by us. so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply
> have access to a single proxy machine and launch through yarn. asking them
> to upgrade java is pretty much out of the question or a 6+ month ordeal. of
> the 10 client clusters i can think of on the top of my head all of them are
> on java 7, none are on java 8. so by doing this you would make spark 2
> basically unusable for us (unless most of them have plans of upgrading in
> near term to java 8, i will ask around and report back...).
>
>
>
> It's not actually mandatory for the process executing in the Yarn cluster
> to run with the same JVM as the rest of the Hadoop stack; all that is
> needed is for the environment variables to set up the JAVA_HOME and PATH.
> Switching JVMs not something which YARN makes it easy to do, but it may be
> possible, especially if Spark itself provides some hooks, so you don't have
> to manually lay with setting things up. That may be something which could
> significantly ease adoption of Spark 2 in YARN clusters. Same for Python.
>
> This is something I could probably help others to address
>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Steve Loughran]

On 24 Mar 2016, at 15:27, Koert Kuipers 
> wrote:

i think the arguments are convincing, but it also makes me wonder if i live in 
some kind of alternate universe... we deploy on customers clusters, where the 
OS, python version, java version and hadoop distro are not chosen by us. so 
think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply have access 
to a single proxy machine and launch through yarn. asking them to upgrade java 
is pretty much out of the question or a 6+ month ordeal. of the 10 client 
clusters i can think of on the top of my head all of them are on java 7, none 
are on java 8. so by doing this you would make spark 2 basically unusable for 
us (unless most of them have plans of upgrading in near term to java 8, i will 
ask around and report back...).


It's not actually mandatory for the process executing in the Yarn cluster to 
run with the same JVM as the rest of the Hadoop stack; all that is needed is 
for the environment variables to set up the JAVA_HOME and PATH. Switching JVMs 
not something which YARN makes it easy to do, but it may be possible, 
especially if Spark itself provides some hooks, so you don't have to manually 
lay with setting things up. That may be something which could significantly 
ease adoption of Spark 2 in YARN clusters. Same for Python.

This is something I could probably help others to address

Re: [discuss] ending support for Java 7 in Spark 2.0

[Marcelo Vanzin]

On Thu, Mar 24, 2016 at 10:13 AM, Reynold Xin  wrote:
> Yes

So is it safe to say the only hard requirements for Java 8 in your list is (4)?

(1) and (3) are infrastructure issues. Yes, it sucks to maintain more
testing infrastructure and potentially more complicated build scripts,
but does that really outweigh maintaining support for Java 7?

A cheap hack would also be to require jdk 1.8 for the build, but still
target java 7. You could then isolate java 8 tests in a separate
module that will get run in all builds because of that requirement.
There are downsides, of course: it's basically the same situation we
were in when we still supported Java 6 but were using jdk 1.7 to build
things. Setting the proper bootclasspath to use jdk 7's rt.jar during
compilation could solve a lot of those. (We already have both JDKs in
jenkins machines as far as I can tell.)

For Scala 2.12, and option might be dropping Java 7 when we decide to
add support for that (unless you're also suggesting Scala 2.12 as part
of 2.0?).

For (2) it seems the jvm used to compile things doesn't really make a
difference. It could be as simple as "we strongly recommend running
Spark 2.0 on Java 8".

Note I'm not for or against the change per se; I'd like to see more
data about what users are really using out there before making that
decision. But there was an explicit desire to maintain java 7
compatibility when we talked about going for Spark 2.0. And with those
kinds of decisions there's always a cost, including spending more
resources on infrastructure and testing.

-- 
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Reynold Xin]

Yes

On Thursday, March 24, 2016, Marcelo Vanzin  wrote:

> On Thu, Mar 24, 2016 at 1:04 AM, Reynold Xin  > wrote:
> > I actually talked quite a bit today with an engineer on the scala
> compiler
> > team tonight and the scala 2.10 + java 8 combo should be ok. The latest
> > Scala 2.10 release should have all the important fixes that are needed
> for
> > Java 8.
>
> So, do you actually get the benefits you're looking for without
> compiling explicitly to the 1.8 jvm? Because:
>
> $ scala -version
> Scala code runner version 2.10.6 -- Copyright 2002-2013, LAMP/EPFL
> $ scalac -target jvm-1.8
> scalac error: Usage: -target:
>  where  choices are jvm-1.5, jvm-1.5-fjbg, jvm-1.5-asm,
> jvm-1.6, jvm-1.7, msil (default: jvm-1.6)
>
> So even if you use jdk 8 to compile with scala 2.10, you can't target
> jvm 1.8 as far as I can tell.
>
> --
> Marcelo
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Marcelo Vanzin]

On Thu, Mar 24, 2016 at 9:54 AM, Koert Kuipers  wrote:
> i guess what i am saying is that in a yarn world the only hard restrictions
> left are the the containers you run in, which means the hadoop version, java
> version and python version (if you use python).

It is theoretically possible to run containers with a different JDK
than the NM (I've done it for testing), although I'm not sure about
whether that's recommended from YARN's perspective.

But I understand your concern is that you're not allowed to modify the
machines where the NMs are hosted. You could hack things and
distribute the JVM with your Spark application, but that would be
incredibly ugly.

-- 
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Marcelo Vanzin]

On Thu, Mar 24, 2016 at 1:04 AM, Reynold Xin  wrote:
> I actually talked quite a bit today with an engineer on the scala compiler
> team tonight and the scala 2.10 + java 8 combo should be ok. The latest
> Scala 2.10 release should have all the important fixes that are needed for
> Java 8.

So, do you actually get the benefits you're looking for without
compiling explicitly to the 1.8 jvm? Because:

$ scala -version
Scala code runner version 2.10.6 -- Copyright 2002-2013, LAMP/EPFL
$ scalac -target jvm-1.8
scalac error: Usage: -target:
 where  choices are jvm-1.5, jvm-1.5-fjbg, jvm-1.5-asm,
jvm-1.6, jvm-1.7, msil (default: jvm-1.6)

So even if you use jdk 8 to compile with scala 2.10, you can't target
jvm 1.8 as far as I can tell.

-- 
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Koert Kuipers]

i guess what i am saying is that in a yarn world the only hard restrictions
left are the the containers you run in, which means the hadoop version,
java version and python version (if you use python).


On Thu, Mar 24, 2016 at 12:39 PM, Koert Kuipers  wrote:

> The group will not upgrade to spark 2.0 themselves, but they are mostly
> fine with vendors like us deploying our application via yarn with whatever
> spark version we choose (and bundle, so they do not install it separately,
> they might not even be aware of what spark version we use). This all works
> because spark does not need to be on the cluster nodes, just on the one
> machine where our application gets launched. Having yarn is pretty awesome
> in this respect.
>
> On Thu, Mar 24, 2016 at 12:25 PM, Sean Owen  wrote:
>
>> (PS CDH5 runs fine with Java 8, but I understand your more general point.)
>>
>> This is a familiar context indeed, but in that context, would a group
>> not wanting to update to Java 8 want to manually put Spark 2.0 into
>> the mix? That is, if this is a context where the cluster is
>> purposefully some stable mix of components, would you be updating just
>> one?
>>
>> You make a good point about Scala being more library than
>> infrastructure component. So it can be updated on a per-app basis. On
>> the one hand it's harder to handle different Scala versions from the
>> framework side, it's less hard on the deployment side.
>>
>> On Thu, Mar 24, 2016 at 4:27 PM, Koert Kuipers  wrote:
>> > i think the arguments are convincing, but it also makes me wonder if i
>> live
>> > in some kind of alternate universe... we deploy on customers clusters,
>> where
>> > the OS, python version, java version and hadoop distro are not chosen
>> by us.
>> > so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply
>> have
>> > access to a single proxy machine and launch through yarn. asking them to
>> > upgrade java is pretty much out of the question or a 6+ month ordeal.
>> of the
>> > 10 client clusters i can think of on the top of my head all of them are
>> on
>> > java 7, none are on java 8. so by doing this you would make spark 2
>> > basically unusable for us (unless most of them have plans of upgrading
>> in
>> > near term to java 8, i will ask around and report back...).
>> >
>> > on a side note, its particularly interesting to me that spark 2 chose to
>> > continue support for scala 2.10, because even for us in our very
>> constricted
>> > client environments the scala version is something we can easily
>> upgrade (we
>> > just deploy a custom build of spark for the relevant scala version and
>> > hadoop distro). and because scala is not a dependency of any hadoop
>> distro
>> > (so not on classpath, which i am very happy about) we can use whatever
>> scala
>> > version we like. also i found the upgrade path from scala 2.10 to 2.11
>> to be
>> > very easy, so i have a hard time understanding why anyone would stay on
>> > scala 2.10. and finally with scala 2.12 around the corner you really
>> dont
>> > want to be supporting 3 versions. so clearly i am missing something
>> here.
>> >
>> >
>> >
>> > On Thu, Mar 24, 2016 at 8:52 AM, Jean-Baptiste Onofré 
>> > wrote:
>> >>
>> >> +1 to support Java 8 (and future) *only* in Spark 2.0, and end support
>> of
>> >> Java 7. It makes sense.
>> >>
>> >> Regards
>> >> JB
>> >>
>> >>
>> >> On 03/24/2016 08:27 AM, Reynold Xin wrote:
>> >>>
>> >>> About a year ago we decided to drop Java 6 support in Spark 1.5. I am
>> >>> wondering if we should also just drop Java 7 support in Spark 2.0
>> (i.e.
>> >>> Spark 2.0 would require Java 8 to run).
>> >>>
>> >>> Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
>> >>> removed public downloads for JDK 7 in July 2015. In the past I've
>> >>> actually been against dropping Java 8, but today I ran into an issue
>> >>> with the new Dataset API not working well with Java 8 lambdas, and
>> that
>> >>> changed my opinion on this.
>> >>>
>> >>> I've been thinking more about this issue today and also talked with a
>> >>> lot people offline to gather feedback, and I actually think the pros
>> >>> outweighs the cons, for the following reasons (in some rough order of
>> >>> importance):
>> >>>
>> >>> 1. It is complicated to test how well Spark APIs work for Java lambdas
>> >>> if we support Java 7. Jenkins machines need to have both Java 7 and
>> Java
>> >>> 8 installed and we must run through a set of test suites in 7, and
>> then
>> >>> the lambda tests in Java 8. This complicates build
>> environments/scripts,
>> >>> and makes them less robust. Without good testing infrastructure, I
>> have
>> >>> no confidence in building good APIs for Java 8.
>> >>>
>> >>> 2. Dataset/DataFrame performance will be between 1x to 10x slower in
>> >>> Java 7. The primary APIs we want users to use in Spark 2.x are
>> >>> Dataset/DataFrame, and this impacts pretty much everything 

Re: [discuss] ending support for Java 7 in Spark 2.0

[Al Pivonka]

Thank you for the context Jean...
I appreciate it...


On Thu, Mar 24, 2016 at 12:40 PM, Jean-Baptiste Onofré 
wrote:

> Hi Al,
>
> Spark 2.0 doesn't mean Spark 1.x will stop. Clearly, new features will go
> on Spark 2.0, but maintenance release can be performed on 1.x branch.
>
> Regards
> JB
>
> On 03/24/2016 05:38 PM, Al Pivonka wrote:
>
>> As an end user (developer) and Cluster Admin.
>> I would have to agree with Koet.
>>
>> To me the real question is timing,  current version is 1.6.1, the
>> question I have is how many more releases till 2.0 and what is the time
>> frame?
>>
>> If you give people six to twelve months to plan and make sure they know
>> (paste it all over the web site) most can plan ahead.
>>
>>
>> Just my two pennies
>>
>>
>>
>>
>>
>> On Thu, Mar 24, 2016 at 12:25 PM, Sean Owen > > wrote:
>>
>> (PS CDH5 runs fine with Java 8, but I understand your more general
>> point.)
>>
>> This is a familiar context indeed, but in that context, would a group
>> not wanting to update to Java 8 want to manually put Spark 2.0 into
>> the mix? That is, if this is a context where the cluster is
>> purposefully some stable mix of components, would you be updating just
>> one?
>>
>> You make a good point about Scala being more library than
>> infrastructure component. So it can be updated on a per-app basis. On
>> the one hand it's harder to handle different Scala versions from the
>> framework side, it's less hard on the deployment side.
>>
>> On Thu, Mar 24, 2016 at 4:27 PM, Koert Kuipers > > wrote:
>>  > i think the arguments are convincing, but it also makes me wonder
>> if i live
>>  > in some kind of alternate universe... we deploy on customers
>> clusters, where
>>  > the OS, python version, java version and hadoop distro are not
>> chosen by us.
>>  > so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we
>> simply have
>>  > access to a single proxy machine and launch through yarn. asking
>> them to
>>  > upgrade java is pretty much out of the question or a 6+ month
>> ordeal. of the
>>  > 10 client clusters i can think of on the top of my head all of
>> them are on
>>  > java 7, none are on java 8. so by doing this you would make spark 2
>>  > basically unusable for us (unless most of them have plans of
>> upgrading in
>>  > near term to java 8, i will ask around and report back...).
>>  >
>>  > on a side note, its particularly interesting to me that spark 2
>> chose to
>>  > continue support for scala 2.10, because even for us in our very
>> constricted
>>  > client environments the scala version is something we can easily
>> upgrade (we
>>  > just deploy a custom build of spark for the relevant scala
>> version and
>>  > hadoop distro). and because scala is not a dependency of any
>> hadoop distro
>>  > (so not on classpath, which i am very happy about) we can use
>> whatever scala
>>  > version we like. also i found the upgrade path from scala 2.10 to
>> 2.11 to be
>>  > very easy, so i have a hard time understanding why anyone would
>> stay on
>>  > scala 2.10. and finally with scala 2.12 around the corner you
>> really dont
>>  > want to be supporting 3 versions. so clearly i am missing
>> something here.
>>  >
>>  >
>>  >
>>  > On Thu, Mar 24, 2016 at 8:52 AM, Jean-Baptiste Onofré
>> >
>>
>>  > wrote:
>>  >>
>>  >> +1 to support Java 8 (and future) *only* in Spark 2.0, and end
>> support of
>>  >> Java 7. It makes sense.
>>  >>
>>  >> Regards
>>  >> JB
>>  >>
>>  >>
>>  >> On 03/24/2016 08:27 AM, Reynold Xin wrote:
>>  >>>
>>  >>> About a year ago we decided to drop Java 6 support in Spark
>> 1.5. I am
>>  >>> wondering if we should also just drop Java 7 support in Spark
>> 2.0 (i.e.
>>  >>> Spark 2.0 would require Java 8 to run).
>>  >>>
>>  >>> Oracle ended public updates for JDK 7 in one year ago (Apr
>> 2015), and
>>  >>> removed public downloads for JDK 7 in July 2015. In the past I've
>>  >>> actually been against dropping Java 8, but today I ran into an
>> issue
>>  >>> with the new Dataset API not working well with Java 8 lambdas,
>> and that
>>  >>> changed my opinion on this.
>>  >>>
>>  >>> I've been thinking more about this issue today and also talked
>> with a
>>  >>> lot people offline to gather feedback, and I actually think the
>> pros
>>  >>> outweighs the cons, for the following reasons (in some rough
>> order of
>>  >>> importance):
>>  >>>
>>  >>> 1. It is complicated to test how well Spark APIs work for Java
>> lambdas
>>  >>> if we 

Re: [discuss] ending support for Java 7 in Spark 2.0

[Jean-Baptiste Onofré]

Hi Al,

Spark 2.0 doesn't mean Spark 1.x will stop. Clearly, new features will 
go on Spark 2.0, but maintenance release can be performed on 1.x branch.


Regards
JB

On 03/24/2016 05:38 PM, Al Pivonka wrote:

As an end user (developer) and Cluster Admin.
I would have to agree with Koet.

To me the real question is timing,  current version is 1.6.1, the
question I have is how many more releases till 2.0 and what is the time
frame?

If you give people six to twelve months to plan and make sure they know
(paste it all over the web site) most can plan ahead.


Just my two pennies





On Thu, Mar 24, 2016 at 12:25 PM, Sean Owen > wrote:

(PS CDH5 runs fine with Java 8, but I understand your more general
point.)

This is a familiar context indeed, but in that context, would a group
not wanting to update to Java 8 want to manually put Spark 2.0 into
the mix? That is, if this is a context where the cluster is
purposefully some stable mix of components, would you be updating just
one?

You make a good point about Scala being more library than
infrastructure component. So it can be updated on a per-app basis. On
the one hand it's harder to handle different Scala versions from the
framework side, it's less hard on the deployment side.

On Thu, Mar 24, 2016 at 4:27 PM, Koert Kuipers > wrote:
 > i think the arguments are convincing, but it also makes me wonder
if i live
 > in some kind of alternate universe... we deploy on customers
clusters, where
 > the OS, python version, java version and hadoop distro are not
chosen by us.
 > so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we
simply have
 > access to a single proxy machine and launch through yarn. asking
them to
 > upgrade java is pretty much out of the question or a 6+ month
ordeal. of the
 > 10 client clusters i can think of on the top of my head all of
them are on
 > java 7, none are on java 8. so by doing this you would make spark 2
 > basically unusable for us (unless most of them have plans of
upgrading in
 > near term to java 8, i will ask around and report back...).
 >
 > on a side note, its particularly interesting to me that spark 2
chose to
 > continue support for scala 2.10, because even for us in our very
constricted
 > client environments the scala version is something we can easily
upgrade (we
 > just deploy a custom build of spark for the relevant scala
version and
 > hadoop distro). and because scala is not a dependency of any
hadoop distro
 > (so not on classpath, which i am very happy about) we can use
whatever scala
 > version we like. also i found the upgrade path from scala 2.10 to
2.11 to be
 > very easy, so i have a hard time understanding why anyone would
stay on
 > scala 2.10. and finally with scala 2.12 around the corner you
really dont
 > want to be supporting 3 versions. so clearly i am missing
something here.
 >
 >
 >
 > On Thu, Mar 24, 2016 at 8:52 AM, Jean-Baptiste Onofré
>
 > wrote:
 >>
 >> +1 to support Java 8 (and future) *only* in Spark 2.0, and end
support of
 >> Java 7. It makes sense.
 >>
 >> Regards
 >> JB
 >>
 >>
 >> On 03/24/2016 08:27 AM, Reynold Xin wrote:
 >>>
 >>> About a year ago we decided to drop Java 6 support in Spark
1.5. I am
 >>> wondering if we should also just drop Java 7 support in Spark
2.0 (i.e.
 >>> Spark 2.0 would require Java 8 to run).
 >>>
 >>> Oracle ended public updates for JDK 7 in one year ago (Apr
2015), and
 >>> removed public downloads for JDK 7 in July 2015. In the past I've
 >>> actually been against dropping Java 8, but today I ran into an
issue
 >>> with the new Dataset API not working well with Java 8 lambdas,
and that
 >>> changed my opinion on this.
 >>>
 >>> I've been thinking more about this issue today and also talked
with a
 >>> lot people offline to gather feedback, and I actually think the
pros
 >>> outweighs the cons, for the following reasons (in some rough
order of
 >>> importance):
 >>>
 >>> 1. It is complicated to test how well Spark APIs work for Java
lambdas
 >>> if we support Java 7. Jenkins machines need to have both Java 7
and Java
 >>> 8 installed and we must run through a set of test suites in 7,
and then
 >>> the lambda tests in Java 8. This complicates build
environments/scripts,
 >>> and makes them less robust. Without good testing
infrastructure, I have
 >>> no confidence in building good APIs for Java 8.
 >>>
 >>> 2. Dataset/DataFrame performance will be between 1x to 10x
slower in
 >>> Java 7. 

Re: [discuss] ending support for Java 7 in Spark 2.0

[Koert Kuipers]

The group will not upgrade to spark 2.0 themselves, but they are mostly
fine with vendors like us deploying our application via yarn with whatever
spark version we choose (and bundle, so they do not install it separately,
they might not even be aware of what spark version we use). This all works
because spark does not need to be on the cluster nodes, just on the one
machine where our application gets launched. Having yarn is pretty awesome
in this respect.

On Thu, Mar 24, 2016 at 12:25 PM, Sean Owen  wrote:

> (PS CDH5 runs fine with Java 8, but I understand your more general point.)
>
> This is a familiar context indeed, but in that context, would a group
> not wanting to update to Java 8 want to manually put Spark 2.0 into
> the mix? That is, if this is a context where the cluster is
> purposefully some stable mix of components, would you be updating just
> one?
>
> You make a good point about Scala being more library than
> infrastructure component. So it can be updated on a per-app basis. On
> the one hand it's harder to handle different Scala versions from the
> framework side, it's less hard on the deployment side.
>
> On Thu, Mar 24, 2016 at 4:27 PM, Koert Kuipers  wrote:
> > i think the arguments are convincing, but it also makes me wonder if i
> live
> > in some kind of alternate universe... we deploy on customers clusters,
> where
> > the OS, python version, java version and hadoop distro are not chosen by
> us.
> > so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply have
> > access to a single proxy machine and launch through yarn. asking them to
> > upgrade java is pretty much out of the question or a 6+ month ordeal. of
> the
> > 10 client clusters i can think of on the top of my head all of them are
> on
> > java 7, none are on java 8. so by doing this you would make spark 2
> > basically unusable for us (unless most of them have plans of upgrading in
> > near term to java 8, i will ask around and report back...).
> >
> > on a side note, its particularly interesting to me that spark 2 chose to
> > continue support for scala 2.10, because even for us in our very
> constricted
> > client environments the scala version is something we can easily upgrade
> (we
> > just deploy a custom build of spark for the relevant scala version and
> > hadoop distro). and because scala is not a dependency of any hadoop
> distro
> > (so not on classpath, which i am very happy about) we can use whatever
> scala
> > version we like. also i found the upgrade path from scala 2.10 to 2.11
> to be
> > very easy, so i have a hard time understanding why anyone would stay on
> > scala 2.10. and finally with scala 2.12 around the corner you really dont
> > want to be supporting 3 versions. so clearly i am missing something here.
> >
> >
> >
> > On Thu, Mar 24, 2016 at 8:52 AM, Jean-Baptiste Onofré 
> > wrote:
> >>
> >> +1 to support Java 8 (and future) *only* in Spark 2.0, and end support
> of
> >> Java 7. It makes sense.
> >>
> >> Regards
> >> JB
> >>
> >>
> >> On 03/24/2016 08:27 AM, Reynold Xin wrote:
> >>>
> >>> About a year ago we decided to drop Java 6 support in Spark 1.5. I am
> >>> wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.
> >>> Spark 2.0 would require Java 8 to run).
> >>>
> >>> Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
> >>> removed public downloads for JDK 7 in July 2015. In the past I've
> >>> actually been against dropping Java 8, but today I ran into an issue
> >>> with the new Dataset API not working well with Java 8 lambdas, and that
> >>> changed my opinion on this.
> >>>
> >>> I've been thinking more about this issue today and also talked with a
> >>> lot people offline to gather feedback, and I actually think the pros
> >>> outweighs the cons, for the following reasons (in some rough order of
> >>> importance):
> >>>
> >>> 1. It is complicated to test how well Spark APIs work for Java lambdas
> >>> if we support Java 7. Jenkins machines need to have both Java 7 and
> Java
> >>> 8 installed and we must run through a set of test suites in 7, and then
> >>> the lambda tests in Java 8. This complicates build
> environments/scripts,
> >>> and makes them less robust. Without good testing infrastructure, I have
> >>> no confidence in building good APIs for Java 8.
> >>>
> >>> 2. Dataset/DataFrame performance will be between 1x to 10x slower in
> >>> Java 7. The primary APIs we want users to use in Spark 2.x are
> >>> Dataset/DataFrame, and this impacts pretty much everything from machine
> >>> learning to structured streaming. We have made great progress in their
> >>> performance through extensive use of code generation. (In many
> >>> dimensions Spark 2.0 with DataFrames/Datasets looks more like a
> compiler
> >>> than a MapReduce or query engine.) These optimizations don't work well
> >>> in Java 7 due to broken code cache flushing. This problem has been
> fixed
> >>> by 

Re: [discuss] ending support for Java 7 in Spark 2.0

[Al Pivonka]

As an end user (developer) and Cluster Admin.
I would have to agree with Koet.

To me the real question is timing,  current version is 1.6.1, the question
I have is how many more releases till 2.0 and what is the time frame?

If you give people six to twelve months to plan and make sure they know
(paste it all over the web site) most can plan ahead.


Just my two pennies





On Thu, Mar 24, 2016 at 12:25 PM, Sean Owen  wrote:

> (PS CDH5 runs fine with Java 8, but I understand your more general point.)
>
> This is a familiar context indeed, but in that context, would a group
> not wanting to update to Java 8 want to manually put Spark 2.0 into
> the mix? That is, if this is a context where the cluster is
> purposefully some stable mix of components, would you be updating just
> one?
>
> You make a good point about Scala being more library than
> infrastructure component. So it can be updated on a per-app basis. On
> the one hand it's harder to handle different Scala versions from the
> framework side, it's less hard on the deployment side.
>
> On Thu, Mar 24, 2016 at 4:27 PM, Koert Kuipers  wrote:
> > i think the arguments are convincing, but it also makes me wonder if i
> live
> > in some kind of alternate universe... we deploy on customers clusters,
> where
> > the OS, python version, java version and hadoop distro are not chosen by
> us.
> > so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply have
> > access to a single proxy machine and launch through yarn. asking them to
> > upgrade java is pretty much out of the question or a 6+ month ordeal. of
> the
> > 10 client clusters i can think of on the top of my head all of them are
> on
> > java 7, none are on java 8. so by doing this you would make spark 2
> > basically unusable for us (unless most of them have plans of upgrading in
> > near term to java 8, i will ask around and report back...).
> >
> > on a side note, its particularly interesting to me that spark 2 chose to
> > continue support for scala 2.10, because even for us in our very
> constricted
> > client environments the scala version is something we can easily upgrade
> (we
> > just deploy a custom build of spark for the relevant scala version and
> > hadoop distro). and because scala is not a dependency of any hadoop
> distro
> > (so not on classpath, which i am very happy about) we can use whatever
> scala
> > version we like. also i found the upgrade path from scala 2.10 to 2.11
> to be
> > very easy, so i have a hard time understanding why anyone would stay on
> > scala 2.10. and finally with scala 2.12 around the corner you really dont
> > want to be supporting 3 versions. so clearly i am missing something here.
> >
> >
> >
> > On Thu, Mar 24, 2016 at 8:52 AM, Jean-Baptiste Onofré 
> > wrote:
> >>
> >> +1 to support Java 8 (and future) *only* in Spark 2.0, and end support
> of
> >> Java 7. It makes sense.
> >>
> >> Regards
> >> JB
> >>
> >>
> >> On 03/24/2016 08:27 AM, Reynold Xin wrote:
> >>>
> >>> About a year ago we decided to drop Java 6 support in Spark 1.5. I am
> >>> wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.
> >>> Spark 2.0 would require Java 8 to run).
> >>>
> >>> Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
> >>> removed public downloads for JDK 7 in July 2015. In the past I've
> >>> actually been against dropping Java 8, but today I ran into an issue
> >>> with the new Dataset API not working well with Java 8 lambdas, and that
> >>> changed my opinion on this.
> >>>
> >>> I've been thinking more about this issue today and also talked with a
> >>> lot people offline to gather feedback, and I actually think the pros
> >>> outweighs the cons, for the following reasons (in some rough order of
> >>> importance):
> >>>
> >>> 1. It is complicated to test how well Spark APIs work for Java lambdas
> >>> if we support Java 7. Jenkins machines need to have both Java 7 and
> Java
> >>> 8 installed and we must run through a set of test suites in 7, and then
> >>> the lambda tests in Java 8. This complicates build
> environments/scripts,
> >>> and makes them less robust. Without good testing infrastructure, I have
> >>> no confidence in building good APIs for Java 8.
> >>>
> >>> 2. Dataset/DataFrame performance will be between 1x to 10x slower in
> >>> Java 7. The primary APIs we want users to use in Spark 2.x are
> >>> Dataset/DataFrame, and this impacts pretty much everything from machine
> >>> learning to structured streaming. We have made great progress in their
> >>> performance through extensive use of code generation. (In many
> >>> dimensions Spark 2.0 with DataFrames/Datasets looks more like a
> compiler
> >>> than a MapReduce or query engine.) These optimizations don't work well
> >>> in Java 7 due to broken code cache flushing. This problem has been
> fixed
> >>> by Oracle in Java 8. In addition, Java 8 comes with better support for
> >>> Unsafe and 

Re: [discuss] ending support for Java 7 in Spark 2.0

[Sean Owen]

(PS CDH5 runs fine with Java 8, but I understand your more general point.)

This is a familiar context indeed, but in that context, would a group
not wanting to update to Java 8 want to manually put Spark 2.0 into
the mix? That is, if this is a context where the cluster is
purposefully some stable mix of components, would you be updating just
one?

You make a good point about Scala being more library than
infrastructure component. So it can be updated on a per-app basis. On
the one hand it's harder to handle different Scala versions from the
framework side, it's less hard on the deployment side.

On Thu, Mar 24, 2016 at 4:27 PM, Koert Kuipers  wrote:
> i think the arguments are convincing, but it also makes me wonder if i live
> in some kind of alternate universe... we deploy on customers clusters, where
> the OS, python version, java version and hadoop distro are not chosen by us.
> so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply have
> access to a single proxy machine and launch through yarn. asking them to
> upgrade java is pretty much out of the question or a 6+ month ordeal. of the
> 10 client clusters i can think of on the top of my head all of them are on
> java 7, none are on java 8. so by doing this you would make spark 2
> basically unusable for us (unless most of them have plans of upgrading in
> near term to java 8, i will ask around and report back...).
>
> on a side note, its particularly interesting to me that spark 2 chose to
> continue support for scala 2.10, because even for us in our very constricted
> client environments the scala version is something we can easily upgrade (we
> just deploy a custom build of spark for the relevant scala version and
> hadoop distro). and because scala is not a dependency of any hadoop distro
> (so not on classpath, which i am very happy about) we can use whatever scala
> version we like. also i found the upgrade path from scala 2.10 to 2.11 to be
> very easy, so i have a hard time understanding why anyone would stay on
> scala 2.10. and finally with scala 2.12 around the corner you really dont
> want to be supporting 3 versions. so clearly i am missing something here.
>
>
>
> On Thu, Mar 24, 2016 at 8:52 AM, Jean-Baptiste Onofré 
> wrote:
>>
>> +1 to support Java 8 (and future) *only* in Spark 2.0, and end support of
>> Java 7. It makes sense.
>>
>> Regards
>> JB
>>
>>
>> On 03/24/2016 08:27 AM, Reynold Xin wrote:
>>>
>>> About a year ago we decided to drop Java 6 support in Spark 1.5. I am
>>> wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.
>>> Spark 2.0 would require Java 8 to run).
>>>
>>> Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
>>> removed public downloads for JDK 7 in July 2015. In the past I've
>>> actually been against dropping Java 8, but today I ran into an issue
>>> with the new Dataset API not working well with Java 8 lambdas, and that
>>> changed my opinion on this.
>>>
>>> I've been thinking more about this issue today and also talked with a
>>> lot people offline to gather feedback, and I actually think the pros
>>> outweighs the cons, for the following reasons (in some rough order of
>>> importance):
>>>
>>> 1. It is complicated to test how well Spark APIs work for Java lambdas
>>> if we support Java 7. Jenkins machines need to have both Java 7 and Java
>>> 8 installed and we must run through a set of test suites in 7, and then
>>> the lambda tests in Java 8. This complicates build environments/scripts,
>>> and makes them less robust. Without good testing infrastructure, I have
>>> no confidence in building good APIs for Java 8.
>>>
>>> 2. Dataset/DataFrame performance will be between 1x to 10x slower in
>>> Java 7. The primary APIs we want users to use in Spark 2.x are
>>> Dataset/DataFrame, and this impacts pretty much everything from machine
>>> learning to structured streaming. We have made great progress in their
>>> performance through extensive use of code generation. (In many
>>> dimensions Spark 2.0 with DataFrames/Datasets looks more like a compiler
>>> than a MapReduce or query engine.) These optimizations don't work well
>>> in Java 7 due to broken code cache flushing. This problem has been fixed
>>> by Oracle in Java 8. In addition, Java 8 comes with better support for
>>> Unsafe and SIMD.
>>>
>>> 3. Scala 2.12 will come out soon, and we will want to add support for
>>> that. Scala 2.12 only works on Java 8. If we do support Java 7, we'd
>>> have a fairly complicated compatibility matrix and testing
>>> infrastructure.
>>>
>>> 4. There are libraries that I've looked into in the past that support
>>> only Java 8. This is more common in high performance libraries such as
>>> Aeron (a messaging library). Having to support Java 7 means we are not
>>> able to use these. It is not that big of a deal right now, but will
>>> become increasingly more difficult as we optimize performance.
>>>
>>>
>>> The downside of not 

Re: [discuss] ending support for Java 7 in Spark 2.0

[Koert Kuipers]

i think the arguments are convincing, but it also makes me wonder if i live
in some kind of alternate universe... we deploy on customers clusters,
where the OS, python version, java version and hadoop distro are not chosen
by us. so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply
have access to a single proxy machine and launch through yarn. asking them
to upgrade java is pretty much out of the question or a 6+ month ordeal. of
the 10 client clusters i can think of on the top of my head all of them are
on java 7, none are on java 8. so by doing this you would make spark 2
basically unusable for us (unless most of them have plans of upgrading in
near term to java 8, i will ask around and report back...).

on a side note, its particularly interesting to me that spark 2 chose to
continue support for scala 2.10, because even for us in our very
constricted client environments the scala version is something we can
easily upgrade (we just deploy a custom build of spark for the relevant
scala version and hadoop distro). and because scala is not a dependency of
any hadoop distro (so not on classpath, which i am very happy about) we can
use whatever scala version we like. also i found the upgrade path from
scala 2.10 to 2.11 to be very easy, so i have a hard time understanding why
anyone would stay on scala 2.10. and finally with scala 2.12 around the
corner you really dont want to be supporting 3 versions. so clearly i am
missing something here.



On Thu, Mar 24, 2016 at 8:52 AM, Jean-Baptiste Onofré 
wrote:

> +1 to support Java 8 (and future) *only* in Spark 2.0, and end support of
> Java 7. It makes sense.
>
> Regards
> JB
>
>
> On 03/24/2016 08:27 AM, Reynold Xin wrote:
>
>> About a year ago we decided to drop Java 6 support in Spark 1.5. I am
>> wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.
>> Spark 2.0 would require Java 8 to run).
>>
>> Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
>> removed public downloads for JDK 7 in July 2015. In the past I've
>> actually been against dropping Java 8, but today I ran into an issue
>> with the new Dataset API not working well with Java 8 lambdas, and that
>> changed my opinion on this.
>>
>> I've been thinking more about this issue today and also talked with a
>> lot people offline to gather feedback, and I actually think the pros
>> outweighs the cons, for the following reasons (in some rough order of
>> importance):
>>
>> 1. It is complicated to test how well Spark APIs work for Java lambdas
>> if we support Java 7. Jenkins machines need to have both Java 7 and Java
>> 8 installed and we must run through a set of test suites in 7, and then
>> the lambda tests in Java 8. This complicates build environments/scripts,
>> and makes them less robust. Without good testing infrastructure, I have
>> no confidence in building good APIs for Java 8.
>>
>> 2. Dataset/DataFrame performance will be between 1x to 10x slower in
>> Java 7. The primary APIs we want users to use in Spark 2.x are
>> Dataset/DataFrame, and this impacts pretty much everything from machine
>> learning to structured streaming. We have made great progress in their
>> performance through extensive use of code generation. (In many
>> dimensions Spark 2.0 with DataFrames/Datasets looks more like a compiler
>> than a MapReduce or query engine.) These optimizations don't work well
>> in Java 7 due to broken code cache flushing. This problem has been fixed
>> by Oracle in Java 8. In addition, Java 8 comes with better support for
>> Unsafe and SIMD.
>>
>> 3. Scala 2.12 will come out soon, and we will want to add support for
>> that. Scala 2.12 only works on Java 8. If we do support Java 7, we'd
>> have a fairly complicated compatibility matrix and testing infrastructure.
>>
>> 4. There are libraries that I've looked into in the past that support
>> only Java 8. This is more common in high performance libraries such as
>> Aeron (a messaging library). Having to support Java 7 means we are not
>> able to use these. It is not that big of a deal right now, but will
>> become increasingly more difficult as we optimize performance.
>>
>>
>> The downside of not supporting Java 7 is also obvious. Some
>> organizations are stuck with Java 7, and they wouldn't be able to use
>> Spark 2.0 without upgrading Java.
>>
>>
>>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Jean-Baptiste Onofré]

+1 to support Java 8 (and future) *only* in Spark 2.0, and end support 
of Java 7. It makes sense.


Regards
JB

On 03/24/2016 08:27 AM, Reynold Xin wrote:

About a year ago we decided to drop Java 6 support in Spark 1.5. I am
wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.
Spark 2.0 would require Java 8 to run).

Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
removed public downloads for JDK 7 in July 2015. In the past I've
actually been against dropping Java 8, but today I ran into an issue
with the new Dataset API not working well with Java 8 lambdas, and that
changed my opinion on this.

I've been thinking more about this issue today and also talked with a
lot people offline to gather feedback, and I actually think the pros
outweighs the cons, for the following reasons (in some rough order of
importance):

1. It is complicated to test how well Spark APIs work for Java lambdas
if we support Java 7. Jenkins machines need to have both Java 7 and Java
8 installed and we must run through a set of test suites in 7, and then
the lambda tests in Java 8. This complicates build environments/scripts,
and makes them less robust. Without good testing infrastructure, I have
no confidence in building good APIs for Java 8.

2. Dataset/DataFrame performance will be between 1x to 10x slower in
Java 7. The primary APIs we want users to use in Spark 2.x are
Dataset/DataFrame, and this impacts pretty much everything from machine
learning to structured streaming. We have made great progress in their
performance through extensive use of code generation. (In many
dimensions Spark 2.0 with DataFrames/Datasets looks more like a compiler
than a MapReduce or query engine.) These optimizations don't work well
in Java 7 due to broken code cache flushing. This problem has been fixed
by Oracle in Java 8. In addition, Java 8 comes with better support for
Unsafe and SIMD.

3. Scala 2.12 will come out soon, and we will want to add support for
that. Scala 2.12 only works on Java 8. If we do support Java 7, we'd
have a fairly complicated compatibility matrix and testing infrastructure.

4. There are libraries that I've looked into in the past that support
only Java 8. This is more common in high performance libraries such as
Aeron (a messaging library). Having to support Java 7 means we are not
able to use these. It is not that big of a deal right now, but will
become increasingly more difficult as we optimize performance.


The downside of not supporting Java 7 is also obvious. Some
organizations are stuck with Java 7, and they wouldn't be able to use
Spark 2.0 without upgrading Java.




--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Steve Loughran]

> On 24 Mar 2016, at 07:27, Reynold Xin  wrote:
> 
> About a year ago we decided to drop Java 6 support in Spark 1.5. I am 
> wondering if we should also just drop Java 7 support in Spark 2.0 (i.e. Spark 
> 2.0 would require Java 8 to run).
> 
> Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and removed 
> public downloads for JDK 7 in July 2015.

Still there, Jan 2016 was the last public one.

> In the past I've actually been against dropping Java 8, but today I ran into 
> an issue with the new Dataset API not working well with Java 8 lambdas, and 
> that changed my opinion on this.
> 
> I've been thinking more about this issue today and also talked with a lot 
> people offline to gather feedback, and I actually think the pros outweighs 
> the cons, for the following reasons (in some rough order of importance):
> 
> 1. It is complicated to test how well Spark APIs work for Java lambdas if we 
> support Java 7. Jenkins machines need to have both Java 7 and Java 8 
> installed and we must run through a set of test suites in 7, and then the 
> lambda tests in Java 8. This complicates build environments/scripts, and 
> makes them less robust. Without good testing infrastructure, I have no 
> confidence in building good APIs for Java 8.

+complicates the test matrix for problems: if something works on java 8 and 
fails on java 7, is that a java 8 problem or a java 7 one?
+most developers would want to be on java 8 on their desktop if they could; the 
risk is that people accidentally code for java 8 even if they don't realise it 
just by using java 8 libraries, etc

> 
> 2. Dataset/DataFrame performance will be between 1x to 10x slower in Java 7. 
> The primary APIs we want users to use in Spark 2.x are Dataset/DataFrame, and 
> this impacts pretty much everything from machine learning to structured 
> streaming. We have made great progress in their performance through extensive 
> use of code generation. (In many dimensions Spark 2.0 with 
> DataFrames/Datasets looks more like a compiler than a MapReduce or query 
> engine.) These optimizations don't work well in Java 7 due to broken code 
> cache flushing. This problem has been fixed by Oracle in Java 8. In addition, 
> Java 8 comes with better support for Unsafe and SIMD.
> 
> 3. Scala 2.12 will come out soon, and we will want to add support for that. 
> Scala 2.12 only works on Java 8. If we do support Java 7, we'd have a fairly 
> complicated compatibility matrix and testing infrastructure.
> 
> 4. There are libraries that I've looked into in the past that support only 
> Java 8. This is more common in high performance libraries such as Aeron (a 
> messaging library). Having to support Java 7 means we are not able to use 
> these. It is not that big of a deal right now, but will become increasingly 
> more difficult as we optimize performance.
> 
> 
> The downside of not supporting Java 7 is also obvious. Some organizations are 
> stuck with Java 7, and they wouldn't be able to use Spark 2.0 without 
> upgrading Java.
> 


One thing you have to consider here is : will the organisations that don't want 
to upgrade to java 8 want to be upgrading to spark 2.0 anyway? 
> 

If there is a price, it means all apps that use any remote Spark APIs will also 
have to be java 8. Something like a REST API is less of an issue, but anything 
loading an JAR in the group org.apache.spark will have to be Java 8+. That's 
what held hadoop back on Java 7 in 2015 : twitter made the case that it 
shouldn't be the hadoop cluster forcing them to upgrade all their client apps 
just to use the IPC and filesystem code.I don't believe that's so much of a 
constraint on Spark.

Finally, Java 8 lines you up better for worrying about Java 9, which is on the 
horizon.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Sean Owen]

Maybe so; I think we have a ticket open to update to 2.10.6, which
maybe fixes it.

It brings up a different point: supporting multiple Scala versions is
much more painful than Java versions because of mutual
incompatibility. Right now I get the sense there's an intent to keep
supporting 2.10, and 2.11, and 2.12 later in Spark 2. This seems like
relatively way more trouble. In the same breath -- why not remove 2.10
support anyway? It's also EOL, 2.11 also brought big improvements,
etc.

On Thu, Mar 24, 2016 at 9:04 AM, Reynold Xin  wrote:
> I actually talked quite a bit today with an engineer on the scala compiler
> team tonight and the scala 2.10 + java 8 combo should be ok. The latest
> Scala 2.10 release should have all the important fixes that are needed for
> Java 8.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Reynold Xin]

I actually talked quite a bit today with an engineer on the scala compiler
team tonight and the scala 2.10 + java 8 combo should be ok. The latest
Scala 2.10 release should have all the important fixes that are needed for
Java 8.

On Thu, Mar 24, 2016 at 1:01 AM, Sean Owen  wrote:

> I generally favor this for the simplification. I didn't realize there
> were actually some performance wins and important bug fixes.
>
> I've had lots of trouble with scalac 2.10 + Java 8. I don't know if
> it's still a problem since 2.11 + 8 seems OK, but for a long time the
> sql/ modules would never compile in this config. If it's actually
> required for 2.12, makes sense.
>
> As ever my general stance is that nobody has to make a major-version
> upgrade; Spark 1.6 does not stop working for those that need Java 7. I
> also think it's reasonable for anyone to expect that major-version
> upgrades require major-version dependency updates. Also remember that
> not removing Java 7 support means committing to it here for a couple
> more years. It's not just about the situation on release day.
>
> On Thu, Mar 24, 2016 at 8:27 AM, Reynold Xin  wrote:
> > About a year ago we decided to drop Java 6 support in Spark 1.5. I am
> > wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.
> > Spark 2.0 would require Java 8 to run).
> >
> > Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
> > removed public downloads for JDK 7 in July 2015. In the past I've
> actually
> > been against dropping Java 8, but today I ran into an issue with the new
> > Dataset API not working well with Java 8 lambdas, and that changed my
> > opinion on this.
> >
> > I've been thinking more about this issue today and also talked with a lot
> > people offline to gather feedback, and I actually think the pros
> outweighs
> > the cons, for the following reasons (in some rough order of importance):
> >
> > 1. It is complicated to test how well Spark APIs work for Java lambdas
> if we
> > support Java 7. Jenkins machines need to have both Java 7 and Java 8
> > installed and we must run through a set of test suites in 7, and then the
> > lambda tests in Java 8. This complicates build environments/scripts, and
> > makes them less robust. Without good testing infrastructure, I have no
> > confidence in building good APIs for Java 8.
> >
> > 2. Dataset/DataFrame performance will be between 1x to 10x slower in
> Java 7.
> > The primary APIs we want users to use in Spark 2.x are Dataset/DataFrame,
> > and this impacts pretty much everything from machine learning to
> structured
> > streaming. We have made great progress in their performance through
> > extensive use of code generation. (In many dimensions Spark 2.0 with
> > DataFrames/Datasets looks more like a compiler than a MapReduce or query
> > engine.) These optimizations don't work well in Java 7 due to broken code
> > cache flushing. This problem has been fixed by Oracle in Java 8. In
> > addition, Java 8 comes with better support for Unsafe and SIMD.
> >
> > 3. Scala 2.12 will come out soon, and we will want to add support for
> that.
> > Scala 2.12 only works on Java 8. If we do support Java 7, we'd have a
> fairly
> > complicated compatibility matrix and testing infrastructure.
> >
> > 4. There are libraries that I've looked into in the past that support
> only
> > Java 8. This is more common in high performance libraries such as Aeron
> (a
> > messaging library). Having to support Java 7 means we are not able to use
> > these. It is not that big of a deal right now, but will become
> increasingly
> > more difficult as we optimize performance.
> >
> >
> > The downside of not supporting Java 7 is also obvious. Some organizations
> > are stuck with Java 7, and they wouldn't be able to use Spark 2.0 without
> > upgrading Java.
> >
> >
>

Re: [discuss] ending support for Java 7 in Spark 2.0

[Sean Owen]

I generally favor this for the simplification. I didn't realize there
were actually some performance wins and important bug fixes.

I've had lots of trouble with scalac 2.10 + Java 8. I don't know if
it's still a problem since 2.11 + 8 seems OK, but for a long time the
sql/ modules would never compile in this config. If it's actually
required for 2.12, makes sense.

As ever my general stance is that nobody has to make a major-version
upgrade; Spark 1.6 does not stop working for those that need Java 7. I
also think it's reasonable for anyone to expect that major-version
upgrades require major-version dependency updates. Also remember that
not removing Java 7 support means committing to it here for a couple
more years. It's not just about the situation on release day.

On Thu, Mar 24, 2016 at 8:27 AM, Reynold Xin  wrote:
> About a year ago we decided to drop Java 6 support in Spark 1.5. I am
> wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.
> Spark 2.0 would require Java 8 to run).
>
> Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
> removed public downloads for JDK 7 in July 2015. In the past I've actually
> been against dropping Java 8, but today I ran into an issue with the new
> Dataset API not working well with Java 8 lambdas, and that changed my
> opinion on this.
>
> I've been thinking more about this issue today and also talked with a lot
> people offline to gather feedback, and I actually think the pros outweighs
> the cons, for the following reasons (in some rough order of importance):
>
> 1. It is complicated to test how well Spark APIs work for Java lambdas if we
> support Java 7. Jenkins machines need to have both Java 7 and Java 8
> installed and we must run through a set of test suites in 7, and then the
> lambda tests in Java 8. This complicates build environments/scripts, and
> makes them less robust. Without good testing infrastructure, I have no
> confidence in building good APIs for Java 8.
>
> 2. Dataset/DataFrame performance will be between 1x to 10x slower in Java 7.
> The primary APIs we want users to use in Spark 2.x are Dataset/DataFrame,
> and this impacts pretty much everything from machine learning to structured
> streaming. We have made great progress in their performance through
> extensive use of code generation. (In many dimensions Spark 2.0 with
> DataFrames/Datasets looks more like a compiler than a MapReduce or query
> engine.) These optimizations don't work well in Java 7 due to broken code
> cache flushing. This problem has been fixed by Oracle in Java 8. In
> addition, Java 8 comes with better support for Unsafe and SIMD.
>
> 3. Scala 2.12 will come out soon, and we will want to add support for that.
> Scala 2.12 only works on Java 8. If we do support Java 7, we'd have a fairly
> complicated compatibility matrix and testing infrastructure.
>
> 4. There are libraries that I've looked into in the past that support only
> Java 8. This is more common in high performance libraries such as Aeron (a
> messaging library). Having to support Java 7 means we are not able to use
> these. It is not that big of a deal right now, but will become increasingly
> more difficult as we optimize performance.
>
>
> The downside of not supporting Java 7 is also obvious. Some organizations
> are stuck with Java 7, and they wouldn't be able to use Spark 2.0 without
> upgrading Java.
>
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [discuss] ending support for Java 7 in Spark 2.0

[Mridul Muralidharan]

+1
Agree, dropping support for java 7 is long overdue - and 2.0 would be
a logical release to do this on.

Regards,
Mridul


On Thu, Mar 24, 2016 at 12:27 AM, Reynold Xin  wrote:
> About a year ago we decided to drop Java 6 support in Spark 1.5. I am
> wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.
> Spark 2.0 would require Java 8 to run).
>
> Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
> removed public downloads for JDK 7 in July 2015. In the past I've actually
> been against dropping Java 8, but today I ran into an issue with the new
> Dataset API not working well with Java 8 lambdas, and that changed my
> opinion on this.
>
> I've been thinking more about this issue today and also talked with a lot
> people offline to gather feedback, and I actually think the pros outweighs
> the cons, for the following reasons (in some rough order of importance):
>
> 1. It is complicated to test how well Spark APIs work for Java lambdas if we
> support Java 7. Jenkins machines need to have both Java 7 and Java 8
> installed and we must run through a set of test suites in 7, and then the
> lambda tests in Java 8. This complicates build environments/scripts, and
> makes them less robust. Without good testing infrastructure, I have no
> confidence in building good APIs for Java 8.
>
> 2. Dataset/DataFrame performance will be between 1x to 10x slower in Java 7.
> The primary APIs we want users to use in Spark 2.x are Dataset/DataFrame,
> and this impacts pretty much everything from machine learning to structured
> streaming. We have made great progress in their performance through
> extensive use of code generation. (In many dimensions Spark 2.0 with
> DataFrames/Datasets looks more like a compiler than a MapReduce or query
> engine.) These optimizations don't work well in Java 7 due to broken code
> cache flushing. This problem has been fixed by Oracle in Java 8. In
> addition, Java 8 comes with better support for Unsafe and SIMD.
>
> 3. Scala 2.12 will come out soon, and we will want to add support for that.
> Scala 2.12 only works on Java 8. If we do support Java 7, we'd have a fairly
> complicated compatibility matrix and testing infrastructure.
>
> 4. There are libraries that I've looked into in the past that support only
> Java 8. This is more common in high performance libraries such as Aeron (a
> messaging library). Having to support Java 7 means we are not able to use
> these. It is not that big of a deal right now, but will become increasingly
> more difficult as we optimize performance.
>
>
> The downside of not supporting Java 7 is also obvious. Some organizations
> are stuck with Java 7, and they wouldn't be able to use Spark 2.0 without
> upgrading Java.
>
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

RE: [discuss] ending support for Java 7 in Spark 2.0

[Raymond Honderdors]

Very good points

Going to support java 8 looks like a good direction
2.0 would be a good release to start with that

Raymond Honderdors
Team Lead Analytics BI
Business Intelligence Developer
raymond.honderd...@sizmek.com<mailto:raymond.honderd...@sizmek.com>
T +972.7325.3569
Herzliya

From: Reynold Xin [mailto:r...@databricks.com]
Sent: Thursday, March 24, 2016 9:37 AM
To: dev@spark.apache.org
Subject: Re: [discuss] ending support for Java 7 in Spark 2.0

One other benefit that I didn't mention is that we'd be able to use Java 8's 
Optional class to replace our built-in Optional.


On Thu, Mar 24, 2016 at 12:27 AM, Reynold Xin 
<r...@databricks.com<mailto:r...@databricks.com>> wrote:
About a year ago we decided to drop Java 6 support in Spark 1.5. I am wondering 
if we should also just drop Java 7 support in Spark 2.0 (i.e. Spark 2.0 would 
require Java 8 to run).

Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and removed 
public downloads for JDK 7 in July 2015. In the past I've actually been against 
dropping Java 8, but today I ran into an issue with the new Dataset API not 
working well with Java 8 lambdas, and that changed my opinion on this.

I've been thinking more about this issue today and also talked with a lot 
people offline to gather feedback, and I actually think the pros outweighs the 
cons, for the following reasons (in some rough order of importance):

1. It is complicated to test how well Spark APIs work for Java lambdas if we 
support Java 7. Jenkins machines need to have both Java 7 and Java 8 installed 
and we must run through a set of test suites in 7, and then the lambda tests in 
Java 8. This complicates build environments/scripts, and makes them less 
robust. Without good testing infrastructure, I have no confidence in building 
good APIs for Java 8.

2. Dataset/DataFrame performance will be between 1x to 10x slower in Java 7. 
The primary APIs we want users to use in Spark 2.x are Dataset/DataFrame, and 
this impacts pretty much everything from machine learning to structured 
streaming. We have made great progress in their performance through extensive 
use of code generation. (In many dimensions Spark 2.0 with DataFrames/Datasets 
looks more like a compiler than a MapReduce or query engine.) These 
optimizations don't work well in Java 7 due to broken code cache flushing. This 
problem has been fixed by Oracle in Java 8. In addition, Java 8 comes with 
better support for Unsafe and SIMD.

3. Scala 2.12 will come out soon, and we will want to add support for that. 
Scala 2.12 only works on Java 8. If we do support Java 7, we'd have a fairly 
complicated compatibility matrix and testing infrastructure.

4. There are libraries that I've looked into in the past that support only Java 
8. This is more common in high performance libraries such as Aeron (a messaging 
library). Having to support Java 7 means we are not able to use these. It is 
not that big of a deal right now, but will become increasingly more difficult 
as we optimize performance.


The downside of not supporting Java 7 is also obvious. Some organizations are 
stuck with Java 7, and they wouldn't be able to use Spark 2.0 without upgrading 
Java.

Re: [discuss] ending support for Java 7 in Spark 2.0

[Reynold Xin]

One other benefit that I didn't mention is that we'd be able to use Java
8's Optional class to replace our built-in Optional.


On Thu, Mar 24, 2016 at 12:27 AM, Reynold Xin  wrote:

> About a year ago we decided to drop Java 6 support in Spark 1.5. I am
> wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.
> Spark 2.0 would require Java 8 to run).
>
> Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
> removed public downloads for JDK 7 in July 2015. In the past I've actually
> been against dropping Java 8, but today I ran into an issue with the new
> Dataset API not working well with Java 8 lambdas, and that changed my
> opinion on this.
>
> I've been thinking more about this issue today and also talked with a lot
> people offline to gather feedback, and I actually think the pros outweighs
> the cons, for the following reasons (in some rough order of importance):
>
> 1. It is complicated to test how well Spark APIs work for Java lambdas if
> we support Java 7. Jenkins machines need to have both Java 7 and Java 8
> installed and we must run through a set of test suites in 7, and then the
> lambda tests in Java 8. This complicates build environments/scripts, and
> makes them less robust. Without good testing infrastructure, I have no
> confidence in building good APIs for Java 8.
>
> 2. Dataset/DataFrame performance will be between 1x to 10x slower in Java
> 7. The primary APIs we want users to use in Spark 2.x are
> Dataset/DataFrame, and this impacts pretty much everything from machine
> learning to structured streaming. We have made great progress in their
> performance through extensive use of code generation. (In many dimensions
> Spark 2.0 with DataFrames/Datasets looks more like a compiler than a
> MapReduce or query engine.) These optimizations don't work well in Java 7
> due to broken code cache flushing. This problem has been fixed by Oracle in
> Java 8. In addition, Java 8 comes with better support for Unsafe and SIMD.
>
> 3. Scala 2.12 will come out soon, and we will want to add support for
> that. Scala 2.12 only works on Java 8. If we do support Java 7, we'd have a
> fairly complicated compatibility matrix and testing infrastructure.
>
> 4. There are libraries that I've looked into in the past that support only
> Java 8. This is more common in high performance libraries such as Aeron (a
> messaging library). Having to support Java 7 means we are not able to use
> these. It is not that big of a deal right now, but will become increasingly
> more difficult as we optimize performance.
>
>
> The downside of not supporting Java 7 is also obvious. Some organizations
> are stuck with Java 7, and they wouldn't be able to use Spark 2.0 without
> upgrading Java.
>
>
>

[discuss] ending support for Java 7 in Spark 2.0

[Reynold Xin]

About a year ago we decided to drop Java 6 support in Spark 1.5. I am
wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.
Spark 2.0 would require Java 8 to run).

Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
removed public downloads for JDK 7 in July 2015. In the past I've actually
been against dropping Java 8, but today I ran into an issue with the new
Dataset API not working well with Java 8 lambdas, and that changed my
opinion on this.

I've been thinking more about this issue today and also talked with a lot
people offline to gather feedback, and I actually think the pros outweighs
the cons, for the following reasons (in some rough order of importance):

1. It is complicated to test how well Spark APIs work for Java lambdas if
we support Java 7. Jenkins machines need to have both Java 7 and Java 8
installed and we must run through a set of test suites in 7, and then the
lambda tests in Java 8. This complicates build environments/scripts, and
makes them less robust. Without good testing infrastructure, I have no
confidence in building good APIs for Java 8.

2. Dataset/DataFrame performance will be between 1x to 10x slower in Java
7. The primary APIs we want users to use in Spark 2.x are
Dataset/DataFrame, and this impacts pretty much everything from machine
learning to structured streaming. We have made great progress in their
performance through extensive use of code generation. (In many dimensions
Spark 2.0 with DataFrames/Datasets looks more like a compiler than a
MapReduce or query engine.) These optimizations don't work well in Java 7
due to broken code cache flushing. This problem has been fixed by Oracle in
Java 8. In addition, Java 8 comes with better support for Unsafe and SIMD.

3. Scala 2.12 will come out soon, and we will want to add support for that.
Scala 2.12 only works on Java 8. If we do support Java 7, we'd have a
fairly complicated compatibility matrix and testing infrastructure.

4. There are libraries that I've looked into in the past that support only
Java 8. This is more common in high performance libraries such as Aeron (a
messaging library). Having to support Java 7 means we are not able to use
these. It is not that big of a deal right now, but will become increasingly
more difficult as we optimize performance.


The downside of not supporting Java 7 is also obvious. Some organizations
are stuck with Java 7, and they wouldn't be able to use Spark 2.0 without
upgrading Java.