Curveball: Is there a need to use lambdas quite yet? On Mon, Apr 4, 2016 at 10:58 PM, Ofir Manor <ofir.ma...@equalum.io> wrote: > I think that a backup plan could be to announce that JDK7 is deprecated in > Spark 2.0 and support for it will be fully removed in Spark 2.1. This gives > admins enough warning to install JDK8 along side their "main" JDK (or fully > migrate to it), while allowing the project to merge JDK8-specific changes to > trunk right after the 2.0 release. > > However, I personally think it is better to drop JDK7 now. I'm sure that > both the community and the distributors (Databricks, Cloudera, Hortonworks, > MapR, IBM etc) will all rush to help their customers migrate their > environment to support Spark 2.0, so I think any backlash won't be dramatic > or lasting. > > Just my two cents, > > Ofir Manor > > Co-Founder & CTO | Equalum > > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io > > > On Mon, Apr 4, 2016 at 6:48 PM, Luciano Resende <luckbr1...@gmail.com> > wrote: >> >> Reynold, >> >> Considering the performance improvements you mentioned in your original >> e-mail and also considering that few other big data projects have already or >> are in progress of abandoning JDK 7, I think it would benefit Spark if we go >> with JDK 8.0 only. >> >> Are there users that will be less aggressive ? Yes, but those would most >> likely be in more stable releases like 1.6.x. >> >> On Sun, Apr 3, 2016 at 10:28 PM, Reynold Xin <r...@databricks.com> wrote: >>> >>> Since my original email, I've talked to a lot more users and looked at >>> what various environments support. It is true that a lot of enterprises, and >>> even some technology companies, are still using Java 7. One thing is that up >>> until this date, users still can't install openjdk 8 on Ubuntu by default. I >>> see that as an indication that it is too early to drop Java 7. >>> >>> Looking at the timeline, JDK release a major new version roughly every 3 >>> years. We dropped Java 6 support one year ago, so from a timeline point of >>> view we would be very aggressive here if we were to drop Java 7 support in >>> Spark 2.0. >>> >>> Note that not dropping Java 7 support now doesn't mean we have to support >>> Java 7 throughout Spark 2.x. We dropped Java 6 support in Spark 1.5, even >>> though Spark 1.0 started with Java 6. >>> >>> In terms of testing, Josh has actually improved our test infra so now we >>> would run the Java 8 tests: https://github.com/apache/spark/pull/12073 >>> >>> >>> >>> >>> On Thu, Mar 24, 2016 at 8:51 PM, Liwei Lin <lwl...@gmail.com> wrote: >>>> >>>> Arguments are really convincing; new Dataset API as well as performance >>>> >>>> improvements is exiting, so I'm personally +1 on moving onto Java8. >>>> >>>> >>>> >>>> However, I'm afraid Tencent is one of "the organizations stuck with >>>> Java7" >>>> >>>> -- our IT Infra division wouldn't upgrade to Java7 until Java8 is out, >>>> and >>>> >>>> wouldn't upgrade to Java8 until Java9 is out. >>>> >>>> >>>> So: >>>> >>>> (non-binding) +1 on dropping scala 2.10 support >>>> >>>> (non-binding) -1 on dropping Java 7 support >>>> >>>> * as long as we figure out a practical way to run >>>> Spark with >>>> >>>> JDK8 on JDK7 clusters, this -1 would then >>>> definitely be +1 >>>> >>>> >>>> Thanks ! >>>> >>>> >>>> On Fri, Mar 25, 2016 at 10:28 AM, Koert Kuipers <ko...@tresata.com> >>>> wrote: >>>>> >>>>> i think that logic is reasonable, but then the same should also apply >>>>> to scala 2.10, which is also unmaintained/unsupported at this point >>>>> (basically has been since march 2015 except for one hotfix due to a >>>>> license >>>>> incompatibility) >>>>> >>>>> who wants to support scala 2.10 three years after they did the last >>>>> maintenance release? >>>>> >>>>> >>>>> On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan <mri...@gmail.com> >>>>> wrote: >>>>>> >>>>>> Removing compatibility (with jdk, etc) can be done with a major >>>>>> release- given that 7 has been EOLed a while back and is now >>>>>> unsupported, we >>>>>> have to decide if we drop support for it in 2.0 or 3.0 (2+ years from >>>>>> now). >>>>>> >>>>>> Given the functionality & performance benefits of going to jdk8, >>>>>> future enhancements relevant in 2.x timeframe ( scala, dependencies) >>>>>> which >>>>>> requires it, and simplicity wrt code, test & support it looks like a good >>>>>> checkpoint to drop jdk7 support. >>>>>> >>>>>> As already mentioned in the thread, existing yarn clusters are >>>>>> unaffected if they want to continue running jdk7 and yet use spark2 >>>>>> (install >>>>>> jdk8 on all nodes and use it via JAVA_HOME, or worst case distribute >>>>>> jdk8 as >>>>>> archive - suboptimal). >>>>>> I am unsure about mesos (standalone might be easier upgrade I guess >>>>>> ?). >>>>>> >>>>>> >>>>>> Proposal is for 1.6x line to continue to be supported with critical >>>>>> fixes; newer features will require 2.x and so jdk8 >>>>>> >>>>>> Regards >>>>>> Mridul >>>>>> >>>>>> >>>>>> On Thursday, March 24, 2016, Marcelo Vanzin <van...@cloudera.com> >>>>>> wrote: >>>>>>> >>>>>>> On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin <r...@databricks.com> >>>>>>> wrote: >>>>>>> > If you want to go down that route, you should also ask somebody who >>>>>>> > has had >>>>>>> > experience managing a large organization's applications and try to >>>>>>> > update >>>>>>> > Scala version. >>>>>>> >>>>>>> I understand both sides. But if you look at what I've been asking >>>>>>> since the beginning, it's all about the cost and benefits of dropping >>>>>>> support for java 1.7. >>>>>>> >>>>>>> The biggest argument in your original e-mail is about testing. And >>>>>>> the >>>>>>> testing cost is much bigger for supporting scala 2.10 than it is for >>>>>>> supporting java 1.7. If you read one of my earlier replies, it should >>>>>>> be even possible to just do everything in a single job - compile for >>>>>>> java 7 and still be able to test things in 1.8, including lambdas, >>>>>>> which seems to be the main thing you were worried about. >>>>>>> >>>>>>> >>>>>>> > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin >>>>>>> > <van...@cloudera.com> wrote: >>>>>>> >> >>>>>>> >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin <r...@databricks.com> >>>>>>> >> wrote: >>>>>>> >> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, >>>>>>> >> > than >>>>>>> >> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and >>>>>>> >> > 2.11 are >>>>>>> >> > not >>>>>>> >> > binary compatible, whereas JVM 7 and 8 are binary compatible >>>>>>> >> > except >>>>>>> >> > certain >>>>>>> >> > esoteric cases. >>>>>>> >> >>>>>>> >> True, but ask anyone who manages a large cluster how long it would >>>>>>> >> take them to upgrade the jdk across their cluster and validate all >>>>>>> >> their applications and everything... binary compatibility is a >>>>>>> >> tiny >>>>>>> >> drop in that bucket. >>>>>>> >> >>>>>>> >> -- >>>>>>> >> Marcelo >>>>>>> > >>>>>>> > >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Marcelo >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>>>>>> For additional commands, e-mail: dev-h...@spark.apache.org >>>>>>> >>>>> >>>> >>> >> >> >> >> -- >> Luciano Resende >> http://twitter.com/lresende1975 >> http://lresende.blogspot.com/ > >
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
