Reynold's 3rd point is particularly strong in my opinion. Supporting Scala 2.12 will require Java 8 anyway, and introducing such a change is probably best done in a major release. Consider what would happen if Spark 2.0 doesn't require Java 8 and hence not support Scala 2.12. Will it be stuck on an older version until 3.0 is out? Will it be introduced in a minor release? I think 2.0 is the best time for such a change.
On Thu, Mar 24, 2016 at 11:46 AM, Stephen Boesch <java...@gmail.com> wrote: > +1 for java8 only +1 for 2.11+ only . At this point scala libraries > supporting only 2.10 are typically less active and/or poorly maintained. > That trend will only continue when considering the lifespan of spark 2.X. > > 2016-03-24 11:32 GMT-07:00 Steve Loughran <ste...@hortonworks.com>: >> >> >> On 24 Mar 2016, at 15:27, Koert Kuipers <ko...@tresata.com> wrote: >> >> i think the arguments are convincing, but it also makes me wonder if i >> live in some kind of alternate universe... we deploy on customers clusters, >> where the OS, python version, java version and hadoop distro are not chosen >> by us. so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply >> have access to a single proxy machine and launch through yarn. asking them >> to upgrade java is pretty much out of the question or a 6+ month ordeal. of >> the 10 client clusters i can think of on the top of my head all of them are >> on java 7, none are on java 8. so by doing this you would make spark 2 >> basically unusable for us (unless most of them have plans of upgrading in >> near term to java 8, i will ask around and report back...). >> >> >> >> It's not actually mandatory for the process executing in the Yarn cluster >> to run with the same JVM as the rest of the Hadoop stack; all that is needed >> is for the environment variables to set up the JAVA_HOME and PATH. Switching >> JVMs not something which YARN makes it easy to do, but it may be possible, >> especially if Spark itself provides some hooks, so you don't have to >> manually lay with setting things up. That may be something which could >> significantly ease adoption of Spark 2 in YARN clusters. Same for Python. >> >> This is something I could probably help others to address >> > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
