If someone else is looking how to try jdk9, you can just pass your own JAVA_HOME environment variables:
spark.yarn.appMasterEnv.JAVA_HOME=/usr/lib/jvm/java-1.8.0 spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-1.8.0 On Fri, Oct 27, 2017 at 5:14 AM, Steve Loughran <ste...@hortonworks.com> wrote: > > On 27 Oct 2017, at 03:21, Zhang, Liyun <liyun.zh...@intel.com> wrote: > > Hi all: > 1. I want to build spark on jdk9 and test it with Hadoop on jdk9 > env. I search for jiras related to JDK9. I only found SPARK-13278 > <https://issues.apache.org/jira/browse/SPARK-13278>. This means now > spark can build or run successfully on JDK9 ? > > > Best Regards > Kelly Zhang/Zhang,Liyun > > > > Don't know about spark itself, but I do know that getting Hadoop on JDK9 > is still a WiP, primarily because they've locked it down so much (good) > including the things hadoop gets at to make things like kerberos auth work > (bad) > > https://issues.apache.org/jira/browse/HADOOP-11123 > > A large part of the issues are with moving its dependencies to Java 9 > compatible ones (Log4J, mockito, JUnit) —all of those which only surface in > testing and in the build itself won't be relevant for Spark standalone. > Other than that, YARN doesn't work > > Most of the work has been done by one person (Akira @ NEC); if there are > other people willing to help, including building & testing Spark against > (locally built) JDK9 Hadoop artifacts life would be better. We could maybe > build & release some alpha-quality Hadoop 3.1.x-alpha-JDK9 artifacts if > that would help > > FWIW, there's long been some background chatter between the old Sun JDK > team & the ASF big data stack devs; modules is something wonderful which > will kill the need for shading and reduce/eliminate classpath hell. Earlier > on there was some discussion about having proper 2d arrays & direct memory > access of some structures, but that's not in this version. Give it time. > > Oracle are being aggressive about retiring Java 8: by Sept 2018 they plan > to not provide public updates for it. Which means building against Java 9 > dev time is here for everyone >
