shaneknapp edited a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521738315 > Many things going on here. So, we build with Java 8 and test with 11. This should be true for Maven and SBT. > "many things going on here" is definitely an understatement... every time i have to hack on the build/test launching plumbing i feel like a small part of my soul dies. ;) > I think `dev/run-tests.py` has to set `JAVA_HOME` only in, say, `run_scala_tests` and also set `-Djava.version=11` for both. But it should not affect the rest of the build/test process. > re: "`run_scala_tests` and `JAVA_HOME`" -- got it. re: "`-Djava.version=11` for both" -- when you say 'both', what exactly are you referring to? i am guessing you mean `run_scala_tests_maven` and `run_scala_tests_sbt`? > I am a little confused about what is meant to be in the Jenkins' script vs what's in this test driver script. Jenkins config _also_ seems to compile Spark with Maven (IIRC; I can't see it right now). Is that redundant? But, that build part should not use Java 11, hence agree that it shouldn't be in any compile-related script in Jenkins config. > the spark compile jobs use maven. when you say "the Jenkins script vs what's in this test driver script", i'm assuming you mean "how the build is configured/launched via jenkins" and "the stuff that lives in dev/" respectively. correct? > And then I'm still confused about why SBT doesn't like it. We shouldn't be asking it to compile anything if it's only applied during the test phase, and its error only concerns applying `scalac -target=jvm-11` But we should never need that. As this appears to be written, though, it sets `SBT_OPTS` for SBT compilation too, and that won't work. Is that the issue? > i lifted the `SBT_OPTS` stuff directly from the SparkPullRequestBuilder bash executable in the config... thought that might make things easier. > (The K8S changes look OK) i never thought, in a million years, that this bit would be the easiest. :) anyways, this is definitely a can of worms and while i feel we're making progress here, i'm still waiting for my brain to successfully understand everything that's going on w/this tooling.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
