[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522133218 > @shaneknapp regarding my earlier question about what exactly runs the build and tests, I'm looking at https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/configure and it seems to not use `run-tests`, but just build and test directly with Maven. Should these call `run-tests`? @srowen that build is based on what @JoshRosen (i think) and others set up years ago... before i was even hired at UC berkeley. so i don't know the reasoning behind the difference. `run-tests` is much much more comprehensive... i can hazard a guess that these `master` and branch tests are lighter weight because before anything is committed to `master`, the changes will be tested w/`run-tests` during the pull request and review process. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522129078 k i kicked some tires/set the httpd retry to 0. that should quicken things up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522125683 > Sorry, never mind. I experiences some connection issues to AmpLab Jenkins 30 minutes ago, but it looks okay. Maybe, it's my local network issue. oh yeah, i see what you mean. page load times are really slow... lemme take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522119608 > BTW, is the Jenkins okay? okay in what sense? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522112187 > **[Test build #109229 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109229/testReport)** for PR 25423 at commit [`4d99b96`](https://github.com/apache/spark/commit/4d99b961e0f1f8ad983c041e8b4180cf94bfbba5). > > * This patch **fails Spark unit tests**. so... i am assuming that this is far out of the scope of this PR. should we merge? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522069012 another question... in the `SparkPullRequestBuilder` config, after `./dev/run-tests-jenkins` completes, it launches the following: ``` # Hack to ensure that at least one JVM suite always runs in order to prevent spurious errors from the # Jenkins JUnit test reporter plugin ./build/sbt unsafe/test > /dev/null 2>&1 ``` 1) do we need this step? 2) if we do, do we test this w/java11? this part of the build config has been here since before i took over the build system so i don't really have any context except for the comment about the junit plugin. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522066557 > **[Test build #109229 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109229/testReport)** for PR 25423 at commit [`4d99b96`](https://github.com/apache/spark/commit/4d99b961e0f1f8ad983c041e8b4180cf94bfbba5). alright, it looks like it's testing against java11: ``` Running Spark unit tests [info] Running Spark tests using SBT with these arguments: -Phadoop-2.7 -Pkubernetes -Phadoop-cloud -Phive -Phive-thriftserver -Pyarn -Pspark-ganglia-lgpl -Pkinesis-asl -Pmesos -Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest,org.apache.spark.tags.ExtendedYarnTest -Djava.version=11 test Using /usr/java/jdk-11.0.1 as default JAVA_HOME. Note, this will be overridden by -java-home if it is set. ``` let's see if we have any test failures... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522059601 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522055428 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522050361 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521855845 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521761223 > I meant both Maven and SBT need the same treatment, to use Java 11 for tests (only). The commented-out part here would only affect SBT. Rather than set env variables, I'd add `-Djava.version=11` to the command line, perhaps in `run_scala_tests`. And this is where you'd set `JAVA_HOME`, IMHO. (Probably we get away with just this as everything before will be Java 8, and nothing else runs after tests.) > got it... i think my commit c4ddeff covers this for the PRB. > Yes I'm wondering if we have some duplication or conflict between what is the Jenkins job config, the bash commands it runs, and what is in `dev/run-tests.py`. I'd imagine we'd prefer to keep the complexity in the latter script. i honestly don't think there's any really bad duplication or conflict here. the only "bad" stuff going on is really how many times we set environment variables... > I can't see Jenkins right now but I do recall it has a fair bit of logic including a `mvn` build command. But then I wonder why the `run-tests.py` script appears to also build Spark. I may well be missing something but is that redundant? It may not be, like, maybe it's in order to be able to run style tests first or something? But at the least it makes maintenance harder. > the PRB only explicitly executes two things: 1) `dev/run-tests-jenkins` (which is a wrapper for `run-tests-jenkins.py`)[1] 2) `build/sbt unsafe/test > /dev/null` [1] -- `dev/run-tests-jenkins` launches `dev/run-tests` (which is also a wrapper for `run-tests.py`. btw `dev/run-tests` definitely builds spark (see the `build_apache_spark` function). > We need not solve that right now, just want to make sure we're making the right changes in the right places before deciding if this works or not. absolutely. as you can tell, maintenance is already a nightmare. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521738315 > Many things going on here. So, we build with Java 8 and test with 11. This should be true for Maven and SBT. > "many things going on here" is definitely an understatement... every time i have to hack on the build/test launching plumbing i feel like a small part of my soul dies. ;) > I think `dev/run-tests.py` has to set `JAVA_HOME` only in, say, `run_scala_tests` and also set `-Djava.version=11` for both. But it should not affect the rest of the build/test process. > re: "`run_scala_tests` and `JAVA_HOME`" -- got it. re: "`-Djava.version=11` for both" -- when you say 'both', what exactly are you referring to? i am guessing you mean `run_scala_tests_maven` and `run_scala_tests_sbt`? > I am a little confused about what is meant to be in the Jenkins' script vs what's in this test driver script. Jenkins config _also_ seems to compile Spark with Maven (IIRC; I can't see it right now). Is that redundant? But, that build part should not use Java 11, hence agree that it shouldn't be in any compile-related script in Jenkins config. > the spark compile jobs use maven. i am also confused > And then I'm still confused about why SBT doesn't like it. We shouldn't be asking it to compile anything if it's only applied during the test phase, and its error only concerns applying `scalac -target=jvm-11` But we should never need that. As this appears to be written, though, it sets `SBT_OPTS` for SBT compilation too, and that won't work. Is that the issue? > > (The K8S changes look OK) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521448528 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521407531 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521344491 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521337980 hmmm. i think i might be approaching this the wrong way for the PRB build. if we want to build w/java 8 and test w/java 11, then i think that i won't actually need to change anything in `dev/run-tests.py` and `dev/run-tests-jenkins.py`. instead, after taking a second look at the SparkPullRequestBuilder config on jenkins, i noticed that after it calls `dev/run-tests-jenkins`, the build executes `./build/sbt unsafe/test > /dev/null`. if `sbt unsafe/test` is what's actually performing the tests, then all we need to do is add `-Djava-version=11` to `$SBT_OPTS` when the test-java11 flag is in the PR title. thoughts? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521329877 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521033586 ok... i looked at how the regular prb tests are run (`dev/run-tests-jenkins` and `build/sbt`), and want to make sure that the regular/local tests devs run (`dev/run-tests`) have identical launch commands. i'm not sure (yet) what exactly will need modifying but a goal here is for people to test against any version of java they need to. hopefully the additional changes to the build infra code will be minimal, and i'll take a closer look at that tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520990868 going to edit the subject and run this against java8 for both prbs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520988931 heads up, @skonto This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520988362 alright i consider this fixed for the k8s integration tests... they're building w/8 and successfully testing w/11. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520987055 should we worry about this? it's not currently/seemingly affecting anything. i mostly raise this concern because of the myriad warning messages these builds generate as it is. :) ``` WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/home/jenkins/workspace/testing-k8s-prb-make-spark-distribution-unified/common/unsafe/target/spark-unsafe_2.12-3.0.0-SNAPSHOT.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520983210 > **[Test build #109060 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109060/testReport)** for PR 25423 at commit [`88014ed`](https://github.com/apache/spark/commit/88014ed0dc5d65a7aaf2647b88e072c872e1a06f). same This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520981132 > **[Test build #109059 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109059/testReport)** for PR 25423 at commit [`1cc6bcb`](https://github.com/apache/spark/commit/1cc6bcbe4d1064e38eb9f8d5ff20cdbb91561e41). killed this as it will fail anyways. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520978202 testing the k8s integration tests w/commit 1cc6bcb... had to modify the logic in the build to accept the new arg for java version... tho i just realized i forgot to add a default. doh! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520900223 > For documentation generation failure, I am sure we can skip it for now if it matters. @shaneknapp mind adding me as a collaborator into your forked repository? I probably help taking a look and updating this PR when you sleep (I remember our timezone is completely opposite) yeah, we'll see how today goes. if i think that i need the help then i'll add you as a collab. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520899965 > If I'm right, it's just a few more lines of change here. yeah, i think so too. > The K8S script change would take a little more work, but that's mostly a decision about how to support this -- new flag or support arbitrary additional args? hmm, regardless of what i change in `dev-run-integration-tests.sh`, it will have to work in a non-jenkins environment... i'm thinking arbitrary args but that might change once i start hacking on the script. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520626290 > @shaneknapp Hm, the K8S failures are symptomatic of building with Java 11, but testing with 8: > > ``` > NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer > ``` > > The signature it's looking for is what changed in Java 9+. > > Ideally we want to build with 8 and test with 11. (For the moment, building and testing with 11 would help a little, but not in the long term.) > > I see that both the new and existing Java 11 PR builder just sets `JAVA_HOME` but doesn't set `java.version` in the build. I believe that means it ends up finding Java 8 to test (well, it probably runs Java 11 with a Java 8 bootclasspath). You could check this hypothesis by adding `-Djava.version=11` to the `mvn` test command but, hm, this is invoking `dev-run-integration-tests.sh` which doesn't look like it's open to adding new arguments. > > So, hm, a few things that are probably at work: > > * This change probably has to add `-Djava.version=11` if Java 11 is selected, but also, set `JAVA_HOME` to 11 only for the test phase > * I think the other Java 11 build jobs need a similar treatment, or really, they'd just use the support here when ready, right -- that's the idea? rather than do something similar in the build scripts > * Somehow we have to plumb through a way to set `-Djava.version=11` to the K8S integration tests support alrighty then... thanks for taking a deeper look at this. it confirms my original suspicion that this wasn't going to be as straightforward of a change as i was hoping. i'll dig more on this tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520574180 > It's possible; I am not sure for example https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/252/console reaches the scaladoc phase. > that is... odd. > Hm, I wasn't aware that one wasn't checking K8S. Let me at least add these to the umbrella. > we have a spark-master k8s build. it passes happily w/java 8: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-k8s/ i just created a master build for k8s and java 11 (https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-k8s-jdk11/1/)... the test steps are identical to how the PRB runs, so i can guarantee that it will fail the same way as i saw in this PR. > I bet we can solve both without too much trouble. sgtm! :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520566895 > Yes, Spark does not yet pass tests on Java 11, because of Hive-related issues. That's the last big chunk of work. It's not a scala issue though. @srowen -- it doesn't seem to be just Hive-related issues... testing this PRB against java11 also shows that it's both failing the java/scala unidoc section *and* the k8s integration tests. i guess the TL;DR here is twofold: 1) spark built w/java11 will continue to be broken for the near future. 2) we should definitely merge in these changes to master to set ourselves up for better java11 testing moving forward. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520554011 > 2.12.8 should work fine enough for Spark's purposes, or at least, we aren't seeing any test failures in all but one module, for a long time now. The java module system thing is not something Spark uses. > 2.12.9 advertises better support, but it has an important bug. > We'll update to 2.12.10 eventually, but I don't think there's a known issue here. well, it might be blocking this PR (SPARK-27365), and every single java11 build on jenkins is currently broken and failing hive integration tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520550385 > Seems scala 2.12.8 is not completely compatible to JDK 11? > > https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html#jdk-11-compatibility-notes @srowen any ideas here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520549588 > Seems scala 2.12.8 is not completely compatible to JDK 11? > > https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html#jdk-11-compatibility-notes ugh. :\ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520548332 need to run the k8s test again... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520548360 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520534767 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520532293 oof... `javadoc` is quite unhappy about java11: ``` [error] (spark/javaunidoc:doc) javadoc returned nonzero exit code [error] Total time: 122 s, completed Aug 12, 2019, 11:01:18 AM [error] running /home/jenkins/workspace/SparkPullRequestBuilder/build/sbt -Phadoop-2.7 -Pkubernetes -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos unidoc ; received return code 1 ``` this is definitely out of scope of this particular PR, but needs to be addressed. since my java skills are rather basic i could really use some help here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520523540 > Merged build finished. Test FAILed. i manually killed the initial test because i wanted to make sure that any k8s-based tests won't be affected by this change... the prb and k8s tests run on different machines (centos vs ubuntu) and while i am certain they won't have any JAVA_HOME collisions it's cheaper to test and be sure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-520522844 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
