[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-16 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-522133218
 
 
   > @shaneknapp regarding my earlier question about what exactly runs the 
build and tests, I'm looking at 
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/configure
 and it seems to not use `run-tests`, but just build and test directly with 
Maven. Should these call `run-tests`?
   
   @srowen that build is based on what @JoshRosen (i think) and others set up 
years ago...  before i was even hired at UC berkeley.  so i don't know the 
reasoning behind the difference.
   
   `run-tests` is much much more comprehensive...  i can hazard a guess that 
these `master` and branch tests are lighter weight because before anything is 
committed to `master`, the changes will be tested w/`run-tests` during the pull 
request and review process.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-16 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-522129078
 
 
   k i kicked some tires/set the httpd retry to 0.  that should quicken things 
up.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-16 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-522125683
 
 
   > Sorry, never mind. I experiences some connection issues to AmpLab Jenkins 
30 minutes ago, but it looks okay. Maybe, it's my local network issue.
   
   oh yeah, i see what you mean.  page load times are really slow...  lemme 
take a look.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-16 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-522119608
 
 
   > BTW, is the Jenkins okay?
   
   okay in what sense?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-16 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-522112187
 
 
   > **[Test build #109229 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109229/testReport)**
 for PR 25423 at commit 
[`4d99b96`](https://github.com/apache/spark/commit/4d99b961e0f1f8ad983c041e8b4180cf94bfbba5).
   > 
   > * This patch **fails Spark unit tests**.
   
   so...  i am assuming that this is far out of the scope of this PR.  should 
we merge?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-16 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-522069012
 
 
   another question...  in the `SparkPullRequestBuilder` config, after 
`./dev/run-tests-jenkins` completes, it launches the following:
   
   ```
   # Hack to ensure that at least one JVM suite always runs in order to prevent 
spurious errors from the
   # Jenkins JUnit test reporter plugin
   ./build/sbt unsafe/test > /dev/null 2>&1
   ```
   
   1) do we need this step?
   2) if we do, do we test this w/java11?
   
   this part of the build config has been here since before i took over the 
build system so i don't really have any context except for the comment about 
the junit plugin.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-16 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-522066557
 
 
   > **[Test build #109229 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109229/testReport)**
 for PR 25423 at commit 
[`4d99b96`](https://github.com/apache/spark/commit/4d99b961e0f1f8ad983c041e8b4180cf94bfbba5).
   
   alright, it looks like it's testing against java11:
   
   ```
   
   Running Spark unit tests
   
   [info] Running Spark tests using SBT with these arguments:  -Phadoop-2.7 
-Pkubernetes -Phadoop-cloud -Phive -Phive-thriftserver -Pyarn 
-Pspark-ganglia-lgpl -Pkinesis-asl -Pmesos 
-Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest,org.apache.spark.tags.ExtendedYarnTest
 -Djava.version=11 test
   Using /usr/java/jdk-11.0.1 as default JAVA_HOME.
   Note, this will be overridden by -java-home if it is set.
   ```
   
   let's see if we have any test failures...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-16 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-522059601
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-16 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-522055428
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-16 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-522050361
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521855845
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521761223
 
 
   > I meant both Maven and SBT need the same treatment, to use Java 11 for 
tests (only). The commented-out part here would only affect SBT. Rather than 
set env variables, I'd add `-Djava.version=11` to the command line, perhaps in 
`run_scala_tests`. And this is where you'd set `JAVA_HOME`, IMHO. (Probably we 
get away with just this as everything before will be Java 8, and nothing else 
runs after tests.)
   > 
   got it...  i think my commit c4ddeff covers this for the PRB.
   
   > Yes I'm wondering if we have some duplication or conflict between what is 
the Jenkins job config, the bash commands it runs, and what is in 
`dev/run-tests.py`. I'd imagine we'd prefer to keep the complexity in the 
latter script. 
   
   i honestly don't think there's any really bad duplication or conflict here.  
the only "bad" stuff going on is really how many times we set environment 
variables...
   
   > I can't see Jenkins right now but I do recall it has a fair bit of logic 
including a `mvn` build command. But then I wonder why the `run-tests.py` 
script appears to also build Spark. I may well be missing something but is that 
redundant? It may not be, like, maybe it's in order to be able to run style 
tests first or something? But at the least it makes maintenance harder.
   > 
   the PRB only explicitly executes two things:
   1) `dev/run-tests-jenkins` (which is a wrapper for `run-tests-jenkins.py`)[1]
   2) `build/sbt unsafe/test > /dev/null`
   
   [1] -- `dev/run-tests-jenkins` launches `dev/run-tests` (which is also a 
wrapper for `run-tests.py`.
   
   btw `dev/run-tests` definitely builds spark (see the `build_apache_spark` 
function).
   
   > We need not solve that right now, just want to make sure we're making the 
right changes in the right places before deciding if this works or not.
   
   absolutely.  as you can tell, maintenance is already a nightmare.  :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521738315
 
 
   > Many things going on here. So, we build with Java 8 and test with 11. This 
should be true for Maven and SBT.
   > 
   "many things going on here" is definitely an understatement...  every time i 
have to hack on the build/test launching plumbing i feel like a small part of 
my soul dies.  ;)
   
   > I think `dev/run-tests.py` has to set `JAVA_HOME` only in, say, 
`run_scala_tests` and also set `-Djava.version=11` for both. But it should not 
affect the rest of the build/test process.
   > 
   re: "`run_scala_tests` and `JAVA_HOME`" -- got it.
   
   re: "`-Djava.version=11` for both" -- when you say 'both', what exactly are 
you referring to?  i am guessing you mean `run_scala_tests_maven` and 
`run_scala_tests_sbt`?
   
   > I am a little confused about what is meant to be in the Jenkins' script vs 
what's in this test driver script. Jenkins config _also_ seems to compile Spark 
with Maven (IIRC; I can't see it right now). Is that redundant? But, that build 
part should not use Java 11, hence agree that it shouldn't be in any 
compile-related script in Jenkins config.
   > 
   the spark compile jobs use maven.
   
   i am also confused
   
   > And then I'm still confused about why SBT doesn't like it. We shouldn't be 
asking it to compile anything if it's only applied during the test phase, and 
its error only concerns applying `scalac -target=jvm-11` But we should never 
need that. As this appears to be written, though, it sets `SBT_OPTS` for SBT 
compilation too, and that won't work. Is that the issue?
   > 
   
   
   > (The K8S changes look OK)
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-14 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521448528
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-14 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521407531
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-14 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521344491
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-14 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521337980
 
 
   hmmm.  i think i might be approaching this the wrong way for the PRB build.  
   
   if we want to build w/java 8 and test w/java 11, then i think that i won't 
actually need to change anything in `dev/run-tests.py` and 
`dev/run-tests-jenkins.py`.  instead, after taking a second look at the 
SparkPullRequestBuilder config on jenkins, i noticed that after it calls 
`dev/run-tests-jenkins`, the build executes `./build/sbt unsafe/test > 
/dev/null`.
   
   if `sbt unsafe/test` is what's actually performing the tests, then all we 
need to do is add `-Djava-version=11` to `$SBT_OPTS` when the test-java11 flag 
is in the PR title.
   
   thoughts?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-14 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521329877
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-13 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521033586
 
 
   ok...  i looked at how the regular prb tests are run 
(`dev/run-tests-jenkins` and `build/sbt`), and want to make sure that the 
regular/local tests devs run (`dev/run-tests`) have identical launch commands.
   
   i'm not sure (yet) what exactly will need modifying but a goal here is for 
people to test against any version of java they need to.  hopefully the 
additional changes to the build infra code will be minimal, and i'll take a 
closer look at that tomorrow.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-13 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520990868
 
 
   going to edit the subject and run this against java8 for both prbs.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-13 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520988931
 
 
   heads up, @skonto 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-13 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520988362
 
 
   alright i consider this fixed for the k8s integration tests...  they're 
building w/8 and successfully testing w/11.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-13 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520987055
 
 
   should we worry about this?  it's not currently/seemingly affecting anything.
   
   i mostly raise this concern because of the myriad warning messages these 
builds generate as it is.  :)
   
   ```
   WARNING: An illegal reflective access operation has occurred
   WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform 
(file:/home/jenkins/workspace/testing-k8s-prb-make-spark-distribution-unified/common/unsafe/target/spark-unsafe_2.12-3.0.0-SNAPSHOT.jar)
 to constructor java.nio.DirectByteBuffer(long,int)
   WARNING: Please consider reporting this to the maintainers of 
org.apache.spark.unsafe.Platform
   WARNING: Use --illegal-access=warn to enable warnings of further illegal 
reflective access operations
   WARNING: All illegal access operations will be denied in a future release
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-13 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520983210
 
 
   > **[Test build #109060 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109060/testReport)**
 for PR 25423 at commit 
[`88014ed`](https://github.com/apache/spark/commit/88014ed0dc5d65a7aaf2647b88e072c872e1a06f).
   
   same


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-13 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520981132
 
 
   > **[Test build #109059 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109059/testReport)**
 for PR 25423 at commit 
[`1cc6bcb`](https://github.com/apache/spark/commit/1cc6bcbe4d1064e38eb9f8d5ff20cdbb91561e41).
   
   killed this as it will fail anyways.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-13 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520978202
 
 
   testing the k8s integration tests w/commit 1cc6bcb...  had to modify the 
logic in the build to accept the new arg for java version...  tho i just 
realized i forgot to add a default.  doh!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-13 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520900223
 
 
   > For documentation generation failure, I am sure we can skip it for now if 
it matters. @shaneknapp mind adding me as a collaborator into your forked 
repository? I probably help taking a look and updating this PR when you sleep 
(I remember our timezone is completely opposite)
   
   yeah, we'll see how today goes.  if i think that i need the help then i'll 
add you as a collab. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-13 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520899965
 
 
   > If I'm right, it's just a few more lines of change here.
   
   yeah, i think so too.
   
   > The K8S script change would take a little more work, but that's mostly a 
decision about how to support this -- new flag or support arbitrary additional 
args?
   
   hmm, regardless of what i change in `dev-run-integration-tests.sh`, it will 
have to work in a non-jenkins environment...  i'm thinking arbitrary args but 
that might change once i start hacking on the script.  :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-12 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520626290
 
 
   > @shaneknapp Hm, the K8S failures are symptomatic of building with Java 11, 
but testing with 8:
   > 
   > ```
   > NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer
   > ```
   > 
   > The signature it's looking for is what changed in Java 9+.
   > 
   > Ideally we want to build with 8 and test with 11. (For the moment, 
building and testing with 11 would help a little, but not in the long term.)
   > 
   > I see that both the new and existing Java 11 PR builder just sets 
`JAVA_HOME` but doesn't set `java.version` in the build. I believe that means 
it ends up finding Java 8 to test (well, it probably runs Java 11 with a Java 8 
bootclasspath). You could check this hypothesis by adding `-Djava.version=11` 
to the `mvn` test command but, hm, this is invoking 
`dev-run-integration-tests.sh` which doesn't look like it's open to adding new 
arguments.
   > 
   > So, hm, a few things that are probably at work:
   > 
   > * This change probably has to add `-Djava.version=11` if Java 11 is 
selected, but also, set `JAVA_HOME` to 11 only for the test phase
   > * I think the other Java 11 build jobs need a similar treatment, or 
really, they'd just use the support here when ready, right -- that's the idea? 
rather than do something similar in the build scripts
   > * Somehow we have to plumb through a way to set `-Djava.version=11` to the 
K8S integration tests support
   
   alrighty then...  thanks for taking a deeper look at this.  it confirms my 
original suspicion that this wasn't going to be as straightforward of a change 
as i was hoping.
   
   i'll dig more on this tomorrow.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-12 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520574180
 
 
   > It's possible; I am not sure for example 
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/252/console
 reaches the scaladoc phase.
   > 
   that is...  odd.
   
   > Hm, I wasn't aware that one wasn't checking K8S. Let me at least add these 
to the umbrella.
   > 
   we have a spark-master k8s build.  it passes happily w/java 8:
   https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-k8s/
   
   i just created a master build for k8s and java 11 
(https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-k8s-jdk11/1/)...  
the test steps are identical to how the PRB runs, so i can guarantee that it 
will fail the same way as i saw in this PR.
   
   > I bet we can solve both without too much trouble.
   
   sgtm!  :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-12 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520566895
 
 
   > Yes, Spark does not yet pass tests on Java 11, because of Hive-related 
issues. That's the last big chunk of work. It's not a scala issue though.
   
   @srowen -- it doesn't seem to be just Hive-related issues...  testing this 
PRB against java11 also shows that it's both failing the java/scala unidoc 
section *and* the k8s integration tests.
   
   i guess the TL;DR here is twofold:
   1) spark built w/java11 will continue to be broken for the near future.
   2) we should definitely merge in these changes to master to set ourselves up 
for better java11 testing moving forward.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-12 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520554011
 
 
   > 2.12.8 should work fine enough for Spark's purposes, or at least, we 
aren't seeing any test failures in all but one module, for a long time now. The 
java module system thing is not something Spark uses.
   > 2.12.9 advertises better support, but it has an important bug.
   > We'll update to 2.12.10 eventually, but I don't think there's a known 
issue here.
   
   well, it might be blocking this PR (SPARK-27365), and every single java11 
build on jenkins is currently broken and failing hive integration tests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-12 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520550385
 
 
   > Seems scala 2.12.8 is not completely compatible to JDK 11?
   > 
   > 
https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html#jdk-11-compatibility-notes
   
   @srowen any ideas here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-12 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520549588
 
 
   > Seems scala 2.12.8 is not completely compatible to JDK 11?
   > 
   > 
https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html#jdk-11-compatibility-notes
   
   ugh.  :\


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-12 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520548360
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-12 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520548332
 
 
   need to run the k8s test again...  


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-12 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520534767
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-12 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520532293
 
 
   oof...  `javadoc` is quite unhappy about java11:
   
   ```
   [error] (spark/javaunidoc:doc) javadoc returned nonzero exit code
   [error] Total time: 122 s, completed Aug 12, 2019, 11:01:18 AM
   [error] running /home/jenkins/workspace/SparkPullRequestBuilder/build/sbt 
-Phadoop-2.7 -Pkubernetes -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl 
-Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos unidoc ; received return code 1
   ```
   
   this is definitely out of scope of this particular PR, but needs to be 
addressed.  since my java skills are rather basic i could really use some help 
here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-12 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520523540
 
 
   > Merged build finished. Test FAILed.
   
   i manually killed the initial test because i wanted to make sure that any 
k8s-based tests won't be affected by this change...  the prb and k8s tests run 
on different machines (centos vs ubuntu) and while i am certain they won't have 
any JAVA_HOME collisions it's cheaper to test and be sure.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-12 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-520522844
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org