HyukjinKwon commented on a change in pull request #25423:
[SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull
request builds
URL: https://github.com/apache/spark/pull/25423#discussion_r317666355
##########
File path: dev/run-tests.py
##########
@@ -404,6 +404,12 @@ def run_scala_tests(build_tool, hadoop_version,
test_modules, excluded_tags):
if excluded_tags:
test_profiles += ['-Dtest.exclude.tags=' + ",".join(excluded_tags)]
+ # set up java11 env if this is a pull request build with 'test-java11' in
the title
+ if "test-java11" in os.environ["ghprbPullTitle"].lower():
+ os.environ["JAVA_HOME"] = "/usr/java/jdk-11.0.1"
+ os.environ["PATH"] = "%s/bin:%s" % (os.environ["JAVA_HOME"],
os.environ["PATH"])
+ test_profiles += ['-Djava.version=11']
Review comment:
No, actually you're right. Yes, seems after Scala tests here, the PATH and
JAVA_HOME still set as are.
I thought:
https://github.com/apache/spark/blob/209b9361ac8a4410ff797cff1115e1888e2f7e66/python/pyspark/java_gateway.py#L45-L60
https://github.com/apache/spark/blob/3cb82047f2f51af553df09b9323796af507d36f8/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L425
Here somehow happened to use JDK 8.
Actually the PySpark tests and SparkR tests passed at
https://github.com/apache/spark/pull/25443#issuecomment-522206143
So, the issue persists here .. but I guess yes we can do it separately since
at least this PR _seems_ setting JDK 11 correctly, and it virtually doesn't
affect any main or test code (if this title is not used).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
