HyukjinKwon opened a new pull request #30099:
URL: https://github.com/apache/spark/pull/30099
### What changes were proposed in this pull request?
This PR proposes to fix:
```
org.apache.spark.deploy.yarn.YarnClusterSuite.run Python application in
yarn-client mode
org.apache.spark.deploy.yarn.YarnClusterSuite.run Python application in
yarn-cluster mode
org.apache.spark.deploy.yarn.YarnClusterSuite.run Python application in
yarn-cluster mode using spark.yarn.appMasterEnv to override local envvar
```
it currently fails as below:
```
20/10/16 19:20:36 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0)
(amp-jenkins-worker-03.amp executor 1): org.apache.spark.SparkException:
Error from python worker:
Traceback (most recent call last):
File "/usr/lib64/python2.6/runpy.py", line 104, in _run_module_as_main
loader, code, fname = _get_module_details(mod_name)
File "/usr/lib64/python2.6/runpy.py", line 79, in _get_module_details
loader = get_loader(mod_name)
File "/usr/lib64/python2.6/pkgutil.py", line 456, in get_loader
return find_loader(fullname)
File "/usr/lib64/python2.6/pkgutil.py", line 466, in find_loader
for importer in iter_importers(fullname):
File "/usr/lib64/python2.6/pkgutil.py", line 422, in iter_importers
__import__(pkg)
File
"/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/__init__.py",
line 53, in <module>
from pyspark.rdd import RDD, RDDBarrier
File
"/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/rdd.py", line
34, in <module>
from pyspark.java_gateway import local_connect_and_auth
File
"/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/java_gateway.py",
line 29, in <module>
from py4j.java_gateway import java_import, JavaGateway, JavaObject,
GatewayParameters
File
"/home/jenkins/workspace/SparkPullRequestBuilder@2/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py",
line 60
PY4J_TRUE = {"yes", "y", "t", "true"}
^
SyntaxError: invalid syntax
```
I think this was broken when Python 2 was dropped but was not caught because
this specific test does not run when there's no change in YARN codes. See also
https://github.com/apache/spark/pull/29843#issuecomment-712540024
### Why are the changes needed?
To make tests pass.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
This issue looks specific to Jenkins. It should run the tests on Jenkins.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
