GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/19524
[SPARK-22302][INFRA] Remove manual backports for subprocess and print
explicit message for < Python 2.7
## What changes were proposed in this pull request?
Seems there is a mistake from missing import for `subprocess.call` which
should be used for Python 2.6 backports for some missing functions in
`subprocess`. Reproduction is:
```
cd dev && python2.6
```
```
>>> from sparktestsupport import shellutils
>>> shellutils.subprocess_check_call("ls")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sparktestsupport/shellutils.py", line 46, in subprocess_check_call
retcode = call(*popenargs, **kwargs)
NameError: global name 'call' is not defined
```
For Jenkins logs, please see
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3950/console
Since we dropped the Python 2.6.x support, looks better we remove those
workarounds and print out explicit error messages in order to duplicate the
efforts to find out the root causes for such cases.
## How was this patch tested?
Manually tested:
```
cd dev && python2.6
```
```
>>> from sparktestsupport import shellutils
[error] Python versions prior to 2.7 are not supported.
```
```
cd dev && python2.7
```
```
>>> from sparktestsupport import shellutils
>>> shellutils.subprocess_check_output("ls")
'README.md\nappveyor-guide.md\nappveyor-install-dependencies.ps1\nchange-scala-version.sh\ncheck-license\ncheckstyle-suppressions.xml\ncheckstyle.xml\ncreate-release\ndeps\ngithub_jira_sync.py\ngithub_jira_sync.pyc\nlint-java\nlint-python\nlint-r\nlint-r-report.log\nlint-r.R\nlint-scala\nmake-distribution.sh\nmerge_spark_pr.py\nmerge_spark_pr.pyc\nmerge_spark_pr_jira.py\nmerge_spark_pr_jira.pyc\nmima\npep8-1.7.0.py\npep8-1.7.0.pyc\npip-sanity-check.py\npip-sanity-check.pyc\nrequirements.txt\nrun-pip-tests\nrun-tests\nrun-tests-jenkins\nrun-tests-jenkins.py\nrun-tests-jenkins.pyc\nrun-tests.py\nrun-tests.pyc\nscalastyle\nsparktestsupport\ntest-dependencies.sh\ntests\ntox.ini\n'
>>> shellutils.subprocess_check_call("ls")
README.md checkstyle-suppressions.xml
github_jira_sync.pyc lint-r.R
merge_spark_pr_jira.py pip-sanity-check.py
run-tests-jenkins scalastyle
appveyor-guide.md checkstyle.xml
lint-java lint-scala
merge_spark_pr_jira.pyc pip-sanity-check.pyc
run-tests-jenkins.py sparktestsupport
appveyor-install-dependencies.ps1 create-release
lint-python make-distribution.sh mima
requirements.txt run-tests-jenkins.pyc
test-dependencies.sh
change-scala-version.sh deps lint-r
merge_spark_pr.py pep8-1.7.0.py
run-pip-tests run-tests.py
tests
check-license github_jira_sync.py
lint-r-report.log merge_spark_pr.pyc
pep8-1.7.0.pyc run-tests
run-tests.pyc tox.ini
0
```
```
cd dev && python3.6
```
```
>>> from sparktestsupport import shellutils
>>> shellutils.subprocess_check_output("ls")
b'README.md\nappveyor-guide.md\nappveyor-install-dependencies.ps1\nchange-scala-version.sh\ncheck-license\ncheckstyle-suppressions.xml\ncheckstyle.xml\ncreate-release\ndeps\ngithub_jira_sync.py\ngithub_jira_sync.pyc\nlint-java\nlint-python\nlint-r\nlint-r-report.log\nlint-r.R\nlint-scala\nmake-distribution.sh\nmerge_spark_pr.py\nmerge_spark_pr.pyc\nmerge_spark_pr_jira.py\nmerge_spark_pr_jira.pyc\nmima\npep8-1.7.0.py\npep8-1.7.0.pyc\npip-sanity-check.py\npip-sanity-check.pyc\nrequirements.txt\nrun-pip-tests\nrun-tests\nrun-tests-jenkins\nrun-tests-jenkins.py\nrun-tests-jenkins.pyc\nrun-tests.py\nrun-tests.pyc\nscalastyle\nsparktestsupport\ntest-dependencies.sh\ntests\ntox.ini\n'
>>> shellutils.subprocess_check_call("ls")
README.md checkstyle-suppressions.xml
github_jira_sync.pyc lint-r.R
merge_spark_pr_jira.py pip-sanity-check.py
run-tests-jenkins scalastyle
appveyor-guide.md checkstyle.xml
lint-java lint-scala
merge_spark_pr_jira.pyc pip-sanity-check.pyc
run-tests-jenkins.py sparktestsupport
appveyor-install-dependencies.ps1 create-release
lint-python make-distribution.sh mima
requirements.txt run-tests-jenkins.pyc
test-dependencies.sh
change-scala-version.sh deps lint-r
merge_spark_pr.py pep8-1.7.0.py
run-pip-tests run-tests.py
tests
check-license github_jira_sync.py
lint-r-report.log merge_spark_pr.pyc
pep8-1.7.0.pyc run-tests
run-tests.pyc tox.ini
0
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-22302
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19524.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19524
----
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
