Yikun opened a new pull request #34993:
URL: https://github.com/apache/spark/pull/34993
### What changes were proposed in this pull request?
Replace `os.environ` direct access with `_find_spark_home` in pyspark
testing utils.
### Why are the changes needed?
When running PySpark unit test in WSL with PyCharm, all test raise the
`KeyError: 'SPARK_HOME'`
```
Launching unittests with arguments python -m unittest
test_rdd.RDDTests.test_range in /home/yikun/spark/python/pyspark/testsTraceback
(most recent call last):
File "/mnt/d/Program Files/JetBrains/PyCharm
2021.1.3/plugins/python/helpers/pycharm/_jb_unittest_runner.py", line 35, in
<module>
sys.exit(main(argv=args, module=None,
testRunner=unittestpy.TeamcityTestRunner, buffer=not JB_DISABLE_BUFFERING))
File "/usr/lib/python3.8/unittest/main.py", line 100, in __init__
self.parseArgs(argv)
File "/usr/lib/python3.8/unittest/main.py", line 147, in parseArgs
self.createTests()
File "/usr/lib/python3.8/unittest/main.py", line 158, in createTests
self.test = self.testLoader.loadTestsFromNames(self.testNames,
File "/usr/lib/python3.8/unittest/loader.py", line 220, in
loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/lib/python3.8/unittest/loader.py", line 220, in <listcomp>
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/lib/python3.8/unittest/loader.py", line 154, in
loadTestsFromName
module = __import__(module_name)
File "/home/yikun/spark/python/pyspark/tests/test_rdd.py", line 37, in
<module>
from pyspark.testing.utils import ReusedPySparkTestCase, SPARK_HOME,
QuietTest
File "/home/yikun/spark/python/pyspark/testing/utils.py", line 47, in
<module>
SPARK_HOME = os.environ["SPARK_HOME"]#_find_spark_home()
File "/usr/lib/python3.8/os.py", line 675, in __getitem__
raise KeyError(key) from None
KeyError: 'SPARK_HOME'
```
We should use `_find_spark_home` to get SPARK_HOME rather than access
`os.environ`.
### Does this PR introduce _any_ user-facing change?
No, test only
### How was this patch tested?
- UT passed.
- Run test in Win WSL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
