Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23117#discussion_r236486298
--- Diff: dev/run-tests.py ---
@@ -434,6 +434,63 @@ def run_python_tests(test_modules, parallelism):
run_cmd(command)
+def run_python_tests_with_coverage(test_modules, parallelism):
+ set_title_and_block("Running PySpark tests with coverage report",
"BLOCK_PYSPARK_UNIT_TESTS")
+
+ command = [os.path.join(SPARK_HOME, "python",
"run-tests-with-coverage")]
+ if test_modules != [modules.root]:
+ command.append("--modules=%s" % ','.join(m.name for m in
test_modules))
+ command.append("--parallelism=%i" % parallelism)
+ run_cmd(command)
+ post_python_tests_results()
+
+
+def post_python_tests_results():
+ if "SPARK_TEST_KEY" not in os.environ:
+ print("[error] 'SPARK_TEST_KEY' environment variable was not set.
Unable to post"
+ "PySpark coverage results.")
+ sys.exit(1)
--- End diff --
@shaneknapp can you add another environment variable that indicates PR
builder and spark-master-test-sbt-hadoop-2.7 where we're going to run Python
coverage? I can check it and explicitly enable it only in that condition.
True, if the condition below (which I checked before at #17669):
```python
os.environ.get("AMPLAB_JENKINS_BUILD_PROFILE", "") ==
"hadoop2.7"
and os.environ.get("SPARK_BRANCH", "") == "master"
and os.environ.get("AMPLAB_JENKINS", "") == "true"
and os.environ.get("AMPLAB_JENKINS_BUILD_TOOL", "") == "sbt")
```
is `True` in Jenkins build or other users environment, it might cause some
problems (even though looks quite unlikely).
For similar instance, if `AMPLAB_JENKINS` is set in users environment who
run the tests locally, it wouldn't work anyway tho.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
