This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 4ab5197529ae [SPARK-46564][PYTHON][INFRA] Exclude unrelated files via
using omit options properly in PySpark coverage report
4ab5197529ae is described below
commit 4ab5197529aed1d5cdaa26a9741bc988b15c08bd
Author: Hyukjin Kwon <gurwls...@apache.org>
AuthorDate: Tue Jan 2 08:01:26 2024 -0800
[SPARK-46564][PYTHON][INFRA] Exclude unrelated files via using omit options
properly in PySpark coverage report
### What changes were proposed in this pull request?
This PR proposes to remove 3rd party or generated files from PySpark test
coverage report by using the proper syntax for `--omit` supported by `coverage`
command. `--omit="pyspark/..." --omit=...` used to work but seems it doesn't
work anymore in newer versions.
### Why are the changes needed?
The files are not excluded for some reasons at the PySpark test coverage
report (https://app.codecov.io/gh/apache/spark)
We should exclude unrelated files.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Manually tested via:
```
./run-tests-with-coverage --python-executables=python3
--testname="pyspark.sql.dataframe"
```
checked those files are not reported:
```
Reporting the coverage data at
/.../spark/python/test_coverage/coverage_data/coverage
Name Stmts Miss Branch BrPart Cover
-------------------------------------------------------------------------
pyspark/__init__.py 48 7 10 3 76%
pyspark/_globals.py 16 3 4 2 75%
pyspark/accumulators.py 121 44 26 5 61%
pyspark/broadcast.py 121 79 40 3 33%
...
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #44559 from HyukjinKwon/SPARK-46564.
Authored-by: Hyukjin Kwon <gurwls...@apache.org>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
python/run-tests-with-coverage | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/python/run-tests-with-coverage b/python/run-tests-with-coverage
index f6b6d965254d..d1c2dacbf9d8 100755
--- a/python/run-tests-with-coverage
+++ b/python/run-tests-with-coverage
@@ -60,10 +60,10 @@ find $COVERAGE_DIR/coverage_data -size 0 -print0 | xargs -0
rm -fr
echo "Combining collected coverage data under $COVERAGE_DIR/coverage_data"
$COV_EXEC combine
echo "Creating XML report file at python/coverage.xml"
-$COV_EXEC xml --ignore-errors --include "pyspark/*" --omit
"pyspark/cloudpickle/*" --omit "pyspark/sql/connect/proto/*" --omit
"python/pyspark/loose_version.py"
+$COV_EXEC xml --ignore-errors --include "pyspark/*" --omit
"*/pyspark/cloudpickle/*,*/pyspark/sql/connect/proto/*,*/python/pyspark/loose_version.py"
echo "Reporting the coverage data at $COVERAGE_DIR/coverage_data/coverage"
-$COV_EXEC report --include "pyspark/*" --omit "pyspark/cloudpickle/*" --omit
"pyspark/sql/connect/proto/*" --omit "python/pyspark/loose_version.py"
+$COV_EXEC report --include "pyspark/*" --omit
"*/pyspark/cloudpickle/*,*/pyspark/sql/connect/proto/*,*/python/pyspark/loose_version.py"
echo "Generating HTML files for PySpark coverage under $COVERAGE_DIR/htmlcov"
-$COV_EXEC html --ignore-errors --include "pyspark/*" --directory
"$COVERAGE_DIR/htmlcov" --omit "pyspark/cloudpickle/*" --omit
"pyspark/sql/connect/proto/*" --omit "python/pyspark/loose_version.py"
+$COV_EXEC html --ignore-errors --include "pyspark/*" --directory
"$COVERAGE_DIR/htmlcov" --omit
"*/pyspark/cloudpickle/*,*/pyspark/sql/connect/proto/*,*/python/pyspark/loose_version.py"
popd
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
