Tian Gao created SPARK-59906:
--------------------------------
Summary: Run relevant tests when JVM-launched Python worker
scripts change under smart test selection
Key: SPARK-59906
URL: https://issues.apache.org/jira/browse/SPARK-59906
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 5.0.0
Reporter: Tian Gao
PySpark smart test selection (PySparkBaseTestCase, driven by
PYSPARK_CHANGED_FILES) keeps a test only when the changed file's module is
reachable from the test module in the static import graph built by grimp.
Scripts that the JVM launches as separate processes (pyspark.worker,
pyspark.daemon, pyspark.worker_util, and the pyspark.sql.worker.* runners) are
never imported by any test module, so a change touching only those files is
treated as irrelevant to every test and all of them are skipped -- including
the UDF, pandas, Arrow, UDTF, and Python data source tests that actually
exercise the changed code. This maps each such entry-point script to the API
module whose use triggers it (the worker/daemon files are treated as relevant
to every test), so the corresponding tests are selected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
