gaogaotiantian opened a new pull request, #59169: URL: https://github.com/apache/spark/pull/59169
### What changes were proposed in this pull request? PySpark smart test selection (`PySparkBaseTestCase`, driven by `PYSPARK_CHANGED_FILES`) keeps a test only when a changed file's module is reachable from the test module in the static import graph built by `grimp`. Scripts that the JVM launches as separate processes -- `pyspark.worker`, `pyspark.daemon`, `pyspark.worker_util`, and the `pyspark.sql.worker.*` runners -- are never imported by any test module, so they have no edge in the graph. A change touching only those files therefore looks irrelevant to every test, and all of them get skipped. This adds a small `_ENTRY_POINT_PROXIES` mapping on `PySparkBaseTestCase` from each such entry-point script to the API module whose use triggers it at runtime, and translates a changed entry-point module to its proxy before the reachability check: - `pyspark.worker` / `pyspark.worker_util` / `pyspark.daemon` -> relevant to every test (they underpin essentially all Python-worker execution); - `pyspark.sql.worker.analyze_udtf` -> `pyspark.sql.udtf`; - the `pyspark.sql.worker.*` data source runners -> `pyspark.sql.datasource`. ### Why are the changes needed? Without this, a PR that changes only `worker.py` (a common case) would mark all UDF, pandas, Arrow, UDTF, and Python data source tests as irrelevant in post-merge smart selection -- exactly the tests that exercise the changed code -- and silently lose that coverage. The mapping only ever over-selects (the safe direction); it never skips a test that could be affected. ### Does this PR introduce _any_ user-facing change? No. This only affects PySpark's CI test selection. ### How was this patch tested? Extended `pyspark/testing/tests/test_changed_files.py`: - `test_relevance` now also asserts that a shared worker file is relevant to the test module, and that the data source / UDTF runners are relevant via their proxy modules. - A new `test_entry_point_proxy_selectivity` uses a leaf utility module that reaches neither `pyspark.sql.datasource` nor `pyspark.sql.udtf` to confirm the shared worker files are relevant to it (relevant to all) while the specialized runners are not. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac (Claude Code) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
