gaogaotiantian opened a new pull request, #59172: URL: https://github.com/apache/spark/pull/59172
### What changes were proposed in this pull request? Pin `spark.python.use.daemon=true` in `WorkerReuseTest` and `WorkerPoolCrashTest` (`python/pyspark/tests/test_worker.py`) by switching them to `ReusedPySparkTestCase` and overriding `conf()`, so they no longer rely on the gateway JVM's ambient default for `spark.python.use.daemon`. ### Why are the changes needed? `WorkerReuseTest.test_reuse_worker_of_parallelize_range` and `WorkerPoolCrashTest.test_worker_crash` assert that Python worker PIDs are reused across consecutive jobs. Worker reuse (and thus stable PIDs) only happens on the daemon path: the forked daemon child loops and serves task after task in the same process. The simple-worker path (`spark.python.use.daemon=false`) runs `python -m pyspark.worker` once per task, so every task gets a fresh process and a fresh PID, and the reuse assertions fail. PySpark tests share one gateway JVM per process. The first `SparkContext` to launch the gateway propagates its conf into the JVM's system properties (via `JavaMainApplication`), and every later `SparkConf(loadDefaults=True)` inherits them (same mechanism as SPARK-59885). Sibling suites in this file set `spark.python.use.daemon=false` (`SimpleWorkerTests`, `WorkerSegfaultNonDaemonTest`). When one of those launches the gateway first, the leaked `use.daemon=false` makes the two reuse suites run in simple-worker mode, so their PID assertions fail. Because test ordering is not deterministic, the failure is flaky. Pinning `spark.python.use.daemon=true` on each suite's own `SparkContext` overrides the leaked default (the flag is read from the context's own conf via `SparkEnv.get.conf`), so the assertions are independent of test ordering. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing `WorkerReuseTest` / `WorkerPoolCrashTest`. Test-only change (a config pin); no new tests needed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
