gaogaotiantian opened a new pull request, #59147: URL: https://github.com/apache/spark/pull/59147
### What changes were proposed in this pull request? Pin `spark.task.cpus=1` in `WorkerMemoryTest` (`python/pyspark/tests/test_worker.py`) so the suite no longer relies on the JVM's ambient default value of `spark.task.cpus`. ### Why are the changes needed? `WorkerMemoryTest.test_memory_limit` sets `spark.executor.pyspark.memory=2g` and asserts the Python worker's `RLIMIT_AS` is the full 2 GiB, but it flakes with the worker capped at 1 GiB. PySpark tests share one gateway JVM per process. The first `SparkContext` to launch the gateway propagates its conf into the JVM's system properties (via `JavaMainApplication`), and every later `SparkConf(loadDefaults=True)` inherits them. When `WorkerMemoryFractionalTaskCpusTest` (`spark.task.cpus=0.5`) launches the gateway first, the leaked fractional value causes the worker memory limit to be split across two task slots (SPARK-58192), so `test_memory_limit` observes 1 GiB and fails. Because test ordering is not deterministic, the failure is flaky. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing `WorkerMemoryTest` / `WorkerMemoryFractionalTaskCpusTest`. Test-only change (a config pin); no new tests needed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
