This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f27eab51dcb1 [SPARK-59885][PYTHON][TEST] Fix flaky WorkerMemoryTest by
pinning spark.task.cpus
f27eab51dcb1 is described below
commit f27eab51dcb183da822bd372ad1ecdc955307626
Author: Tian Gao <[email protected]>
AuthorDate: Wed Sep 30 11:43:00 2026 -0700
[SPARK-59885][PYTHON][TEST] Fix flaky WorkerMemoryTest by pinning
spark.task.cpus
### What changes were proposed in this pull request?
Pin `spark.task.cpus=1` in `WorkerMemoryTest`
(`python/pyspark/tests/test_worker.py`) so the suite no longer relies on the
JVM's ambient default value of `spark.task.cpus`.
### Why are the changes needed?
`WorkerMemoryTest.test_memory_limit` sets
`spark.executor.pyspark.memory=2g` and asserts the Python worker's `RLIMIT_AS`
is the full 2 GiB, but it flakes with the worker capped at 1 GiB.
PySpark tests share one gateway JVM per process. The first `SparkContext`
to launch the gateway propagates its conf into the JVM's system properties (via
`JavaMainApplication`), and every later `SparkConf(loadDefaults=True)` inherits
them. When `WorkerMemoryFractionalTaskCpusTest` (`spark.task.cpus=0.5`)
launches the gateway first, the leaked fractional value causes the worker
memory limit to be split across two task slots (SPARK-58192), so
`test_memory_limit` observes 1 GiB and fails [...]
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing `WorkerMemoryTest` / `WorkerMemoryFractionalTaskCpusTest`.
Test-only change (a config pin); no new tests needed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #59147 from gaogaotiantian/fix-worker-memory-limit-task-cpus.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/tests/test_worker.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/python/pyspark/tests/test_worker.py
b/python/pyspark/tests/test_worker.py
index 1fb25b7d1986..79df02daf9ea 100644
--- a/python/pyspark/tests/test_worker.py
+++ b/python/pyspark/tests/test_worker.py
@@ -204,6 +204,9 @@ class WorkerMemoryTest(unittest.TestCase):
def setUp(self):
class_name = self.__class__.__name__
conf = SparkConf().set("spark.executor.pyspark.memory", "2g")
+ # Pin spark.task.cpus so a fractional value leaked from another suite
via JVM system
+ # properties (shared gateway JVM) cannot split the worker memory limit
and flake this.
+ conf = conf.set("spark.task.cpus", "1")
self.sc = SparkContext("local[4]", class_name, conf=conf)
def test_memory_limit(self):
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
