This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c5eebb341db5 [SPARK-59060][PYTHON] Add missing cpuAmount to the
benchmark mock task context
c5eebb341db5 is described below
commit c5eebb341db59ad82dd013ff0067c8f742b0907b
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Thu Aug 27 14:57:33 2026 -0700
[SPARK-59060][PYTHON] Add missing cpuAmount to the benchmark mock task
context
### What changes were proposed in this pull request?
Adds the `cpuAmount` key to the mock task-context JSON that
`MockProtocolWriter` writes in `python/benchmarks/bench_eval_type.py`.
### Why are the changes needed?
Every benchmark in `bench_eval_type.py` currently fails at worker
initialization. `TaskContextInfo.from_stream` reads
`task_context_json["cpuAmount"]` without a default
(`python/pyspark/worker_message.py`), so the worker raises `KeyError:
'cpuAmount'` before any UDF runs:
```
File "python/pyspark/worker_message.py", line 63, in from_stream
cpu_amount=Decimal(task_context_json["cpuAmount"]),
KeyError: 'cpuAmount'
```
The key became required in SPARK-58192, which added it on the JVM side and
in `worker_message.py` but did not update this mock.
The failure is easy to miss because `setup()` only builds the input buffer;
only running `time_worker` / `peakmem_worker` reaches the worker. Nothing under
`python/benchmarks/` is listed in `dev/sparktestsupport/modules.py`, so no CI
job exercises these benchmarks.
### Does this PR introduce _any_ user-facing change?
No. Benchmark-only change.
### How was this patch tested?
Ran every benchmark class in the file, driving the worker end to end:
| | before | after |
|---|---|---|
| `*TimeBench` | 0 of 26 | 26 of 26 |
| `*PeakmemBench` | 0 of 26 | 26 of 26 |
Also checked out each release branch and ran the same benchmarks there:
| branch | result |
|---|---|
| `master` | 0 of 26; 26 of 26 with this fix |
| `branch-4.x` | same failure; the same one-line fix applies |
| `branch-4.3` | 0 of 26; 26 of 26 with this fix |
| `branch-4.2` | unaffected -- no `worker_message.py`, so the key is not
required |
So this is worth backporting to `branch-4.x` and `branch-4.3`.
The value is a plain decimal string, matching `CpuAmount.toDisplayString`
on the JVM side
(`core/src/main/scala/org/apache/spark/api/python/PythonWorkerUtils.scala`),
and is consistent with the existing `"cpus": 1` in the same mock.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
Closes #58353 from viirya/SPARK-bench-cpuamount.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
python/benchmarks/bench_eval_type.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/python/benchmarks/bench_eval_type.py
b/python/benchmarks/bench_eval_type.py
index 85a1bd03b0d3..1c0a4e47dccf 100644
--- a/python/benchmarks/bench_eval_type.py
+++ b/python/benchmarks/bench_eval_type.py
@@ -117,6 +117,9 @@ class MockProtocolWriter:
"attemptNumber": 0,
"taskAttemptId": 0,
"cpus": 1,
+ # Plain decimal string, matching CpuAmount.toDisplayString
on
+ # the JVM side; TaskContextInfo.from_stream requires the
key.
+ "cpuAmount": "1",
"resources": {},
"localProperties": {},
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
