This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 7943eb8922a9 [MINOR][PYTHON] Coerce the JVM error message to str in
ArrowCollectSerializer.load_stream
7943eb8922a9 is described below
commit 7943eb8922a9098855dce84fd473cea8c7b5033a
Author: Spenser Sun <[email protected]>
AuthorDate: Tue Oct 6 05:54:55 2026 +0000
[MINOR][PYTHON] Coerce the JVM error message to str in
ArrowCollectSerializer.load_stream
### What changes were proposed in this pull request?
Wrap the JVM error message in `str()` when building `messageParameters` in
`ArrowCollectSerializer.load_stream`.
### Why are the changes needed?
`PySparkRuntimeError` declares `messageParameters: Optional[Dict[str,
str]]`, but `UTF8Deserializer.loads` is unannotated and its body can return
`str`, raw bytes, or `None` (on a `SpecialLengths.NULL` length). Type checkers
that infer return types from unannotated bodies (Pyright, for instance, which
backs the Python language server in several editors) therefore report
`dict[str, str | Unknown | None]` as not assignable to `Dict[str, str]`.
`str()` makes the value match the declared [...]
`dev/lint-python` does not flag this: mypy treats a call to an unannotated
function as returning `Any`, and `pyspark.serializers` is exempt from
`disallow_untyped_defs` in `python/mypy.ini`.
### Does this PR introduce _any_ user-facing change?
No. The error framework already stringifies parameters during template
substitution, so the rendered message is identical.
### How was this patch tested?
Type checkers on the changed file: mypy with `dev/lint-python`'s invocation
stays clean, and Pyright goes from 1 error to 0.
Behavior was checked against a simulated collect stream (Arrow batches,
then the `-1` error signal, then the message): a normal message still renders
`An error occurred while calling ArrowCollectSerializer.load_stream:
java.lang.RuntimeException: boom.`, a NULL length renders the same `None` as
before the change, and out-of-order indices `(2, 0, 1)` still reorder correctly.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
Closes #58324 from Spenserrrr/arrow-collect-error-msg-str.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
(cherry picked from commit 29b60f527793f4ca0ce1f2a616fc1fdca409d32c)
Signed-off-by: Yicong-Huang <[email protected]>
---
python/pyspark/sql/pandas/serializers.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/sql/pandas/serializers.py
b/python/pyspark/sql/pandas/serializers.py
index 6e70e69fa2f4..b829e4a8636f 100644
--- a/python/pyspark/sql/pandas/serializers.py
+++ b/python/pyspark/sql/pandas/serializers.py
@@ -126,7 +126,7 @@ class ArrowCollectSerializer(ArrowStreamSerializer):
errorClass="ERROR_OCCURRED_WHILE_CALLING",
messageParameters={
"func_name": "ArrowCollectSerializer.load_stream",
- "error_msg": error_msg,
+ "error_msg": str(error_msg),
},
)
# Yield the batches in order, dropping our reference to each as it
goes so
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
