This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new c1caba3d8b83 [SPARK-58529][4.0][PYTHON] Relax RESULT_ROWS_MISMATCH
assertion for cross-version old-client compatibility
c1caba3d8b83 is described below
commit c1caba3d8b83ce2a50c59cf332842cad1bebb3ee
Author: Yicong Huang <[email protected]>
AuthorDate: Wed Aug 5 00:15:08 2026 +0000
[SPARK-58529][4.0][PYTHON] Relax RESULT_ROWS_MISMATCH assertion for
cross-version old-client compatibility
### What changes were proposed in this pull request?
This backports the test-side change of SPARK-58529 to `branch-4.0`. It
relaxes the `test_vectorized_udf_invalid_length` assertion in
`test_pandas_udf_scalar.py` from the pandas-specific substring `"Result vector
from pandas_udf was not the required length"` to the shared `"The number of
output rows.*must match the number of input rows"` template text.
No production code changes -- `branch-4.0` continues to raise the message
with its trailing `Result vector from pandas_udf ...` clause.
### Why are the changes needed?
The `pyspark-connect-old-client` job on `master` clones `branch-4.0`'s
tests and runs them against a `master` server. After SPARK-58529 removed the
pandas-specific tail from the `RESULT_ROWS_MISMATCH` message on `master`,
`branch-4.0`'s assertion on that removed substring fails against the new
server. The relaxed regex matches both the old (`branch-4.0` server) and new
(`master` server) messages, so `branch-4.0`'s own CI and the cross-version
old-client job both pass.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The relaxed regex was verified to match both the old message (with the
pandas tail) and the new message (without it). Existing
`test_vectorized_udf_invalid_length` continues to run.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57772 from Yicong-Huang/backport-58529-branch-4.0.
Authored-by: Yicong Huang <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
---
python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py
b/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py
index d7af5dd14632..73d74af6b650 100644
--- a/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py
+++ b/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py
@@ -667,8 +667,13 @@ class ScalarPandasUDFTestsMixin:
def check_vectorized_udf_invalid_length(self):
df = self.spark.range(10)
raise_exception = pandas_udf(lambda _: pd.Series(1), LongType())
+ # Accept both this branch's SCHEMA_MISMATCH_FOR_PANDAS_UDF message and
the
+ # RESULT_ROWS_MISMATCH message a newer server raises (SPARK-58529), so
the
+ # cross-version old-client job passes against a master server.
with self.assertRaisesRegex(
- Exception, "Result vector from pandas_udf was not the required
length"
+ Exception,
+ "Result vector from pandas_udf was not the required length"
+ "|The number of output rows.*must match the number of input rows",
):
df.select(raise_exception(col("id"))).collect()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
