This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new ae7559aa3242 [SPARK-58529][4.2][PYTHON] Relax RESULT_ROWS_MISMATCH
assertion for cross-version old-client compatibility
ae7559aa3242 is described below
commit ae7559aa3242cb062a9071ed73474a4e3b64b112
Author: Yicong Huang <[email protected]>
AuthorDate: Tue Aug 4 23:48:45 2026 -0700
[SPARK-58529][4.2][PYTHON] Relax RESULT_ROWS_MISMATCH assertion for
cross-version old-client compatibility
### What changes were proposed in this pull request?
This backports the test-side change of SPARK-58529 to this maintenance
branch. It relaxes the `test_vectorized_udf_invalid_length` assertion in
`test_pandas_udf_scalar.py` to accept both this branch's
`SCHEMA_MISMATCH_FOR_PANDAS_UDF` message (`Result vector from pandas_udf was
not the required length`) and the `RESULT_ROWS_MISMATCH` message a newer server
raises (`The number of output rows ... must match the number of input rows`).
No production code changes -- this branch continues to raise its existing
message.
### Why are the changes needed?
The `pyspark-connect-old-client` cross-version job clones a maintenance
branch's tests and runs them against a newer (master) server. It currently pins
`branch-4.0`, but the base is expected to advance to `branch-4.1` /
`branch-4.2`. When it does, an assertion pinned to the old pandas-specific
substring would fail against a master server that no longer emits it (after
SPARK-58529). Relaxing the assertion now -- to match both messages -- keeps
this branch's own CI green today and preve [...]
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The relaxed regex was verified to match both this branch's message and the
newer `RESULT_ROWS_MISMATCH` message. Existing
`test_vectorized_udf_invalid_length` continues to run.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57777 from Yicong-Huang/backport-58529-branch-4.2.
Authored-by: Yicong Huang <[email protected]>
Signed-off-by: Yicong Huang <[email protected]>
---
python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py
b/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py
index 72d9fa566dee..28f6b506eedd 100644
--- a/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py
+++ b/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py
@@ -677,8 +677,13 @@ class ScalarPandasUDFTestsMixin:
def check_vectorized_udf_invalid_length(self):
df = self.spark.range(10)
raise_exception = pandas_udf(lambda _: pd.Series(1), LongType())
+ # Accept both this branch's SCHEMA_MISMATCH_FOR_PANDAS_UDF message and
the
+ # RESULT_ROWS_MISMATCH message a newer server raises (SPARK-58529), so
the
+ # cross-version old-client job passes against a master server.
with self.assertRaisesRegex(
- Exception, "Result vector from pandas_udf was not the required
length"
+ Exception,
+ "Result vector from pandas_udf was not the required length"
+ "|The number of output rows.*must match the number of input rows",
):
df.select(raise_exception(col("id"))).collect()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
