Alwaysgaurav1 opened a new pull request, #57203:
URL: https://github.com/apache/spark/pull/57203
### What changes were proposed in this pull request?
This PR resolves a TODO in `python/pyspark/worker.py` to standardize row
count mismatch error reporting by using the unified `RESULT_ROWS_MISMATCH`
error class instead of the legacy `SCHEMA_MISMATCH_FOR_PANDAS_UDF` class.
Specifically:
- Refactored the local `verify_result_length` helper in
`wrap_scalar_pandas_udf` to raise `RESULT_ROWS_MISMATCH` and passed matching
`messageParameters` (`output_length` and `input_length`).
- Refactored the second local `verify_result_length` helper in
`wrap_arrow_batch_udf` to also raise `RESULT_ROWS_MISMATCH`.
- Removed the deprecated `"SCHEMA_MISMATCH_FOR_PANDAS_UDF"` entry from the
python error conditions registry
(`python/pyspark/errors/error-conditions.json`).
### Why are the changes needed?
Previously, PySpark raised a custom `SCHEMA_MISMATCH_FOR_PANDAS_UDF` error
class during UDF row count mismatches. Standardizing this error class to
`RESULT_ROWS_MISMATCH` makes error codes and message patterns uniform across
PySpark.
### Does this PR introduce _any_ user-facing change?
Yes, the error class raised on a Pandas/Arrow UDF row count mismatch changes:
- **Previous Error**: `[SCHEMA_MISMATCH_FOR_PANDAS_UDF] Result vector from
pandas_udf was not the required length: expected 1, got 0.`
- **New Error**: `[RESULT_ROWS_MISMATCH] The number of output rows (0) must
match the number of input rows (1).`
### How was this patch tested?
1. Ran unit tests targeting Pandas UDF execution:
```bash
python/run-tests.py --python-executables venv/bin/python --testnames
pyspark.sql.tests.pandas.test_pandas_udf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
