gaurav kumar pandey created SPARK-58161:
-------------------------------------------
Summary: [PYTHON] Standardize PySpark UDF row count mismatch error
to use RESULT_ROWS_MISMATCH
Key: SPARK-58161
URL: https://issues.apache.org/jira/browse/SPARK-58161
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.0.0
Reporter: gaurav kumar pandey
Currently, PySpark Pandas and Arrow UDFs raise a custom
SCHEMA_MISMATCH_FOR_PANDAS_UDF error class when the returned result length does
not match the input row count.
To align with Spark's error standardization efforts, we should migrate this to
the standard RESULT_ROWS_MISMATCH error class and remove the redundant
SCHEMA_MISMATCH_FOR_PANDAS_UDF from error-conditions.json.
Proposed Changes:
1. Migrate UDF scalar and arrow verification logic in worker.py to raise
RESULT_ROWS_MISMATCH with input/output length parameters.
2. Retain the old error message format ("Result vector from pandas_udf was not
the required length...") in the message string for backward-compatibility with
user code that asserts on the exception message.
3. Update affected unit tests, documentation guidelines (debugging.rst), and
clean up error-conditions.json.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
