[
https://issues.apache.org/jira/browse/SPARK-55723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yicong Huang updated SPARK-55723:
---------------------------------
Affects Version/s: 4.2.0
(was: 4.1.2)
> Generalize RESULT_COLUMNS_MISMATCH_FOR_ARROW_UDTF error in enforce_schema for
> all UDF types
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-55723
> URL: https://issues.apache.org/jira/browse/SPARK-55723
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.2.0
> Reporter: Yicong Huang
> Priority: Minor
>
> The `enforce_schema` method in `ArrowBatchTransformer` raises
> `RESULT_COLUMNS_MISMATCH_FOR_ARROW_UDTF` when Arrow type casting fails. This
> error class is UDTF-specific, but `enforce_schema` is a general-purpose
> utility that will be shared across other Arrow-based UDF types (e.g., scalar
> Arrow UDFs).
> We should introduce a more general error class (e.g.,
> `RESULT_COLUMNS_MISMATCH_FOR_ARROW_UDF`) and use it in `enforce_schema`, so
> the error message is appropriate regardless of the calling UDF type.
> See: https://github.com/apache/spark/pull/54296#discussion_r2861772381
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
