Holden Karau created SPARK-59090:
------------------------------------
Summary: Drop a transpiled option that fails to resolve, instead
of failing the query
Key: SPARK-59090
URL: https://issues.apache.org/jira/browse/SPARK-59090
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.3.0
Reporter: Holden Karau
`ResolveTranspiledPythonUDFOptions` prunes options whose declared categories do
not match the
bound column types. It does not check that a kept option's expression actually
*resolves*. When
one does not, `CheckAnalysis` raises INTERNAL_ERROR (SQLSTATE XX000) and the
query dies.
In the pruning collect, keep an option only if its expression resolves; drop it
otherwise, which
routes the UDF to interpreted Python exactly as a category mismatch already
does. Add a test that
a deliberately unresolvable option falls back rather than raising.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
