Holden Karau created SPARK-59881:
------------------------------------
Summary: Allow array-of-atomic return types through the
transpilation gate
Key: SPARK-59881
URL: https://issues.apache.org/jira/browse/SPARK-59881
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.4.0
Reporter: Holden Karau
`_transpile_func` turns down any non-atomic return type before asking any
transpiler:
```python
if not isinstance(returnType, (NumericType, StringType, BooleanType,
BinaryType)):
return ([], ["return type ... is not supported by the transpiler"], [], [],
[])
```
The reason — an unresolvable `Cast` fails the whole query at `CheckAnalysis`,
because options ride along as children of `TranspiledPythonUDF` — holds for
maps,
structs and datetimes. It does not hold for an array of an atomic: a variety can
produce a genuinely array-typed expression whose cast to the declared array type
resolves, and the interpreted converter takes lists.
Behaviour-neutral for the built-in transpiler, which still refuses every
array-typed body in its per-variant check, so only a bare-`None` body lowers and
`NULL` casts to any array cleanly. The diff's test pins that.
Land after SPARK-59880
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
