[
https://issues.apache.org/jira/browse/SPARK-44479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takuya Ueshin updated SPARK-44479:
----------------------------------
Fix Version/s: 3.5.0
> Support Python UDTFs with empty schema
> --------------------------------------
>
> Key: SPARK-44479
> URL: https://issues.apache.org/jira/browse/SPARK-44479
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 3.5.0
> Reporter: Takuya Ueshin
> Assignee: Takuya Ueshin
> Priority: Major
> Fix For: 3.5.0
>
>
> Support UDTFs with empty schema, for example:
> {code:python}
> >>> class TestUDTF:
> ... def eval(self):
> ... yield tuple()
> {code}
> Currently it fails with `useArrow=True`:
> {code:python}
> >>> udtf(TestUDTF, returnType=StructType())().collect()
> Traceback (most recent call last):
> ...
> ValueError: not enough values to unpack (expected 2, got 0)
> {code}
> whereas without Arrow:
> {code:python}
> >>> udtf(TestUDTF, returnType=StructType(), useArrow=False)().collect()
> [Row()]
> {code}
> Otherwise, we should raise an error without Arrow, too, to be consistent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
