[
https://issues.apache.org/jira/browse/SPARK-44479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takuya Ueshin updated SPARK-44479:
----------------------------------
Description:
Support UDTFs with empty schema, for example:
{code:python}
>>> class TestUDTF:
... def eval(self):
... yield tuple()
{code}
Currently it fails with `useArrow=True`:
{code:python}
>>> udtf(TestUDTF, returnType=StructType())().collect()
Traceback (most recent call last):
...
ValueError: not enough values to unpack (expected 2, got 0)
{code}
whereas without Arrow:
{code:python}
>>> udtf(TestUDTF, returnType=StructType(), useArrow=False)().collect()
[Row()]
{code}
Otherwise, we should raise an error without Arrow, too, to be consistent.
was:
Support UDTFs with empty schema, for example:
{code:python}
>>> class TestUDTF:
... def eval(self):
... yield tuple()
{code}
Currently it fails with `useArrow=True`:
{code:python}
>>> udtf(TestUDTF, returnType=StructType())().collect()
Traceback (most recent call last):
...
ValueError: not enough values to unpack (expected 2, got 0)
{code}
whereas without Arrow:
{code:python}
>>> udtf(TestUDTF, returnType=StructType(), useArrow=False)().collect()
[Row()]
{code}
Otherwise, we should raise an error without Arrow, too.
> Support Python UDTFs with empty schema
> --------------------------------------
>
> Key: SPARK-44479
> URL: https://issues.apache.org/jira/browse/SPARK-44479
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 3.5.0
> Reporter: Takuya Ueshin
> Priority: Major
>
> Support UDTFs with empty schema, for example:
> {code:python}
> >>> class TestUDTF:
> ... def eval(self):
> ... yield tuple()
> {code}
> Currently it fails with `useArrow=True`:
> {code:python}
> >>> udtf(TestUDTF, returnType=StructType())().collect()
> Traceback (most recent call last):
> ...
> ValueError: not enough values to unpack (expected 2, got 0)
> {code}
> whereas without Arrow:
> {code:python}
> >>> udtf(TestUDTF, returnType=StructType(), useArrow=False)().collect()
> [Row()]
> {code}
> Otherwise, we should raise an error without Arrow, too, to be consistent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
