F. H. created SPARK-31374:
-----------------------------
Summary: Returning complex types in Pandas UDF
Key: SPARK-31374
URL: https://issues.apache.org/jira/browse/SPARK-31374
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 3.0.0
Reporter: F. H.
I would like to return a complex type in an GROUPED_AGG operation:
{code:python}
window_overlap_schema = t.StructType([
t.StructField("counts", t.ArrayType(t.LongType())),
t.StructField("starts", t.ArrayType(t.LongType())),
t.StructField("ends", t.ArrayType(t.LongType())),
])
@f.pandas_udf(window_overlap_schema, f.PandasUDFType.GROUPED_AGG)
def spark_window_overlap([...]):
[...]
{code}
However, I get the following error when trying to run this:
{code:python}
NotImplementedError: Invalid returnType with grouped aggregate Pandas UDFs:
StructType(List(StructField(counts,ArrayType(LongType,true),true),StructField(starts,ArrayType(LongType,true),true),StructField(ends,ArrayType(LongType,true),true)))
is not supported
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
