F. H. created SPARK-31374: ----------------------------- Summary: Returning complex types in Pandas UDF Key: SPARK-31374 URL: https://issues.apache.org/jira/browse/SPARK-31374 Project: Spark Issue Type: Improvement Components: PySpark Affects Versions: 3.0.0 Reporter: F. H.
I would like to return a complex type in an GROUPED_AGG operation: {code:python} window_overlap_schema = t.StructType([ t.StructField("counts", t.ArrayType(t.LongType())), t.StructField("starts", t.ArrayType(t.LongType())), t.StructField("ends", t.ArrayType(t.LongType())), ]) @f.pandas_udf(window_overlap_schema, f.PandasUDFType.GROUPED_AGG) def spark_window_overlap([...]): [...] {code} However, I get the following error when trying to run this: {code:python} NotImplementedError: Invalid returnType with grouped aggregate Pandas UDFs: StructType(List(StructField(counts,ArrayType(LongType,true),true),StructField(starts,ArrayType(LongType,true),true),StructField(ends,ArrayType(LongType,true),true))) is not supported {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org