Hyukjin Kwon created SPARK-61152:
------------------------------------
Summary: mapInPandas, mapInArrow and the grouped and cogrouped
applyIn functions accept a non-struct return type
Key: SPARK-61152
URL: https://issues.apache.org/jira/browse/SPARK-61152
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
mapInPandas, mapInArrow, groupBy.applyInPandas, applyInArrow,
applyInPandasWithState and cogroup.applyInPandas/applyInArrow take a non-struct
return type such as "int", IntegerType() or ArrayType(...) and send the plan to
the server, which fails with UnknownException (ClassCastException) or
IllegalArgumentException "The returnType of the udf must be a struct". PySpark
checks the return type when it builds the UDF and raises PySparkTypeError
INVALID_RETURN_TYPE_FOR_PANDAS_UDF (INVALID_RETURN_TYPE_FOR_ARROW_UDF for the
Arrow functions) on the client, at the call. Repro: df.mapInPandas(f, "int").
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
