[
https://issues.apache.org/jira/browse/SPARK-61128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61128:
-----------------------------------
Labels: pull-request-available (was: )
> Make ArrayType, MapType, StructField and StructType give back and pickle the
> user-defined types they are made of, and StructField picklable, in the Python
> client as PySpark does
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61128
> URL: https://issues.apache.org/jira/browse/SPARK-61128
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark's ArrayType, MapType, StructField and StructType hold the type
> objects they are made of. So ArrayType(udt).elementType is udt, a
> user-defined type, and so are MapType(StringType(), udt).valueType,
> StructField("p", udt).dataType and the dataType of the fields of a
> StructType. A type pickles of those objects, and a user-defined type pickles
> by its class.
> The Python client of spark-connect-rust lowers these types into its Rust
> core, and rebuilds a user-defined type from its JSON, by importing its
> pyClass, whenever it gives the type back. The pyClass need not name the
> module of the class: for Apache Spark's ExamplePointUDT, it is
> pyspark.sql.tests.ExamplePointUDT, while the class is in
> pyspark.testing.objects. So on the client:
> - ArrayType(ExamplePointUDT()).elementType, and the types of the other
> containers above, raise PySparkValueError UNSUPPORTED_OPERATION ("unpickling
> user defined types is not supported")
> - so do their repr and simpleString, and
> udf(returnType=ArrayType(ExamplePointUDT())), which Apache Spark's
> test_udf_with_udt calls
> - the pickle of such a type, by its JSON, fails the same way on the server's
> Python worker
> Also, a StructField of the client cannot be pickled at all ("TypeError:
> cannot pickle 'pyspark.sql.types.StructField' object"), where PySpark's can.
> The client's types should give back the user-defined types they are made of,
> and pickle with them, and its StructField should pickle, as PySpark's do.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
