[
https://issues.apache.org/jira/browse/SPARK-60946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60946:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Send user-defined, VARIANT, GEOMETRY and GEOGRAPHY values in Spark Connect
> Rust createDataFrame as PySpark does
> ---------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60946
> URL: https://issues.apache.org/jira/browse/SPARK-60946
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect's {{createDataFrame}} converts these values of local rows to
> Arrow ({{LocalDataToArrowConversion}}). The Rust-backed Python client raises
> {{TypeError: unsupported value type for createDataFrame}} for each of them,
> so these PySpark 4.3.0 results differ:
> * A value of a user-defined type is sent as the serialized value of its SQL
> type: with {{ExamplePoint}} of {{pyspark.testing.objects}},
> {{createDataFrame([(ExamplePoint(1.0, 2.0),)])}} gives {{struct<_1:udt>}} and
> the row {{Row(_1=ExamplePoint(1.0,2.0))}}.
> * A {{VariantVal}} is sent as a VARIANT:
> {{createDataFrame([(spark.sql("SELECT parse_json('{\"a\":
> 1}')").first()[0],)])}} gives {{struct<_1:variant>}}, whose value's
> {{toJson()}} is {{'{"a":1}'}}.
> * A {{Geometry}} or a {{Geography}} is sent as a value of its type:
> {{createDataFrame([(Geometry.fromWKB(wkb, 4326),)])}} gives
> {{struct<_1:geometry(4326)>}}, and a {{Geography}}
> {{struct<_1:geography(4326)>}}.
> Their types are inferred as in PySpark with SPARK-60908. Send the values as
> PySpark does, in the Rust core and in the Python client, for inferred and
> explicit schemas.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
