Hyukjin Kwon created SPARK-60946:
------------------------------------
Summary: Send user-defined, VARIANT, GEOMETRY and GEOGRAPHY values
in Spark Connect Rust createDataFrame as PySpark does
Key: SPARK-60946
URL: https://issues.apache.org/jira/browse/SPARK-60946
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect's {{createDataFrame}} converts these values of local rows to
Arrow ({{LocalDataToArrowConversion}}). The Rust-backed Python client raises
{{TypeError: unsupported value type for createDataFrame}} for each of them, so
these PySpark 4.3.0 results differ:
* A value of a user-defined type is sent as the serialized value of its SQL
type: with {{ExamplePoint}} of {{pyspark.testing.objects}},
{{createDataFrame([(ExamplePoint(1.0, 2.0),)])}} gives {{struct<_1:udt>}} and
the row {{Row(_1=ExamplePoint(1.0,2.0))}}.
* A {{VariantVal}} is sent as a VARIANT: {{createDataFrame([(spark.sql("SELECT
parse_json('{\"a\": 1}')").first()[0],)])}} gives {{struct<_1:variant>}}, whose
value's {{toJson()}} is {{'{"a":1}'}}.
* A {{Geometry}} or a {{Geography}} is sent as a value of its type:
{{createDataFrame([(Geometry.fromWKB(wkb, 4326),)])}} gives
{{struct<_1:geometry(4326)>}}, and a {{Geography}}
{{struct<_1:geography(4326)>}}.
Their types are inferred as in PySpark with SPARK-60908. Send the values as
PySpark does, in the Rust core and in the Python client, for inferred and
explicit schemas.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
