[
https://issues.apache.org/jira/browse/SPARK-61291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Meehan updated SPARK-61291:
-------------------------------
Fix Version/s: connect-rust-4.3.0
> Let VariantVal be subclassed in the Python client, as PySpark's can
> -------------------------------------------------------------------
>
> Key: SPARK-61291
> URL: https://issues.apache.org/jira/browse/SPARK-61291
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Tim Meehan
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark's VariantVal (pyspark.sql.types) is a plain Python class: it can be
> subclassed, and createDataFrame takes an instance of a subclass as a variant.
> In the Python client of spark-connect-rust, VariantVal is a pyo3 class
> declared without subclass (crates/pyspark-rs/src/values.rs:9), so the class
> statement of a subclass raises TypeError.
> {code:python}
> from pyspark.sql import SparkSession
> from pyspark.sql.types import VariantVal
> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> v = spark.sql("""SELECT parse_json('{"a": 1}') AS v""").collect()[0].v
> class TaggedVariant(VariantVal):
> pass
> print(spark.createDataFrame([(TaggedVariant(v.value, v.metadata),)], "v
> variant").collect()[0].v)
> {code}
> Against an OSS Spark 4.3.0 Connect server on loopback:
> * pyspark-client 4.3.0: \{"a":1\}
> * spark-connect-rust master 8768cc4: TypeError: type
> 'pyspark.sql.types.VariantVal' is not an acceptable base type, raised by the
> class statement.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
