Tim Meehan created SPARK-61291:
----------------------------------
Summary: Let VariantVal be subclassed in the Python client, as
PySpark's can
Key: SPARK-61291
URL: https://issues.apache.org/jira/browse/SPARK-61291
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
PySpark's VariantVal (pyspark.sql.types) is a plain Python class: it can be
subclassed, and createDataFrame takes an instance of a subclass as a variant.
In the Python client of spark-connect-rust, VariantVal is a pyo3 class declared
without subclass (crates/pyspark-rs/src/values.rs:9), so the class statement of
a subclass raises TypeError.
{code:python}
from pyspark.sql import SparkSession
from pyspark.sql.types import VariantVal
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
v = spark.sql("""SELECT parse_json('{"a": 1}') AS v""").collect()[0].v
class TaggedVariant(VariantVal):
pass
print(spark.createDataFrame([(TaggedVariant(v.value, v.metadata),)], "v
variant").collect()[0].v)
{code}
Against an OSS Spark 4.3.0 Connect server on loopback:
* pyspark-client 4.3.0: \{"a":1\}
* spark-connect-rust master 8768cc4: TypeError: type
'pyspark.sql.types.VariantVal' is not an acceptable base type, raised by the
class statement.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
