Takuya Ueshin created SPARK-50063:
-------------------------------------
Summary: Support Variant in Spark Connect Scala client
Key: SPARK-50063
URL: https://issues.apache.org/jira/browse/SPARK-50063
Project: Spark
Issue Type: Sub-task
Components: Connect
Affects Versions: 4.0.0
Reporter: Takuya Ueshin
Currently Spark Connect Scala client doesn't support Variant type when
collecting data.
{code:scala}
scala> spark.sql("select parse_json('1')").collect()
org.apache.spark.SparkUnsupportedOperationException: [UNSUPPORTED_DATATYPE]
Unsupported data type "VARIANT". SQLSTATE: 0A000
org.apache.spark.sql.errors.ExecutionErrors.unsupportedDataTypeError(ExecutionErrors.scala:159)
org.apache.spark.sql.errors.ExecutionErrors.unsupportedDataTypeError$(ExecutionErrors.scala:156)
org.apache.spark.sql.errors.ExecutionErrors$.unsupportedDataTypeError(ExecutionErrors.scala:254)
org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.deserializerFor(ArrowDeserializer.scala:370)
...
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
