[
https://issues.apache.org/jira/browse/SPARK-24943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17414639#comment-17414639
]
Hyukjin Kwon commented on SPARK-24943:
--------------------------------------
uniontype is not supported in Spark at all. For varchar and char, they are not
supported from Spark 3.0:
{code}
scala> spark.createDataFrame(Seq("a").toDS.rdd.map(r =>
org.apache.spark.sql.Row(r)),
org.apache.spark.sql.types.StructType.fromDDL("fullName varchar(10)"))
org.apache.spark.sql.AnalysisException: char/varchar type can only be used in
the table schema. You can set spark.sql.legacy.charVarcharAsString to true, so
that Spark treat them as string type as same as Spark 3.0 and earlier
at
org.apache.spark.sql.errors.QueryCompilationErrors$.charOrVarcharTypeAsStringUnsupportedError(QueryCompilationErrors.scala:1614)
at
org.apache.spark.sql.catalyst.util.CharVarcharUtils$.failIfHasCharVarchar(CharVarcharUtils.scala:64)
at
org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$3(SparkSession.scala:354)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:353)
... 47 elided
{code}
> Convert a SQL Struct to StructType
> ----------------------------------
>
> Key: SPARK-24943
> URL: https://issues.apache.org/jira/browse/SPARK-24943
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.1
> Reporter: mahmoud mehdi
> Priority: Minor
> Fix For: 2.4.0
>
>
> The main goal of this User Story is to add a method to StructType which does
> the opposite to what does the sql method.
> For example, for the following SQL Struct :
> {code:java}
> df.schema.sql
> //STRUCT<`price`: STRUCT<`amount`: BIGINT, `currency`: STRING>>{code}
> We'll have the following output :
> {code:java}
> StructType.fromSql(df.schema.sql)
> //StructType(StructField(price,StructType(StructField(amount,LongType,true),
> //StructField(currency,StringType,true)),true))
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
