[
https://issues.apache.org/jira/browse/SPARK-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-23477.
----------------------------------
Resolution: Cannot Reproduce
I am resolving this as it's not reproduced in the master. It'd be nicer if we
could find the Jira fixing this and backports if applicable.
> Misleading exception message when union fails due to metadata
> --------------------------------------------------------------
>
> Key: SPARK-23477
> URL: https://issues.apache.org/jira/browse/SPARK-23477
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.1
> Reporter: Tomasz Bartczak
> Priority: Minor
>
> When I have two DF's that are different only in terms of metadata in fields
> inside a struct - I cannot union them but the error message shows that they
> are the same:
> {code:java}
> df = spark.createDataFrame([{'a':1}])
> a = df.select(struct('a').alias('x'))
> b =
> df.select(col('a').alias('a',metadata={'description':'xxx'})).select(struct(col('a')).alias('x'))
> a.union(b).printSchema(){code}
> gives:
> {code:java}
> An error occurred while calling o1076.union.
> : org.apache.spark.sql.AnalysisException: Union can only be performed on
> tables with the compatible column types. struct<a:bigint> <> struct<a:bigint>
> at the first column of the second table{code}
> and this part:
> {code:java}
> struct<a:bigint> <> struct<a:bigint>{code}
> does not make any sense because those are the same.
>
> Since metadata must be the same for union -> it should be incuded in the
> error message
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
