[ https://issues.apache.org/jira/browse/SPARK-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-23477. ---------------------------------- Resolution: Cannot Reproduce I am resolving this as it's not reproduced in the master. It'd be nicer if we could find the Jira fixing this and backports if applicable. > Misleading exception message when union fails due to metadata > -------------------------------------------------------------- > > Key: SPARK-23477 > URL: https://issues.apache.org/jira/browse/SPARK-23477 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.1 > Reporter: Tomasz Bartczak > Priority: Minor > > When I have two DF's that are different only in terms of metadata in fields > inside a struct - I cannot union them but the error message shows that they > are the same: > {code:java} > df = spark.createDataFrame([{'a':1}]) > a = df.select(struct('a').alias('x')) > b = > df.select(col('a').alias('a',metadata={'description':'xxx'})).select(struct(col('a')).alias('x')) > a.union(b).printSchema(){code} > gives: > {code:java} > An error occurred while calling o1076.union. > : org.apache.spark.sql.AnalysisException: Union can only be performed on > tables with the compatible column types. struct<a:bigint> <> struct<a:bigint> > at the first column of the second table{code} > and this part: > {code:java} > struct<a:bigint> <> struct<a:bigint>{code} > does not make any sense because those are the same. > > Since metadata must be the same for union -> it should be incuded in the > error message -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org