[ https://issues.apache.org/jira/browse/SPARK-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371238#comment-16371238 ]
Marco Gaido commented on SPARK-23477: ------------------------------------- I cannot reproduce this on master. > Misleading exception message when union fails due to metadata > -------------------------------------------------------------- > > Key: SPARK-23477 > URL: https://issues.apache.org/jira/browse/SPARK-23477 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.1 > Reporter: Tomasz Bartczak > Priority: Minor > > When I have two DF's that are different only in terms of metadata in fields > inside a struct - I cannot union them but the error message shows that they > are the same: > {code:java} > df = spark.createDataFrame([{'a':1}]) > a = df.select(struct('a').alias('x')) > b = > df.select(col('a').alias('a',metadata={'description':'xxx'})).select(struct(col('a')).alias('x')) > a.union(b).printSchema(){code} > gives: > {code:java} > An error occurred while calling o1076.union. > : org.apache.spark.sql.AnalysisException: Union can only be performed on > tables with the compatible column types. struct<a:bigint> <> struct<a:bigint> > at the first column of the second table{code} > and this part: > {code:java} > struct<a:bigint> <> struct<a:bigint>{code} > does not make any sense because those are the same. > > Since metadata must be the same for union -> it should be incuded in the > error message -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org