SaurabhChawla100 edited a comment on pull request #32972:
URL: https://github.com/apache/spark/pull/32972#issuecomment-866899685
There is one scenario which is failing for the allowMissing as true in the
existing master branch, This is the scenario where there is sorting done on the
case C2 and c1 here after sorting it gives C2, c1 where as on the other side if
c1, c2 so it fails with Union can only be performed on tables with the
compatible column types.
```
scala> case class Struct2(C2: Int, c1: Int)
defined class Struct2
scala> case class Struct1(c1: Int, c2: Int)
defined class Struct1
scala> var df2 = Seq((1, Struct2(1, 2))).toDF("a", "b")
df2: org.apache.spark.sql.DataFrame = [a: int, b: struct<C2: int, c1: int>]
scala> var df1 = Seq((1, Struct1(1, 2))).toDF("a", "b")
df1: org.apache.spark.sql.DataFrame = [a: int, b: struct<c1: int, c2: int>]
scala> var unionDF = df1.unionByName(df2, true)
org.apache.spark.sql.AnalysisException: Union can only be performed on
tables with the compatible column types. struct<C2:int,c1:int> <>
struct<c1:int,c2:int> at the second column of the second table;
'Union false, false
```
There is need to do a sorting on the lower case to handle this case.
After fix
```
scala> case class Struct2(C2: Int, c1: Int)
defined class Struct2
scala> case class Struct1(c1: Int, c2: Int)
defined class Struct1
scala> var df1 = Seq((1, Struct1(1, 2))).toDF("a", "b")
df1: org.apache.spark.sql.DataFrame = [a: int, b: struct<c1: int, c2: int>]
scala> var df2 = Seq((1, Struct2(1, 2))).toDF("a", "b")
df2: org.apache.spark.sql.DataFrame = [a: int, b: struct<C2: int, c1: int>]
scala> var unionDF = df1.unionByName(df2, true)
unionDF: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [a: int,
b: struct<c1: int, c2: int ... 1 more field>]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
