maropu commented on a change in pull request #29587:
URL: https://github.com/apache/spark/pull/29587#discussion_r501036503
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2721,6 +2721,16 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val UNION_BYNAME_STRUCT_SUPPORT_ENABLED =
+ buildConf("spark.sql.unionByName.structSupport.enabled")
+ .doc("When true, the `allowMissingColumns` feature of
`Dataset.unionByName` supports " +
+ "nested column in struct types. Missing nested columns of struct
columns with same " +
+ "name will also be filled with null values. This currently does not
support nested " +
+ "columns in array and map types.")
Review comment:
Thanks for the update. Btw, to follow a top-level case (See an example
below), is it technically difficult to reorder struct fields by adding
`Project` on the top of `Union`? Does this make the logic ugly?
```
// The top-level case
scala> val df1 = spark.range(1).selectExpr("id c", "id a")
scala> val df2 = spark.range(1).selectExpr("id c", "id b")
scala> df1.unionByName(df2, true).printSchema
root
|-- c: long (nullable = false)
|-- a: long (nullable = true)
|-- b: long (nullable = true)
// The struct case (the current behaviour)
scala> val df3 = spark.range(1).selectExpr("named_struct('c', id, 'a', id)
col")
scala> val df4 = spark.range(1).selectExpr("named_struct('c', id, 'b', id)
col")
scala> df3.unionByName(df4, true).printSchema
root
|-- col: struct (nullable = false)
| |-- a: long (nullable = true)
| |-- b: long (nullable = true)
| |-- c: long (nullable = false)
// The following column order is more natural?
scala> df3.unionByName(df4, true).printSchema
root
|-- col: struct (nullable = false)
| |-- c: long (nullable = true)
| |-- a: long (nullable = true)
| |-- b: long (nullable = false)
== Analyzed Logical Plan ==
col: struct<c:bigint,a:bigint,b:bigint>
Project [struct(c, col#114.c, a, col#114.a, b, col#114.b) AS col#116] <===
Adding `Project` in the end of `ResolveUnion`?
+- Union false, false
:- Project [named_struct(a, col#87.a, b, null, c, col#87.c) AS col#114]
: +- Project [named_struct(c, id#85L, a, id#85L) AS col#87]
: +- Range (0, 1, step=1, splits=Some(4))
+- Project [named_struct(a, null, b, col#91.b, c, col#91.c) AS col#113]
+- Project [named_struct(c, id#89L, b, id#89L) AS col#91]
+- Range (0, 1, step=1, splits=Some(4))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
