maropu commented on a change in pull request #28988:
URL: https://github.com/apache/spark/pull/28988#discussion_r449868997
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##########
@@ -174,9 +176,12 @@ object NestedColumnAliasing {
// If all nested fields of `attr` are used, we don't need to introduce
new aliases.
// By default, ColumnPruning rule uses `attr` already.
+ // Note that we need to remove cosmetic variations first, so we only
count a
+ // nested field once.
if (nestedFieldToAlias.nonEmpty &&
- nestedFieldToAlias
- .map { case (nestedField, _) =>
totalFieldNum(nestedField.dataType) }
+ dedupNestedFields.map(_.canonicalized.asInstanceOf[ExtractValue])
Review comment:
btw, we cannot avoid the generating duplicated aliases (`name#1.first AS
_gen_alias_52#52` and `name#1.first AS _gen_alias_53#53`) below? Is this
technically difficult? (This is not related to this PR and just a question)
```
scala> sql("select name.first from contact_alias").explain()
== Physical Plan ==
*(2) Project [_gen_alias_52#52 AS first#50]
+- Exchange hashpartitioning(_gen_alias_53#53, _gen_alias_54#54, 100),
false, [id=#46]
+- *(1) Project [name#1.first AS _gen_alias_52#52, name#1.first AS
_gen_alias_53#53, name#1.last AS _gen_alias_54#54]
+- FileScan parquet [name#1,p#8] Batched: false, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex[file:/tmp/contacts],
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<name:struct<first:string,last:string>>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
