viirya commented on a change in pull request #28898:
URL: https://github.com/apache/spark/pull/28898#discussion_r448783457
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##########
@@ -152,7 +168,7 @@ object NestedColumnAliasing {
val exclusiveAttrSet = AttributeSet(exclusiveAttrs ++ otherRootReferences)
val aliasSub = nestedFieldReferences.asInstanceOf[Seq[ExtractValue]]
.filter(!_.references.subsetOf(exclusiveAttrSet))
- .groupBy(_.references.head)
+ .groupBy(t => (t.references.head.exprId, t.references.head.dataType))
Review comment:
Ok. Can you add a comment here?
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##########
@@ -172,13 +188,23 @@ object NestedColumnAliasing {
(f, Alias(f, s"_gen_alias_${exprId.id}")(exprId, Seq.empty, None))
}
+
+ // Do deduplication based on semanticEquals, and then sum.
+ val nestedFieldNum = nestedFieldToAlias
+ .foldLeft(Seq[ExtractValue]()) {
+ (unique, curr) => if (!unique.exists(curr._1.semanticEquals(_))) {
+ curr._1 +: unique
+ } else {
+ unique
+ }
+ }
+ .map { t => totalFieldNum(t.dataType) }
+ .sum
Review comment:
I see. I can see the bad thing about this is, you will have duplicate
aliases for the semantically same `ExtractValue`s, e.g. two aliases for two
`name.first`s.
It may not be a big deal here, but it is possibly in the optimized query
plan, you will have multiple gen_alias_xxx which refer to the same
`ExtractValue`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
