fqaiser94 commented on a change in pull request #27066:
URL: https://github.com/apache/spark/pull/27066#discussion_r441229325
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala
##########
@@ -39,7 +41,18 @@ object SimplifyExtractValueOps extends Rule[LogicalPlan] {
// Remove redundant field extraction.
case GetStructField(createNamedStruct: CreateNamedStruct, ordinal, _) =>
createNamedStruct.valExprs(ordinal)
-
+ case GetStructField(WithFields(struct, nameExprs, valExprs), ordinal,
maybeName) =>
+ val extractFieldName = maybeName.getOrElse(
+ struct.dataType.asInstanceOf[StructType](ordinal).name)
+ val resolver = SQLConf.get.resolver
+ val names = nameExprs.map(e => e.eval().toString)
+ if (names.exists(n => resolver(n, extractFieldName))) {
Review comment:
You're right, this is more concise. To give the same behaviour though, I
had to change it slightly:
```
val matches = names.zip(valExprs).filter { case (name, _) => resolver(name,
extractFieldName) }
if (matches.nonEmpty) {
matches.last._2
} else {
GetStructField(struct, ordinal, Some(extractFieldName))
}
```
Reason: It's possible for there to be multiple matches and we only want the
last one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
