viirya commented on a change in pull request #31966:
URL: https://github.com/apache/spark/pull/31966#discussion_r613834905
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala
##########
@@ -351,6 +351,41 @@ abstract class SchemaPruningSuite
}
}
+ testSchemaPruning("SPARK-34638: nested column prune on generator output") {
+ val query1 = spark.table("contacts")
+ .select(explode(col("friends")).as("friend"))
Review comment:
The higher order function expression is a special case. I just looked at
it with more details. For example, `ArrayFilter` holds the complex expression
`friends`, so from the point of view of Spark, `friends` is thought to be used
by `ArrayFilter`.
I think it has some space to optimize it more, targeting higher order
function expression. It is out of the scope of this PR, however. I will take
some time to look at this after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
