viirya commented on a change in pull request #35850:
URL: https://github.com/apache/spark/pull/35850#discussion_r827211803
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##########
@@ -321,6 +321,40 @@ object GeneratorNestedColumnAliasing {
// need to prune nested columns through Project and under Generate. The
difference is
// when `nestedSchemaPruningEnabled` is on, nested columns will be pruned
further at
// file format readers if it is supported.
+
+ // There are [[ExtractValue]] expressions on or not on the output of the
generator. Generator
+ // can also have different types:
+ // 1. For [[ExtractValue]]s not on the output of the generator,
theoretically speaking, there
+ // lots of expressions that we can push down, including non
ExtractValues and GetArrayItem
+ // and GetMapValue. But to be safe, we only handle GetStructField and
GetArrayStructFields.
+ // 2. For [[ExtractValue]]s on the output of the generator, the situation
depends on the type
+ // of the generator expression.
+ // 2.1 Inline
Review comment:
We don't actually handle `Inline`. `canPruneGenerator` wrongly lists
`Stack` and `Inline`. Even they enter this rule, they won't be handled, I
think. It's better to make `canPruneGenerator` consistent with the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
