dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate URL: https://github.com/apache/spark/pull/24637#issuecomment-512358172 Thank you for adding a new test. Are you going to add more tests since `Stack` is one of them? In fact, we need more to be exhaustive. One another approach is simply reducing the scope to the original goal. We can match only-`Explode` like the following in this PR. ``` case p @ Project(projectList, g: Generate) if ... case p @ Project(projectList, g @ Generate(_: Explode, _, _, _, _, _)) if ... ``` Later, to cover more patterns, I think we need `unapply` and a white-list approach like the following. ``` private def canProjectPushThrough(plan: LogicalPlan) = plan match { case _: GlobalLimit => true ... case _ => false } ``` Since this PR is here for a long time, how about finishing here with `Explode` first, @viirya ?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
