bersprockets opened a new pull request #35232: URL: https://github.com/apache/spark/pull/35232
### What changes were proposed in this pull request? This PR updates the ExtractGenerator rule to extract a generator from a GeneratorOuter expression contained by a Generate operator. ### Why are the changes needed? This works: ``` select * from values 1, 2 lateral view outer explode(array()) as b; ``` But this does not work: ``` select * from values 1, 2 lateral view explode_outer(array()) as b; ``` It produces the error: ``` Error in query: Column 'b' does not exist. Did you mean one of the following? [col1]; line 1 pos 26; ``` This is because the parser directly creates a Generate operator with the (as of yet unresolved) generator function. Later, the ResolveFunctions rule converts the unresolved function to a resolved generator wrapped by a GeneratorOuter expression. While the ExtractGenerator rule will extract the generator function from a GeneratorOuter, it doesn't work if the GeneratorOuter is an expression in a Generate operator. Because the generator is not extracted, the ResolveGenerator rule fails to match on the Generate operator, and therefore fails to turn the generator's output expressions into attributes. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - Existing unit tests. - New unit test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
