HeartSaVioR commented on PR #59943: URL: https://github.com/apache/spark/pull/59943#issuecomment-6515284986
1. Personally I prefer we do prune the columns in optimization phase. If there is technical challenge that we can't, let's probably list it out. 2. If we reach consensus that we want to prune the columns based on analysis result (or run some simple logic to track the columns, which the PR seems to do now), tests against the logic must be exhaustive, e.g. multiple transformations with and without expressions which do calculation and/or SQL function call, multiple phases of assigning alias, from_json, struct and de-struct, etcetc. Basically we have lots of operators in the query, so if we want to correlate the query output and source's columns, we have to be 100% sure we do not hit anything on operators on top of the source. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
