HeartSaVioR commented on PR #59943: URL: https://github.com/apache/spark/pull/59943#issuecomment-6514691027
I just read through the recent commit - so the commit was not really to prune the columns but to fix the issue with metadata column. https://github.com/apache/spark/commit/2adc69aca3de7415d011fbe4b3dedaa3a987e8d5 > 3. **This change fixes metadata column reads only, not column pruning.** > We call `pruneColumns(output.toStructType)` where `output` is the full analyzed relation > output — all data columns plus any metadata columns added by `AddMetadataColumns`. This > communicates required metadata columns to the scan builder so they appear in `readSchema()`, > but does not prune data columns. Full column pruning in streaming, along with filter and > aggregate pushdown, is deferred to the existing TODO. I'm not sure this commit is beyond the above commit. It doesn't still seem to address the TODO the commit message stated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
