cloud-fan commented on issue #26150: [SPARK-29501][SQL] SupportsPushDownRequiredColumns should report pruned schema immediately URL: https://github.com/apache/spark/pull/26150#issuecomment-544375957 > The read schema may be impacted by filters. For example, if I project column a and filter by column b then the read schema can be (a, b) or just a. So the read schema can't be fully determined when the method to push the projection is called. In fact Spark always do column pruning at the end, so this can be determined then `pushRequiredColumns` is called.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
