HyukjinKwon commented on PR #59943: URL: https://github.com/apache/spark/pull/59943#issuecomment-6656816297
Thank you, @dongjoon-hyun and @zikangh. As suggested, 9cbc7b8 moves the filter pushdown to SPARK-60926, in the draft #60129, which is based on this PR, so this PR only prunes the columns. 89114e3 addresses the comments on the pruning; the replies are in the threads. In short: - Every cast of a string to a timestamp with local time zone, at any depth of its types and whatever its child is, becomes a placeholder when the query starts, so the columns of a scan do not depend on the date of the start, also when the optimizer only makes the string a literal later, as through an alias. - The real-time mode is checked before the output of the scan node. - The docs say that Spark asks a pruned scan for its read schema before it creates the stream. - The fallback to a scan of all the columns stays; the reasons are in the thread. The comments on the filters are open items in #60129. The title and the description of this PR are updated. --- <!-- GITHUB_MCP_FOOTER: This attribution is automatically appended by GitHub MCP. --> _This comment was generated with [GitHub MCP](http://go/mcps)._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
