HyukjinKwon commented on PR #59943: URL: https://github.com/apache/spark/pull/59943#issuecomment-6640777245
Thank you, @dongjoon-hyun. becbceb addresses the fifth round; the replies are in the threads. The two most important items changed the design: - **The pushdowns are opt-in**, with two new table capabilities, `STREAMING_COLUMN_PRUNING` and `STREAMING_FILTER_PUSHDOWN`, by which a table declares that the streams of its scans return the columns of the read schema, and that its scan builders can take filters for a streaming scan. A table that declares neither gets the same calls as before this PR, so nothing changes for an existing connector, and the migration guide entries are gone. Python tables do not declare the filter one, which replaces `BatchOnlyFilterPushDown`, and a third-party builder that needs a batch read for its filters leaves it out too. - **Each micro-batch applies the pushed filters** to the rows of each read of the relation, with a `Filter` of all the filters that the scan builder got, which the query applies anyway, so the scan cannot drop a row which the plan keeps, as you suggested on L1160. The check of the filters, which failed the queries whose micro-batches fold the pushed filters in part, is removed, with `implies`, `isTrue` and the record of the taken filters. The planner still checks the columns. Also: the fallback to a scan of all the columns with no filter covers every call of the pushdown, `StartupValue` keeps its expression, only casts are checked with `ComputeCurrentTime`, and `ColumnarScan` keeps a failure to prune, returns the same projected batch until the next one, and shows `SourceReadSchema` when it drops columns after the read. Your three queries with partial folding fail with `INTERNAL_ERROR` on 69d9c5e and run now. The streaming, columnar, native and Python streaming suites pass locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
