HyukjinKwon commented on PR #59943: URL: https://github.com/apache/spark/pull/59943#issuecomment-6637132245
Thank you, @dongjoon-hyun and @gengliangwang. 3a71891 addresses this round; the replies are in the threads. In short: - **The check of the pushed filters accepts what the filters of a micro-batch imply**: either they contain the pushed filter, or it holds once their `column = literal` equalities are put in place of the columns and the constants are folded. That is what `ConstantPropagation` does before it drops a filter that became true, both with an equality inferred through a join (`id > 1` with `id = 5`) and with the literals of `now()` or `current_user()`. New tests run all the reported queries. They failed with the containment check of 9ef62c1, and a filter that a micro-batch no longer implies at all still fails loudly. - **Only the filters that the scan builder took** (its `pushedFilters()` / `pushedPredicates()`) are recorded and shown. - **Inline tables** become placeholders at the start, like the relations of batch scans, and the time values follow `ComputeCurrentTime` itself instead of a copy of its guards. - **Python**: `PythonScanBuilder` declares a new internal mix-in, `BatchOnlyFilterPushDown`, rather than being matched by class. - The fallback warning keeps the throwable, `StreamingScan` lost its unused type parameter, and the test readers use `ProjectingInternalRow`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
