szehon-ho commented on PR #57727: URL: https://github.com/apache/spark/pull/57727#issuecomment-5209104867
Thanks @peter-toth for the second round -- findings 9 through 15 are all addressed in the latest revision, replies inline. The one place I did not follow the suggestion is finding 9. Instead of ordering the match so `SupportsRuntimeV2Filtering` wins, a scan implementing both interfaces is now rejected with an internal error, checked from `runtimeFilterAttrs`, `fullyPushedRuntimeFilterAttrs` and `pushRuntimeFilters`. Uniform precedence makes the plan safe but silently ignores `fullyPushedFilterAttributes()`; the contract already says only one interface may be implemented, so this makes Spark enforce what the Javadoc asks for. Finding 15 also invalidated the motivating example in the PR description -- `part > (subquery) + 1` does translate to a V2 predicate. The description now leads with `RLIKE`, and the suite covers both: one test asserting there is no V2 translation, one asserting the arithmetic shape survives. @cloud-fan your comment is answered on the thread: the rebase onto master picks up #57760, so a non-deterministic filter never becomes a fully pushed candidate, and the drop from `postScanFilters` now applies the same pushability test as the push itself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
