peter-toth commented on PR #57357: URL: https://github.com/apache/spark/pull/57357#issuecomment-5033474878
Thanks @szehon-ho! We should be really careful with non-deterministic expressions — the optimizer never moves or duplicates them on purpose. So IMO allowing them to be pushed down at all was a mistake. Pushing a non-det filter into a scan is a move, and if a source prunes with it *and* returns it for post-scan re-check, it's also a duplicate. As I see the three pushdown paths: - `SupportsPushDownFilters` (V1): never pushed non-det — `sources.Filter` can only carry a pushable column vs a literal, so `rand()` isn't even translatable. Nothing to toggle. - `SupportsPushDownCatalystFilters`: pushed non-det until SPARK-58112 fixed it. But this is an internal interface (`o.a.s.sql.internal.connector`), not publicly available, and only `FileScanBuilder` implements it — and built-in file sources default to V1 anyway (`spark.sql.sources.useV1SourceList`). So ~no blast radius; correct as-is, no flag needed. - `SupportsPushDownV2Filters`: the only path with real, tested non-det pushdown (the H2 `RAND(1) < bonus` case in `JDBCV2Suite`), so it's the only one that needs care as a behavior change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
