peter-toth commented on PR #57357: URL: https://github.com/apache/spark/pull/57357#issuecomment-5035680273
Thanks @cloud-fan. I don't think @szehon-ho was suggesting a connector-level enablement as the resolution — he raised it conditionally and seemed fine with a SQL conf otherwise. My own view is a bit stronger: allowing non-det pushdown at all was a mistake, so the default should just be to not push. That said, if a per-connector control is preferred over the session-wide flag, two options both fix the "too coarse" concern: (a) Drop the config and add a capability, e.g. `default boolean pushDownNondeterministicPredicates()` (default `false`) on `SupportsPushDownV2Filters`. Though I doubt any real connector wants non-det pushdown, so a new public API feels like overkill. (b) Keep a legacy config but make it a connector allowlist like `spark.sql.sources.useV1SourceList` (default empty), keyed on the source class name. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
