szehon-ho commented on PR #57357: URL: https://github.com/apache/spark/pull/57357#issuecomment-5037742559
> Drop the config and add a capability, e.g. default boolean pushDownNondeterministicPredicates() (default false) on SupportsPushDownV2Filters. Though I doubt any real connector wants non-det pushdown, so a new public API feels like overkill. Yea that's what i had in mind, though it is still a public API? I just see the session-wide conf as too wide, if the goal is to ensure no behavior change for a specific DSV2 connector. I do realize its ugly, but Im leaning towards that at the moment. The whitelist seems more confusing to configure, but maybe there is precedent. Cc @aokolnychyi , will try to talk to him today about it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
