szehon-ho commented on PR #57357: URL: https://github.com/apache/spark/pull/57357#issuecomment-5028450921
Thanks for this @peter-toth ! I had some thoughts. Currently jdbc connector uses SupportsPushDown[V2]Filters, so it should not get the non-deterministic version anyway? But agree there could be some proprietary DSV2 connector out there that depend on it getting pushed down. So are we saying we want to support this feature fully (pushdown of non-deterministic filter)? If so, just wondering should we have a separate boolean API for each flavor - SupportsPushDownFIlter - SupportsPushDownV2Filter - SupportsPushDownCatalystFilter That makes more sense as the same Spark could be reading from multiple data source and one flag is too coarse. At the very least , we can make it an API that default to false on SupportsPushDownCatalystFilter? (to prevent behavior change as is the goal here) Or are we quite sure that not pushing down filter is always correct, then a sql conf make sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
