szehon-ho commented on PR #57357: URL: https://github.com/apache/spark/pull/57357#issuecomment-5028545927
The SupportsV2PushdownFilter is ugly, iiuc you are saying the first round translates non-deterministic filters, but second round PartiitonPredicate does not. I think if we go the multi-API approach, we can be more fine grain in keeping the old behavior and having a path forward to fix the new behavior. 1. SupportsPushDownFIlter.pushdownNonDeterministic (default false) 2. SupportsPushDownV2Filter.pushdownNonDeterministic (default true) 3. SupportsPushDownCatalystFilter.pushdownNonDeterministic (default true) We can just leave out the second round PartitionPredicate and say that never pushdown non-deterministic filter (unless there is a need later), wdyt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
