aokolnychyi commented on PR #58145: URL: https://github.com/apache/spark/pull/58145#issuecomment-6533720658
I am not convinced the design we landed with is the right choice. It feels too complicated and yet still doesn't do the right thing. Specifically, we want the inferred filters to compound with other filters but not with the filters that they were inferred from. Today, this is not true. We compare the original filter condition with only inferred filters condition, completely ignoring how these inferred filters would compound with other unrelated predicates. We completely loose the mapping. The motivation for adding inferred filters is to support generated columns and hidden partitioning. I feel like the right approach here is to keep the solution simple and add all inferred predicates to Filter and accept the double counting as the temp limitation. Connectors will have to explicitly opt in and having this would be already better than not adjusting at all. If we plan to infer predicates on the Spark side in the future, maybe, it is not the right call to add this capability at all. If we feel connectors will always be smarter than Spark, then we should structure the inferred filters API in a way that specifies the mapping (e.g. say expose a map of source predicates to a sequence of inferred predicates and teach the visitors to dedup and do the right calculation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
