GitHub user ryux1 added a comment to the discussion: Automating predicate pushdown via declared capabilities
There is already a capability negotiation point for scan filters, but it is expression-level rather than a static declaration: `TableProvider::supports_filters_pushdown(&[&Expr])` returns `Exact`, `Inexact`, or `Unsupported` for each candidate. The logical filter-pushdown rule uses those answers to decide which predicates may move into `TableScan.filters` and which must remain above the scan. That dynamic shape is important. Support often depends on more than an operator name: column type, casts, function volatility, nested fields, null semantics, collation, connector version, and combinations such as `a = 1 AND unsupported_udf(b)`. A declaration such as “supports equality and ranges” would still need a shared expression matcher precise enough to encode those constraints, or it risks incorrect `Exact` claims that remove a necessary residual filter. A useful Capabilities API could therefore be additive: reusable predicates/builders that implement common expression subsets and produce the existing per-expression `TableProviderFilterPushDown` results. That would remove repeated tree-walking code without replacing the correctness boundary. Providers with backend-specific semantics could still override the dynamic method. Join pushdown is a different layer. A `TableProvider` represents one scan and does not currently own a general join-capability contract. Pushing a join into an external engine needs to establish that both inputs share a compute context and that the engine can preserve the join type, predicates, casts, and output schema. That fits a federation/extension optimizer (such as `datafusion-federation`) better than a scan-only `TableProvider` flag. So the short version is: automatic pruning already exists after providers classify filters; shared declarative helpers could make that classification much easier, but a fully static list is probably not expressive enough, and joins need a plan-level federation capability rather than only table metadata. GitHub link: https://github.com/apache/datafusion/discussions/19566#discussioncomment-18317600 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
