phillipleblanc commented on code in PR #9923:
URL: https://github.com/apache/arrow-datafusion/pull/9923#discussion_r1549921253
##########
datafusion/core/src/datasource/provider.rs:
##########
@@ -154,28 +158,31 @@ pub trait TableProvider: Sync + Send {
limit: Option<usize>,
) -> Result<Arc<dyn ExecutionPlan>>;
- /// Tests whether the table provider can make use of a filter expression
- /// to optimise data retrieval.
- #[deprecated(since = "20.0.0", note = "use supports_filters_pushdown
instead")]
- fn supports_filter_pushdown(
- &self,
- _filter: &Expr,
- ) -> Result<TableProviderFilterPushDown> {
- Ok(TableProviderFilterPushDown::Unsupported)
- }
-
- /// Tests whether the table provider can make use of any or all filter
expressions
- /// to optimise data retrieval.
- /// Note: the returned vector much have the same size as the filters
argument.
- #[allow(deprecated)]
+ /// Specify if DataFusion should provide filter expressions to the
+ /// TableProvider to apply *during* the scan.
+ ///
+ /// The return value must have one element for each filter expression
passed
+ /// in. The value of each element indicates if the TableProvider can apply
+ /// that particular filter during the scan.
+ ///
+ /// Some TableProviders can evaluate filters more efficiently than the
+ /// `Filter` operator in DataFusion, for example by using an index.
Review Comment:
```suggestion
/// `Filter` operator in DataFusion, for example by using an index.
```
##########
datafusion/expr/src/table_source.rs:
##########
@@ -24,20 +24,29 @@ use datafusion_common::{Constraints, Result};
use std::any::Any;
-/// Indicates whether and how a filter expression can be handled by a
-/// TableProvider for table scans.
+/// Indicates how a filter expression is handled by
+/// [`TableProvider::scan`].
+///
+/// Filter expressions are boolean expressions used to reduce the number of
+/// rows that are read from a table. Only rows that evaluate to `true` ("pass
+/// the filter") are returned. Rows that evaluate to `false` or `NULL` are
+/// omitted.
+///
+/// [`TableProvider::scan`]:
https://docs.rs/datafusion/latest/datafusion/datasource/provider/trait.TableProvider.html#tymethod.scan
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum TableProviderFilterPushDown {
- /// The expression cannot be used by the provider.
+ /// The filter cannot be used by the provider and will not be pushed down.
Unsupported,
- /// The expression can be used to reduce the data retrieved,
- /// but the provider cannot guarantee it will omit all tuples that
- /// may be filtered. In this case, DataFusion will apply an additional
- /// `Filter` operation after the scan to ensure all rows are filtered
correctly.
+ /// The filter can be used, but the provider might still return some tuples
+ /// that do not pass the filter.
+ ///
+ /// In this case, DataFusion applies an additional `Filter` operation
+ /// after the scan to ensure all rows are filtered correctly.
Inexact,
- /// The provider **guarantees** that it will omit **all** tuples that are
- /// filtered by the filter expression. This is the fastest option, if
available
- /// as DataFusion will not apply additional filtering.
+ /// The provider **guarantees** that it will omit **only** tuples which
Review Comment:
```suggestion
/// The provider **guarantees** that it will omit **only** tuples which
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
