alamb commented on code in PR #7759:
URL: https://github.com/apache/arrow-datafusion/pull/7759#discussion_r1349500837
##########
datafusion/core/src/datasource/provider.rs:
##########
@@ -54,24 +54,87 @@ pub trait TableProvider: Sync + Send {
None
}
- /// Get the Logical Plan of this table, if available.
+ /// Get the [`LogicalPlan`] of this table, if available
fn get_logical_plan(&self) -> Option<&LogicalPlan> {
None
}
- /// Create an ExecutionPlan that will scan the table.
- /// The table provider will be usually responsible of grouping
- /// the source data into partitions that can be efficiently
- /// parallelized or distributed.
+ /// Create an [`ExecutionPlan`] for scanning the table with optionally
+ /// specified `projection`, `filter` and `limit`, described below.
+ ///
+ /// The `ExecutionPlan` is responsible scanning the datasource's
+ /// partitions in a streaming, parallelized fashion.
+ ///
+ /// # Projection
+ ///
+ /// If specified, only a subset of columns should be returned, in the order
+ /// specified. The projection is a set of indexes of the fields in
+ /// [`Self::schema`].
+ ///
+ /// DataFusion provides the projection to scan only the columns actually
+ /// used in the query to improve performance, an optimization called
+ /// "Projection Pushdown". Some datasources, such as Parquet, can use this
+ /// information to go significantly faster when only a subset of columns is
+ /// required.
+ ///
+ /// # Filters
+ ///
+ /// A list of boolean filter [`Expr`]s to evaluate *during* the scan, in
the
+ /// manner specified by [`Self::supports_filters_pushdown`]. Only rows for
+ /// which *all* of the `Expr`s evaluate to `true` must be returned (aka the
+ /// expressions are `AND`ed together).
+ ///
+ /// DataFusion pushes filtering into the scans whenever possible
+ /// ("Projection Pushdown"), and depending on the format and the
+ /// implementation of the format, evaluating the predicate during the scan
+ /// can increase performance significantly.
+ ///
+ /// ## Note: Some columns may appear *only* in Filters
+ ///
+ /// In certain cases, a query may only use a certain column in a Filter
that
+ /// has been completely pushed down to the scan. In this case, the
+ /// projection wll not contain all the columns found in the filter
Review Comment:
```suggestion
/// projection will not contain all the columns found in the filter
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
