alamb commented on code in PR #11994:
URL: https://github.com/apache/datafusion/pull/11994#discussion_r1719983009
##########
datafusion/core/src/datasource/physical_plan/parquet/row_filter.rs:
##########
@@ -243,13 +295,17 @@ impl<'a> TreeNodeRewriter for FilterCandidateBuilder<'a> {
Ok(Transformed::no(node))
}
+ /// After visiting all children, rewrite column references to nulls if
+ /// they are not in the file schema
fn f_up(
&mut self,
expr: Arc<dyn PhysicalExpr>,
) -> Result<Transformed<Arc<dyn PhysicalExpr>>> {
+ // if the expression is a column, is it in the file schema?
if let Some(column) = expr.as_any().downcast_ref::<Column>() {
if self.file_schema.field_with_name(column.name()).is_err() {
- // the column expr must be in the table schema
+ // Replace the column reference with a NULL (using the type
from the table schema)
+ // e.g. `column = 'foo'` is rewritten be transformed to `NULL
= 'foo'`
Review Comment:
I agree -- I tried to provide this information in c0b9012f9
##########
datafusion/core/src/datasource/physical_plan/parquet/row_filter.rs:
##########
@@ -15,6 +15,50 @@
// specific language governing permissions and limitations
// under the License.
+//! Utilities to push down of DataFusion filter predicates (any DataFusion
+//! `PhysicalExpr` that evaluates to a [`BooleanArray`]) to the parquet decoder
+//! level in `arrow-rs`.
+//!
+//! DataFusion will use a `ParquetRecordBatchStream` to read data from parquet
+//! into [`RecordBatch`]es.
+//!
+//! The `ParquetRecordBatchStream` takes an optional `RowFilter` which is
itself
+//! a Vec of `Box<dyn ArrowPredicate>`. During decoding, the predicates are
+//! evaluated in order, to generate a mask which is used to avoid decoding rows
+//! in projected columns which do not pass the filter which can significantly
+//! reduce the amount of compute required for decoding and thus improve query
+//! performance.
+//!
+//! Since the predicates are applied serially in the order defined in the
+//! `RowFilter`, the optimal ordering depends on the exact filters. The best
+//! filters to execute first have two properties:
+//!
+//! 1. They are relatively inexpensive to evaluate (e.g. they read
+//! column chunks which are relatively small)
+//!
+//! 2. They filter many (contiguous) rows, reducing the amount of decoding
+//! required for subsequent filters and projected columns
+//!
+//! If requested, this code will reorder the filters based on heuristics try
and
+//! reduce the evaluation cost.
+//!
+//! The basic algorithm for constructing the `RowFilter` is as follows
+//!
+//! 1. Break conjunctions into separate predicates. An expression
+//! like `a = 1 AND (b = 2 AND c = 3)` would be
+//! separated into the expressions `a = 1`, `b = 2`, and `c = 3`.
+//! 2. Determine whether each predicate can be evaluated as an
`ArrowPredicate`.
+//! 3. Determine, for each predicate, the total compressed size of all
+//! columns required to evaluate the predicate.
+//! 4. Determine, for each predicate, whether all columns required to
+//! evaluate the expression are sorted.
+//! 5. Re-order the predicate by total size (from step 3).
+//! 6. Partition the predicates according to whether they are sorted (from
step 4)
+//! 7. "Compile" each predicate `Expr` to a `DatafusionArrowPredicate`.
+//! 8. Build the `RowFilter` with the sorted predicates followed by
+//! the unsorted predicates. Within each partition, predicates are
Review Comment:
I think @thinkharderdev wrote it back in the day.
This PR just simplifies the wording slightly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
