xudong963 commented on code in PR #18868:
URL: https://github.com/apache/datafusion/pull/18868#discussion_r2670791839
##########
datafusion/datasource-parquet/src/row_group_filter.rs:
##########
@@ -70,6 +79,109 @@ impl RowGroupAccessPlanFilter {
self.access_plan
}
+ /// Returns the is_fully_matched vector
+ pub fn is_fully_matched(&self) -> &Vec<bool> {
+ &self.is_fully_matched
+ }
+
+ /// Prunes the access plan based on the limit and fully contained row
groups.
+ ///
+ /// The pruning works by leveraging the concept of fully matched row
groups. Consider a query like:
+ /// `WHERE species LIKE 'Alpine%' AND s >= 50 LIMIT N`
+ ///
+ /// After initial filtering, row groups can be classified into three
states:
+ ///
+ /// 1. Not Matching / Pruned
+ /// 2. Partially Matching (Row Group/Page contains some matches)
+ /// 3. Fully Matching (Entire range is within predicate)
+ ///
+ ///
+-----------------------------------------------------------------------+
+ /// | NOT MATCHING
|
+ /// | Partition 1
|
+ /// | +-----------------------------------+-----------------------------+
|
+ /// | | SPECIES (min: 'B...',max: 'S...') | S (min: 7, max: 133) |
|
+ /// | +-----------------------------------+-----------------------------+
|
+ /// | | Snow Vole | 7 |
|
+ /// | | Brown Bear | 133 |
|
+ /// | | Gray Wolf | 82 |
|
+ /// | +-----------------------------------+-----------------------------+
|
+ ///
+-----------------------------------------------------------------------+
+ ///
+ ///
+-----------------------------------------------------------------------+
+ /// | PARTIALLY MATCHING
|
+ /// | Partition 2 Partition 4
|
+ /// | +------------------+--------------+
+------------------+-------+ |
+ /// | | SPECIES | S | | SPECIES | S
| |
+ /// | | (min:A, max:R) |(min:6,max:70)| | (min:A, max:P) |[4-51]
| |
+ /// | +------------------+--------------+
+------------------+-------+ |
+ /// | | Lynx | 71 | | Europ. Mole | 4
| |
+ /// | | Red Fox | 40 | | Polecat | 16
| |
+ /// | | Alpine Bat | 6 | | Alpine Ibex | 97
| |
+ /// | +------------------+--------------+
+------------------+-------+ |
+ ///
+-----------------------------------------------------------------------+
+ ///
+ ///
+-----------------------------------------------------------------------+
+ /// | FULLY MATCHING
|
+ /// | Partition 3
|
+ /// | +-----------------------------------+-----------------------------+
|
+ /// | | SPECIES (min: 'A...',max: 'A...') | S (min: 76, max: 101) |
|
Review Comment:
@adriangb Oh, I got you, I think the pic left out some details, I'll change
to min: 'Alpine', max: 'Alpine Sheep'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
