xudong963 commented on code in PR #18868:
URL: https://github.com/apache/datafusion/pull/18868#discussion_r2670743397
##########
datafusion/datasource-parquet/src/row_group_filter.rs:
##########
@@ -70,6 +79,109 @@ impl RowGroupAccessPlanFilter {
self.access_plan
}
+ /// Returns the is_fully_matched vector
+ pub fn is_fully_matched(&self) -> &Vec<bool> {
+ &self.is_fully_matched
+ }
+
+ /// Prunes the access plan based on the limit and fully contained row
groups.
+ ///
+ /// The pruning works by leveraging the concept of fully matched row
groups. Consider a query like:
+ /// `WHERE species LIKE 'Alpine%' AND s >= 50 LIMIT N`
+ ///
+ /// After initial filtering, row groups can be classified into three
states:
+ ///
+ /// 1. Not Matching / Pruned
+ /// 2. Partially Matching (Row Group/Page contains some matches)
+ /// 3. Fully Matching (Entire range is within predicate)
+ ///
+ ///
+-----------------------------------------------------------------------+
+ /// | NOT MATCHING
|
+ /// | Partition 1
|
Review Comment:
Good catch, I'd like to use Row group here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
