JingsongLi commented on code in PR #463:
URL: https://github.com/apache/paimon-rust/pull/463#discussion_r3534148445
##########
crates/paimon/src/table/table_scan.rs:
##########
@@ -866,6 +867,30 @@ impl<'a> TableScan<'a> {
can_push_down_limit_hint_for_scan(&self.data_predicates, row_ranges)
}
+ /// The predicate set that may prune WHOLE FILES by their stats.
+ ///
+ /// For primary-key tables read by merging (everything except DV tables,
+ /// which read raw with per-row deletion masks), only key conjuncts are
+ /// safe: a key's versions agree on the key columns but not on value
+ /// columns, so a value conjunct could prune the file holding the newest
+ /// version and resurrect an older one from a surviving file. The dropped
+ /// conjuncts are still enforced exactly by the post-merge residual filter
+ /// in `KeyValueFileReader`.
+ fn stats_pruning_predicates(&self) -> Vec<Predicate> {
+ let has_primary_keys = !self.table.schema().primary_keys().is_empty();
+ let deletion_vectors_enabled =
+
CoreOptions::new(self.table.schema().options()).deletion_vectors_enabled();
+ if has_primary_keys && !deletion_vectors_enabled {
Review Comment:
This also disables value-column stats pruning for `merge-engine=first-row`
PK tables because the condition only exempts DV tables. Unlike the merge-read
engines, FirstRow is planned with `skip_level_zero` and read via
`DataFileReader` rather than `KeyValueFileReader` (see the
`read_merges_overlapping_keys` gate below and `TableRead::to_arrow`), so there
is no post-merge resurrection risk there. Keeping only PK conjuncts can make
queries such as `WHERE non_key_col = ...` scan every compacted file on FirstRow
tables. Could this gate also exempt `MergeEngine::FirstRow`, matching the split
generation path, and keep the full predicates for that engine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
