JunRuiLee commented on code in PR #463:
URL: https://github.com/apache/paimon-rust/pull/463#discussion_r3534480501
##########
crates/paimon/src/table/table_scan.rs:
##########
@@ -866,6 +867,30 @@ impl<'a> TableScan<'a> {
can_push_down_limit_hint_for_scan(&self.data_predicates, row_ranges)
}
+ /// The predicate set that may prune WHOLE FILES by their stats.
+ ///
+ /// For primary-key tables read by merging (everything except DV tables,
+ /// which read raw with per-row deletion masks), only key conjuncts are
+ /// safe: a key's versions agree on the key columns but not on value
+ /// columns, so a value conjunct could prune the file holding the newest
+ /// version and resurrect an older one from a surviving file. The dropped
+ /// conjuncts are still enforced exactly by the post-merge residual filter
+ /// in `KeyValueFileReader`.
+ fn stats_pruning_predicates(&self) -> Vec<Predicate> {
+ let has_primary_keys = !self.table.schema().primary_keys().is_empty();
+ let deletion_vectors_enabled =
+
CoreOptions::new(self.table.schema().options()).deletion_vectors_enabled();
+ if has_primary_keys && !deletion_vectors_enabled {
Review Comment:
Good catch — fixed in 48c5565. The gate now exempts `merge-engine=first-row`
(matching the split-generation gate): first-row plans with `skip_level_zero`
and reads via `DataFileReader`, so there is no cross-file merge on the read
path and full-predicate stats pruning stays safe — pruning a file drops exactly
the rows the raw path's exact residual filter would drop anyway.
An unparseable merge engine stays conservative (key-only pruning). Added
`test_first_row_table_stats_pruning_keeps_non_key_conjuncts` locking the
exemption: a non-key conjunct on a first-row table must still stats-prune
compacted files (verified red without the exemption).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
