JingsongLi commented on code in PR #672:
URL: https://github.com/apache/paimon-rust/pull/672#discussion_r3713340028
##########
crates/paimon/src/table/read_builder.rs:
##########
@@ -56,6 +60,15 @@ fn is_exact_filter_pushdown_for_schema(
let (_, data_predicates) =
split_partition_and_data_predicates(filter.clone(), fields,
partition_keys);
data_predicates.is_empty()
+ || data_predicates.iter().all(|predicate| {
+ matches!(
+ predicate,
+ Predicate::Leaf {
+ op: PredicateOperator::Eq | PredicateOperator::Like,
Review Comment:
[P1] Enable exact Eq/LIKE pushdown for unpartitioned tables
This new exact classification is unreachable when `partition_keys` is empty
because the function returns `false` above. DataFusion therefore keeps the
`Inexact` filter above `TableScan`; `PushDownLimit` cannot commute the limit
through that filter, so `PaimonTableProvider::scan` receives no limit and
`safe_global_index_limit` is never reached. This leaves the documented
unpartitioned row-tracking/data-evolution global-index tables exposed to the
same unbounded index planning and OOM risk that this PR is intended to fix.
Please classify these exact data leaves even when there are no partition keys,
and add an unpartitioned physical-plan regression asserting that
`PaimonTableScan` receives the limit.
##########
crates/paimon/src/table/table_scan.rs:
##########
@@ -928,6 +928,38 @@ struct PaimonTableScan<'a> {
projected_read_field_ids: Option<HashSet<i32>>,
}
+fn safe_global_index_limit(
+ limit: Option<usize>,
+ data_predicates: &[Predicate],
+ has_primary_keys: bool,
+ deletion_vectors_enabled: bool,
+ has_partition_filter: bool,
+ has_bucket_predicate: bool,
+) -> Option<usize> {
+ // Early-stop is valid only when every index row id is an exact match.
+ // In particular, bitmap floating-point range queries deliberately return
+ // all non-null rows as residual candidates, so they must see every shard.
+ // Multiple leaves are also unsafe because limiting either side before an
+ // intersection can under-fill the final result.
+ let exact_single_predicate = matches!(
+ data_predicates,
+ [Predicate::Leaf {
+ op: PredicateOperator::Eq | PredicateOperator::Like,
Review Comment:
[P2] Exclude bitmap NaN equality from index early-stop
This treats every `Eq` predicate as an exact index match, but bitmap
float/double keys canonicalize all NaN payloads into one dictionary key. A
limited lookup can therefore return an earlier NaN with a different payload and
truncate a later row whose payload matches the literal. The exact Arrow
residual uses IEEE `totalOrder` equality, drops the earlier candidate, and
cannot recover the truncated match, so `WHERE f = <NaN> LIMIT 1` can return
zero rows despite a match. Please disable early-stop for Float/Double equality
when bitmap indexes may participate, or align the index and residual NaN
equality semantics, and add a multi-payload NaN + LIMIT regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
