XiaoHongbo-Hope commented on code in PR #672:
URL: https://github.com/apache/paimon-rust/pull/672#discussion_r3713145936
##########
crates/paimon/src/table/read_builder.rs:
##########
@@ -56,6 +60,15 @@ fn is_exact_filter_pushdown_for_schema(
let (_, data_predicates) =
split_partition_and_data_predicates(filter.clone(), fields,
partition_keys);
data_predicates.is_empty()
+ || data_predicates.iter().all(|predicate| {
+ matches!(
+ predicate,
+ Predicate::Leaf {
+ op: PredicateOperator::Eq | PredicateOperator::Like,
Review Comment:
Fixed in 0f1b057. Filter execution exactness is now separate from statistics
exactness: PaimonTableScan records whether the normalized filter contains data
predicates and keeps num_rows Inexact whenever it does. Added equality and
exact-LIKE COUNT(*) regressions; partition-only COUNT(*) remains exact.
##########
crates/paimon/src/table/global_index_scanner.rs:
##########
@@ -722,35 +823,104 @@ impl GlobalIndexScanner {
Some(probe) => Some(probe.enter().await),
None => None,
};
+ let remaining = limit.saturating_sub(all_row_ids.len() as
usize);
let result = self
- .query_entry(entry, data_type, between, &plan,
effective_predicates)
+ .query_entry(
+ entry,
+ data_type,
+ between,
+ &plan,
+ effective_predicates,
+ Some(remaining),
+ )
.await?;
- Ok((entry.row_range_start, result))
- });
- let all_row_ids = try_fold_bounded(
- futures,
- self.global_index_thread_num,
- RoaringTreemap::new(),
- |all_row_ids, (row_range_start, file_result)| {
- if let Some(bitmap) = file_result {
+ if let Some(bitmap) = result {
for row_id in bitmap.iter() {
- all_row_ids.insert(row_id + row_range_start as u64);
+ all_row_ids.insert(row_id + entry.row_range_start as
u64);
}
}
- },
- )
- .await?;
+ if all_row_ids.len() >= limit as u64 {
Review Comment:
Fixed in 0f1b057. The planning LIMIT is now enabled only for one exact
Eq/Like data predicate; range and compound predicates receive no index
early-stop limit. Added a guard regression covering the floating-range case.
##########
crates/paimon/src/table/global_index_scanner.rs:
##########
@@ -722,35 +823,104 @@ impl GlobalIndexScanner {
Some(probe) => Some(probe.enter().await),
None => None,
};
+ let remaining = limit.saturating_sub(all_row_ids.len() as
usize);
let result = self
- .query_entry(entry, data_type, between, &plan,
effective_predicates)
+ .query_entry(
+ entry,
+ data_type,
+ between,
+ &plan,
+ effective_predicates,
+ Some(remaining),
Review Comment:
Fixed in 0f1b057. Selected shard ranges are checked for overlap. When they
overlap, reader-local limiting is disabled and aggregation continues through
each shard until the global de-duplicated row-ID set reaches LIMIT. Added a
partial-overlap mixed BTree/bitmap regression.
##########
crates/paimon/src/table/global_index_scanner.rs:
##########
@@ -929,8 +1100,9 @@ impl GlobalIndexScanner {
plan.allow_btree = plan.selected_btree > 0
&& btree_valid
- && self.btree_fallback_scan_max_size > 0
- && btree_total <= self.btree_fallback_scan_max_size;
+ && (allow_large_btree
Review Comment:
Fixed in 0f1b057. Only the cheap preferred point/prefix probe may run above
the BTree fallback byte budget. If it does not fill LIMIT, the configured bound
is re-applied; an over-budget complex LIKE returns unsupported and falls back
to the normal data scan. Added regressions for both the bounded fallback and
successful preferred-probe paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
