JeonDaehong commented on code in PR #1145:
URL: https://github.com/apache/paimon-rust/pull/1145#discussion_r4481221109
##########
crates/paimon/src/table/global_index_scanner/evaluator.rs:
##########
@@ -349,7 +394,14 @@ impl GlobalIndexScanner {
match result {
Some(child_result) => {
fully_evaluated &=
child_result.fully_evaluated;
+ if child_result.fully_evaluated {
+ exact_ranges.extend(exclude_row_ranges(
Review Comment:
Thanks. `exclude_row_ranges` now skips cuts that end before the current
segment and stops at the first one past it, so it's linear; the OR arm also
skips the subtraction when a child has no candidates. For K disjoint singletons
minus themselves it goes from 437 ms to 1.3 ms at K=16,384 (debug build), and a
random comparison against the old version gives the same output.
##########
crates/paimon/src/arrow/residual.rs:
##########
@@ -258,6 +250,63 @@ pub(crate) fn evaluate_predicate_mask(
/// Two [`DataField`]s refer to the same logical column when their IDs match,
or
/// (for schemas without stable IDs) their names match.
+/// Evaluates `NOT predicate`. The `NOT` is pushed down to the leaves with
+/// De Morgan first: a leaf masks NULL as false, so negating a combined mask
+/// would turn UNKNOWN into true.
+fn evaluate_negated_mask(
+ batch: &RecordBatch,
+ predicate: &Predicate,
+ file_fields: &[DataField],
+ scan_fields: &[DataField],
+) -> crate::Result<Option<BooleanArray>> {
+ match predicate {
+ // NOT(a AND b) = NOT a OR NOT b, evaluated like the `Or` arm.
+ Predicate::And(children) => {
+ let mut combined = BooleanArray::from(vec![false;
batch.num_rows()]);
+ for child in children {
+ let Some(mask) = evaluate_negated_mask(batch, child,
file_fields, scan_fields)?
+ else {
+ return Ok(None);
+ };
+ combined = combine_filter_masks(&combined, &mask, true);
+ }
+ Ok(Some(combined))
+ }
+ // NOT(a OR b) = NOT a AND NOT b, evaluated like the `And` arm.
+ Predicate::Or(children) => {
+ let mut combined = None;
+ for child in children {
+ let Some(mask) = evaluate_negated_mask(batch, child,
file_fields, scan_fields)?
+ else {
+ continue;
+ };
+ combined = Some(match combined {
+ Some(existing) => combine_filter_masks(&existing, &mask,
false),
+ None => mask,
+ });
+ }
+ Ok(combined)
+ }
+ Predicate::Not(inner) => evaluate_predicate_mask(batch, inner,
file_fields, scan_fields),
+ _ => {
+ let Some(mask) = evaluate_predicate_mask(batch, predicate,
file_fields, scan_fields)?
+ else {
+ return Ok(None);
+ };
+ let negated =
+ boolean_mask_from_predicate(mask.len(), |row_index|
!mask.value(row_index));
+ if let Some(guard) = predicate.not_null_guard() {
Review Comment:
Thanks, reproduced it with the sparse fixture. `remap_predicate` now pushes
the NOT down before folding missing fields, so a missing leaf under NOT becomes
`AlwaysFalse` (IS NULL / IS NOT NULL still flip). Added
`test_read_predicate_missing_column_under_not_stays_null`, which returns all
six ids before the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]