JingsongLi commented on code in PR #1145:
URL: https://github.com/apache/paimon-rust/pull/1145#discussion_r4306230027
##########
crates/paimon/src/spec/predicate.rs:
##########
@@ -284,6 +284,34 @@ pub enum PredicateOperator {
NotBetween,
}
+impl PredicateOperator {
+ /// The operator selecting the complement of the non-NULL rows, matching
+ /// Java `LeafFunction.negate`. `None` when there is no such operator.
+ fn negate(self) -> Option<Self> {
+ Some(match self {
+ Self::IsNull => Self::IsNotNull,
+ Self::IsNotNull => Self::IsNull,
+ Self::Eq => Self::NotEq,
Review Comment:
[P1] Refine floating bitmap candidates before ranking normalized NOT filters
Turning `NOT(f = 1)` into `NotEq` now enters the floating bitmap
scalar-index path. `BitmapGlobalIndexReader::query` deliberately returns all
non-null rows for floating `NotEq`, but `matching_ranges_with_exactness` labels
this leaf exact, so `scalar_matched_row_ids` skips refinement before vector
ranking. I reproduced this through public write/commit/index/search APIs: two
rows have `f=[1,2]`, vectors `[[1,0],[0,1]]`, a bitmap index on `f`, and an
ivf-flat index on the vectors. Searching `[1,0]` with limit 1 and
`Predicate::negate(equal(f,1))` returns row 0 (`f=1`), which violates the
filter; expected row 1. The same test passes on base `9568f39a`, and explicitly
constructing raw `Predicate::Not` still returns row 1 on this head. Propagate
floating bitmap candidate inexactness to the scanner and refine before ranking;
add this filtered Top-K regression test.
##########
crates/paimon/src/spec/predicate.rs:
##########
@@ -284,6 +284,34 @@ pub enum PredicateOperator {
NotBetween,
}
+impl PredicateOperator {
+ /// The operator selecting the complement of the non-NULL rows, matching
+ /// Java `LeafFunction.negate`. `None` when there is no such operator.
+ fn negate(self) -> Option<Self> {
+ Some(match self {
+ Self::IsNull => Self::IsNotNull,
+ Self::IsNotNull => Self::IsNull,
+ Self::Eq => Self::NotEq,
+ Self::NotEq => Self::Eq,
+ Self::Lt => Self::GtEq,
Review Comment:
[P2] Preserve NaN rows when negating floating partition comparisons
These operator flips assume complementary ordering, but native `datum_cmp`
uses `partial_cmp` for FLOAT/DOUBLE, so NaN makes both `<` and `>=` false. I
reproduced a scan with rows `(id,f)=[(0,0.0),(1,NaN),(2,2.0)]` and
`Predicate::negate(less_than(f,1.0))`: this head returns `[1,2]` when `f` is a
data column but only `[2]` when partitioned by `f`, because the normalized
`GtEq` prunes the NaN partition. The identical test returns `[1,2]` in both
layouts on base `9568f39a`. Java's floating comparison and the Arrow residual
path retain the NaN row here. Make native floating comparison compatible with
those semantics, or retain NULL-aware NOT until the complements are sound, and
cover partitioned/nonpartitioned NaN reads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
