jordepic opened a new pull request, #619: URL: https://github.com/apache/paimon-rust/pull/619
### Purpose Linked issue: close #617 The shared residual evaluator (used by both the parquet row filter and the exact batch backstop) evaluates `In`/`NotIn` as one comparison kernel per literal, OR-combining the masks — O(rows × literals). An engine pushing a batch of point-lookup keys as an `In` predicate makes every table probe quadratic. ### Brief change log - `set_membership_hash_mask`: build a hash set of the literal values once, answer membership in a single pass over the column, for Binary/LargeBinary/BinaryView, Utf8/LargeUtf8/Utf8View, and Int8–Int64 columns. - Any other column/literal shape — and any literal that would not convert for the column — returns `None` and falls back to the existing per-literal loop, preserving its exact semantics including its error behavior for unconvertible literals. - Null rows produce `false` for both `In` and `NotIn`, matching the loop's `sanitize_filter_mask` convention. ### Tests - New: `test_in_hash_path_filters_exactly_with_nulls`, `test_not_in_hash_path_excludes_nulls`, `test_in_hash_path_on_binary_column` (In and NotIn masks incl. nulls), `test_in_unconvertible_literal_still_errors` (error behavior preserved through the fallback). - `cargo test -p paimon --lib arrow::residual` (26), `read_builder` (33), and `arrow::format::parquet` (41) all pass. ### API and Format None — internal evaluation only; results are unchanged. ### Documentation Doc comment on the fast path explains the shape and the fallback contract. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
