JunRuiLee opened a new pull request, #609:
URL: https://github.com/apache/paimon-rust/pull/609
### Purpose
Primary-key vector search reused the whole `with_filter` predicate as the
per-row residual applied after ANN recall — even though the same predicate is
already pushed into scan planning (`PkVectorScan`), where partition-only
conjuncts prune partitions/files. Re-checking partition conjuncts per row is
redundant, and re-reading partition columns for the residual is wasted I/O.
### Change
Derive the residual from the DATA (non-partition) conjuncts only, via
`split_partition_and_data_predicates`:
- A partition-only filter (e.g. `dt = 'a'`) now needs no residual at all —
the
residual read is skipped entirely (planning already enforces the
partition).
- `AND(partition, data)` residual-checks only the data conjunct (one fewer
column re-read per candidate file).
- A mixed conjunct (e.g. `dt = 'a' OR id > 5`) is not partition-only, so it
stays whole in the residual and evaluates against the materialized
partition
column (primary-key data files carry partition columns) — results
unchanged.
Planning still receives the whole filter, so partition pruning is unaffected.
Mirrors Java `BatchVectorSearchBuilderImpl.withFilter`. Covers the
whole-snapshot
`execute_read` path (single-query and batch converge on the shared core).
Output is unchanged; this reduces residual I/O and per-row work for
partition-referencing filters and keeps the residual's inputs to data-only
predicates (so it never depends on partition columns being materialized).
### Tests
`residual_uses_only_data_conjuncts_of_the_filter`: partition-only → no
residual;
data-only → kept; `AND(partition, data)` → only the data conjunct; mixed `OR`
→ stays whole.
`cargo test -p paimon`, `cargo fmt --all --check`, and
`cargo clippy -p paimon --all-targets -- -D warnings` pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
