shyjsarah opened a new pull request, #65: URL: https://github.com/apache/paimon-vector-index/pull/65
## Summary Reuse batch-local IVF-PQ list tables when many queries probe the same lists. The optimization applies to both filtered and unfiltered batch search and is enabled only when the estimated reuse amortizes table construction. This PR is stacked on #64. Until #64 merges, GitHub also shows the filter-scanning commit; the PR-specific commit is `ddbf60a`. ## Changes - Build the list-dependent part of residual L2 PQ tables once per active loaded list. - Build the query-dependent inner-product table once per active query in each loaded-list batch. - Combine the list and query components before scanning instead of rebuilding a full residual table for every query/list pair. - Apply the same optimization to `filter=None`; treat no filter as all loaded rows being active. - Skip ephemeral precomputation for small batches, insufficient probe reuse, empty filtered lists, unsupported metrics/layouts, or readers with persistent precomputed tables. - Keep ephemeral tables batch-local so they do not remain resident on the reader. ## Testing - `cargo fmt --all -- --check` - `cargo clippy -p paimon-vindex-core --all-targets -- -D warnings` - `cargo test -p paimon-vindex-core --locked` - Verified filtered and unfiltered batch results against individual reader searches with floating-point tolerance. - Covered activation for large filtered and unfiltered batches. - Covered fallback for small batches, single-probe work, and empty filters. ## Notes - Depends on #64. - Public API: unchanged. - The precomputed and direct residual formulas are algebraically equivalent but can differ slightly due to floating-point accumulation order. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
