shyjsarah opened a new pull request, #68: URL: https://github.com/apache/paimon-vector-index/pull/68
## Purpose Filtered automatic IVF search currently expands the initial `nprobe` by inverse selectivity without a configurable bound. For selective filters, that can make the first search round much wider than desired even though the progressive search path can retry with a larger width when needed. ## Changes - Add an optional `max_initial_filter_expansion_factor` for automatic IVF search in Rust and Java. - Keep the option unset by default, preserving the current unlimited initial expansion behavior. - Interpret factor `1` as no filter-driven initial expansion; for example, with base `nprobe = 16`, factor `4` caps the initial width at `64`. - Apply the cap only to the initial width. Progressive retries may expand beyond it if filtered results do not fill `top_k`. - Reject zero factors, explicit IVF widths combined with the option, and use with DiskANN. - Preserve the existing C FFI ABI; C-created search parameters retain the current behavior. - Document the Rust and Java APIs. The capped automatic calculation is: ```text base = unfiltered automatic nprobe scaled = ceil(base * vector_count / min(matching_count, vector_count)) cap = min(nlist, base * factor) initial = min(scaled, cap) ``` ## Compatibility - No default behavior change. - No storage-format change. - No C ABI change. - Explicit `nprobe` behavior is unchanged. ## Tests - `cargo test --workspace` - `cargo clippy --workspace --all-targets -- -D warnings` - `cargo fmt --all -- --check` - `mvn -f java/pom.xml test -q` - Java/JNI native validation, panic-boundary, and safety tests - `python3 tools/check_license_headers.py` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
