jerry-024 commented on PR #68:
URL:
https://github.com/apache/paimon-vector-index/pull/68#issuecomment-5173771582
Overall, this approach is aligned with common industry practice for filtered
IVF search: start with a bounded probe budget and expand progressively when the
filtered result set is too short. Keeping the option unset by default also
preserves existing behavior.
I think two documentation changes are still needed before merge:
1. Please state the recall trade-off explicitly. Progressive expansion is
triggered only when fewer than `top_k` valid results are returned. If a capped
initial `nprobe` already fills `top_k`, the search stops even though probing
more lists could improve Recall@K. Suggested wording:
> Lower factors reduce initial search work but may reduce recall compared
with uncapped automatic search. Progressive expansion occurs only when fewer
than `top_k` valid results are returned.
2. The primary Rust and Java examples currently use factor `4`, which may
look like a generally recommended value. Please keep the main examples as
`automatic(10)` and move the capped form to an optional tuning example. If
factor `4` remains in the primary examples, please support that recommendation
with a small benchmark comparing uncapped and capped search latency and
Recall@K on representative filtered workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
