shyjsarah commented on code in PR #85:
URL:
https://github.com/apache/paimon-vector-index/pull/85#discussion_r3911762636
##########
core/src/index.rs:
##########
@@ -600,6 +625,20 @@ fn parse_nlist_options(
}
}
+fn parse_ivf_coarse_assignment_option(options: &mut ConfigOptions) ->
io::Result<bool> {
+ match options
+ .optional("ivf.coarse-assignment")
+ .as_deref()
+ .map(str::trim)
+ {
+ None | Some("auto") => Ok(true),
Review Comment:
**[major] This leaves the original blocker on the default path.** The new
`exact` option works, but omitted/`auto` still assigns through Vamana while
search selects lists using exact centroid top-k. Re-running the original
release probe on this commit still moved 1,807/4,096 centroid vectors and
returned `-1 / f32::MAX` for a self-query with `nprobe=1`; explicit exact mode
moved 0 and succeeded. Documenting the loss makes it visible, but does not make
add and lookup use a consistent default contract. Please default to exact or
unify the query-side policy.
##########
core/src/ivfpq.rs:
##########
@@ -114,9 +116,40 @@ impl IVFPQIndex {
codes: vec![Vec::new(); nlist],
precomputed_table: Vec::new(),
fastscan_codes: Vec::new(),
+ coarse_assignment: CoarseAssignment::default(),
}
}
+ pub fn quantizer_centroids(&self) -> &[f32] {
+ &self.quantizer_centroids
+ }
+
+ /// Enables automatic Vamana coarse assignment for large centroid matrices.
+ /// Disable it to keep vector assignment exact.
+ pub fn set_approximate_coarse_assignment(&mut self, enabled: bool) {
+ assert!(
Review Comment:
**[minor] This guard permits changing the policy after `train()` (or on an
index created by `from_trained`) as long as no rows were added.** For residual
IVF-PQ, training uses the current coarse assignments to build the PQ training
residuals, so switching here makes add encode a different assignment
distribution than the one used for training. IVF-SQ has the same issue for its
residual/list quantizers. Please restrict policy changes to pre-training state,
or invalidate/retrain assignment-dependent state; also document the lifecycle
requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
