jerry-024 commented on code in PR #85:
URL:
https://github.com/apache/paimon-vector-index/pull/85#discussion_r3912120575
##########
core/src/ivfpq.rs:
##########
@@ -114,9 +116,40 @@ impl IVFPQIndex {
codes: vec![Vec::new(); nlist],
precomputed_table: Vec::new(),
fastscan_codes: Vec::new(),
+ coarse_assignment: CoarseAssignment::default(),
}
}
+ pub fn quantizer_centroids(&self) -> &[f32] {
+ &self.quantizer_centroids
+ }
+
+ /// Enables automatic Vamana coarse assignment for large centroid matrices.
+ /// Disable it to keep vector assignment exact.
+ pub fn set_approximate_coarse_assignment(&mut self, enabled: bool) {
+ assert!(
Review Comment:
Fixed in `0dd8915`. The coarse-assignment setters on all four direct IVF
index types are now crate-private, so callers cannot switch policy after
training (or on an index returned by `from_trained`). Public callers select the
policy through `VectorIndexConfig` or `ivf.coarse-assignment=auto|exact` before
the writer/training lifecycle starts. `from_trained` only copies the source
index policy internally. The API and release documentation now state this
lifecycle.
##########
core/src/index.rs:
##########
@@ -600,6 +625,20 @@ fn parse_nlist_options(
}
}
+fn parse_ivf_coarse_assignment_option(options: &mut ConfigOptions) ->
io::Result<bool> {
+ match options
+ .optional("ivf.coarse-assignment")
+ .as_deref()
+ .map(str::trim)
+ {
+ None | Some("auto") => Ok(true),
Review Comment:
I investigated the corresponding Lance behavior and am intentionally
retaining `auto` as the default because it aligns with Lance’s existing IVF
build contract introduced by
[lance-format/lance#4089](https://github.com/lance-format/lance/pull/4089):
- Lance defaults its indexing speedup status to `Auto` and enables it when
`nlist × dimension >= 1,000,000`
([source](https://github.com/lance-format/lance/blob/742e6a317c95840ca18adf47646ea79e0070c57b/rust/lance-index/src/vector/utils.rs#L24-L103)).
- Build-time partition assignment uses HNSW top-1
([source](https://github.com/lance-format/lance/blob/742e6a317c95840ca18adf47646ea79e0070c57b/rust/lance-index/src/vector/ivf/transform.rs#L46-L116)),
while query-time `find_partitions` still uses exact centroid top-k
([source](https://github.com/lance-format/lance/blob/742e6a317c95840ca18adf47646ea79e0070c57b/rust/lance-index/src/vector/ivf.rs#L343-L353)).
- Lance provides an exact opt-out through
`LANCE_USE_HNSW_SPEEDUP_INDEXING=disabled`.
Paimon follows the same policy boundary and threshold, using Vamana (`R=12`,
`L=15`) instead of HNSW (`M=12`, `ef=15`), and provides the finer-grained
`ivf.coarse-assignment=exact` opt-out. The low-`nprobe` effect is therefore an
intentional, documented build-speed/recall tradeoff of `auto`; callers
requiring exact nearest-centroid placement can select `exact`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
