JingsongLi commented on code in PR #82:
URL:
https://github.com/apache/paimon-vector-index/pull/82#discussion_r3854334767
##########
core/src/ivfpq.rs:
##########
@@ -2834,6 +2937,18 @@ fn seed_heaps(heaps: &mut [TopKHeap], seed_ids: &[i64],
seed_distances: &[f32],
// --- Utilities ---
+/// Evenly strided rows of the training data for projection calibration.
+fn calibration_sample(data: &[f32], n: usize, d: usize) -> Vec<f32> {
+ let rows = crate::projected_assign::CALIBRATION_ROWS.min(n);
+ if rows == 0 || d == 0 {
+ return Vec::new();
+ }
+ let stride = n / rows;
Review Comment:
[P2] Sample across the full calibration input
`stride = n / rows` floors the ratio before selecting rows. For `n=4095` and
`rows=2048`, it becomes 1, so this takes exactly rows 0..2047 and ignores the
entire latter half; at `n=4096` it abruptly changes to every other row. Because
this sample drives Auto’s projected-width and cost gate, ordered training data
can make the decision from an unrepresentative prefix and enable a path that is
more expensive for the remaining rows (or reject a useful one). Please compute
each source index fractionally (for example `i * n / rows`, with checked
arithmetic, or endpoint spacing) and test the 2049/4095 boundaries with
order-separated calibration data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
