SteNicholas commented on code in PR #1486:
URL: https://github.com/apache/paimon-rust/pull/1486#discussion_r4657825092
##########
crates/paimon/src/table/pk_search_ranker.rs:
##########
@@ -172,10 +184,39 @@ pub(crate) fn weighted_score(
max = max.max(position.score());
}
for position in ranking.positions() {
- let normalized = f64::from(normalize_route_score(position.score(),
min, max));
+ let normalized = if ranking.cosine_distance_scores && max > min {
+ (f64::from(position.score()) - f64::from(min)) /
(f64::from(max) - f64::from(min))
Review Comment:
**Adding a full-text route that matches only row 2 makes a near match
outrank the exact match.** For query `[1, 0]`, I used row 0 `[1, 2^-30]`, row 1
`[1, 0]`, and row 2 `[0, 1]`, all padded to four dimensions with zeros. With
cosine route weight `1`, candidate limit `3`, and refine factor `2`, the
reranked distances are `2^-61`, `0`, and `1`. `execute_read()` returned row 1.
Adding a full-text route with weight `0.5` that matched only row 2 changed the
winner to row 0.
Min–max normalization computes `1 - 2^-61` for row 0, which rounds to `1`
even in `f64`. Please preserve that difference when comparing fused scores. In
this case the other route contributes equally to rows 0 and 1, so their
original cosine distances can break the apparent tie. A regression should
expect row 1 with and without the full-text route.
##########
crates/paimon/src/table/pk_search_ranker.rs:
##########
@@ -172,10 +184,39 @@ pub(crate) fn weighted_score(
max = max.max(position.score());
}
for position in ranking.positions() {
- let normalized = f64::from(normalize_route_score(position.score(),
min, max));
+ let normalized = if ranking.cosine_distance_scores && max > min {
+ (f64::from(position.score()) - f64::from(min)) /
(f64::from(max) - f64::from(min))
+ } else {
+ f64::from(normalize_route_score(position.score(), min, max))
+ };
*fused_scores.entry(position.clone()).or_insert(0.0) +=
ranking.weight() * normalized;
}
}
+ if rankings
+ .iter()
+ .any(|ranking| ranking.cosine_distance_scores)
+ {
+ // Min-max normalization is monotone within one route. Use its input
+ // order in that case and the f64 fused totals for multiple routes.
+ let mut ranked = fused_scores
+ .into_iter()
+ .map(|(position, score)| {
+ let order = if rankings.len() == 1 {
+ f64::from(position.score())
+ } else {
+ score
+ };
+ Ok((position.with_score(score as f32)?, order))
+ })
+ .collect::<crate::Result<Vec<_>>>()?;
+ ranked.sort_by(|(left, left_order), (right, right_order)| {
+ right_order
+ .total_cmp(left_order)
Review Comment:
**Reordering three equally weighted cosine routes changes which row wins
`weighted_score`.** I used row 0 `A = [1, 0, 0, 0]`, row 1 `B = [0, 1, 0, 0]`,
and row 2 `[-1, -1, 15, 0]`. Each route retrieved all three rows with refine
factor `2`. With `M = [1, 1, 0, 0]`, `execute_read()` returned row 0 for `[A,
M, B]` and row 1 for `[B, M, A]`.
Rows 0 and 1 each receive the contributions `1`, `1`, and
`0.06224118620351357`. Summing them in different orders leaves the `f64` totals
one ULP apart, so the physical-position tie-break never runs. Please make the
sum reproducible for the same set of contributions and add a regression that
expects row 0 for both permutations.
##########
crates/paimon/src/table/de_vector_read.rs:
##########
@@ -3229,21 +3556,11 @@ fn compute_raw_vector_score_from_values(
debug_assert_eq!(query.len(), end - start);
match metric {
RawVectorMetric::L2 => compute_raw_vector_l2_score(query,
&values.values()[start..end]),
- RawVectorMetric::Cosine => {
- let mut dot = 0.0;
- let mut norm_b = 0.0;
- for (q, value_index) in query.iter().zip(start..end) {
- let stored = values.value(value_index);
- dot += q * stored;
- norm_b += stored * stored;
- }
- let denominator = query_l2_squared_norm.sqrt() * norm_b.sqrt();
- if denominator == 0.0 {
- 0.0
- } else {
- dot / denominator
- }
- }
+ RawVectorMetric::Cosine => cosine_similarity(
Review Comment:
**Wrapping each vector in a one-element list changes exact Top-1 from row 11
to row 10.** Let `t = 2^-27`, with query `[1, t, 1]`, row 10 `[1, 0, -1]`, and
row 11 `[1, t, -1]`. For flat vectors, both the single-query scorer and a
four-query batch selected row 11 with score `2^-55`. With one embedding per
list, `ListScorer` returned row 10 with score `0` in both `Independent` and
`MaxSim` modes. Its call to `multi_vector::cosine` loses the middle term when
evaluating `1 + 2^-54 - 1`.
Please share the dot-product calculation that preserves this residual with
list reranking, while keeping MaxSim’s sum across queries in `f64`. The same
fixture should select row 11 for both column shapes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]