raghavyadav01 opened a new pull request, #20073: URL: https://github.com/apache/pinot/pull/20073
Adds `VECTOR_DISTANCE(column, ARRAY[...], k)`, which returns the cosine distance of each row from the query vector, served by the column's vector index. A typical vector query filters with `VECTOR_SIMILARITY(column, vec, k)` and then projects `cosineDistance(column, vec)` to rank. The projection reads every matched vector from the forward index, even though the HNSW search has just computed those distances. `VECTOR_DISTANCE` with the same vector and `k` reuses them: ```sql SELECT id, VECTOR_DISTANCE(embedding, ARRAY[...], 10) AS d FROM t WHERE VECTOR_SIMILARITY(embedding, ARRAY[...], 10) ORDER BY d LIMIT 10 ``` - `VectorIndexReader#getDocDistances(vector, topK)`: a new default method returning `null`, so existing readers are unaffected. - `HnswVectorIndexReader` implements it for COSINE indexes, converting Lucene's score `(1 + cos) / 2` to `1 - cos`. It keeps the distances of its most recent searches, so the projection reuses the filter's search instead of searching again. - Rows the index did not return, or indexes that serve no distances, fall back to `cosineDistance` on the forward index. The values match `cosineDistance` up to the index's single-precision arithmetic. In a vector-search benchmark this took vector query p50 from 68 ms to 13 ms. **Tests**: - `IndexVectorDistanceTransformFunctionTest` covers the no-index fallback (equals `cosineDistance`), index-served distances, the full fallback when the index misses a row, and argument validation. - `HnswVectorIndexCreatorTest#testServesCosineDistances` checks the served values against `cosineDistance`, reuse after a filtered search, and that EUCLIDEAN indexes serve none. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
