JingsongLi opened a new pull request, #11290:
URL: https://github.com/apache/paimon/pull/11290

   ### Purpose
   
   A Data Evolution row can contain multiple embeddings, for example one per 
document chunk. This change lets callers search that list in Spark SQL with 
`vector_search` or in Java with `withMultiVector(float[][])`. A query matrix 
ranks each row once by summing the best cosine match for each query vector. A 
single query vector ranks the row by its best child match.
   
   The five native range-sharded indexes retrieve child vectors and map their 
hits back to table row IDs. A matrix query unions candidate rows and reranks 
their complete embedding lists from the query snapshot. Candidate retrieval is 
approximate. List columns support cosine similarity and each list column can 
have its own index and dimension.
   
   ### Tests
   
   Added Spark SQL end-to-end tests for `ivf-flat`, `ivf-pq`, `ivf-sq`, 
`ivf-rq`, and `diskann`. Each checks the ranked row IDs and exact MaxSim scores 
for a document that wins through two separate child matches. Additional tests 
cover typed dimensions, independent queries in a batch, distinct parent hits, 
deletes, filtering, incomplete writes and index-header validation.
   
   Ran 255 targeted tests with JDK 8, Spark 3 and Flink 1, including existing 
single-vector and centroid search regressions. All passed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to