JingsongLi opened a new pull request, #11290: URL: https://github.com/apache/paimon/pull/11290
### Purpose A Data Evolution row can contain multiple embeddings, for example one per document chunk. This change lets callers search that list in Spark SQL with `vector_search` or in Java with `withMultiVector(float[][])`. A query matrix ranks each row once by summing the best cosine match for each query vector. A single query vector ranks the row by its best child match. The five native range-sharded indexes retrieve child vectors and map their hits back to table row IDs. A matrix query unions candidate rows and reranks their complete embedding lists from the query snapshot. Candidate retrieval is approximate. List columns support cosine similarity and each list column can have its own index and dimension. ### Tests Added Spark SQL end-to-end tests for `ivf-flat`, `ivf-pq`, `ivf-sq`, `ivf-rq`, and `diskann`. Each checks the ranked row IDs and exact MaxSim scores for a document that wins through two separate child matches. Additional tests cover typed dimensions, independent queries in a batch, distinct parent hits, deletes, filtering, incomplete writes and index-header validation. Ran 255 targeted tests with JDK 8, Spark 3 and Flink 1, including existing single-vector and centroid search regressions. All passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
