JingsongLi opened a new pull request, #8352: URL: https://github.com/apache/paimon/pull/8352
## Summary Add a LanceDB-style refine factor for vector search so approximate IVF candidates can be reranked with the original vectors stored in the Paimon table. This is especially useful for compressed vector indexes such as IVF-PQ, where index scores may differ from exact raw-vector scores. ## Changes - Thread a separate index candidate limit through single-vector, batch-vector, and Spark vector read paths. - Add refine/rerank option parsing with query options taking precedence over table options, including index-specific and IVF-prefixed forms such as `ivf.refine_factor`. - Rerank indexed candidates by reading raw vectors from the table and recomputing exact scores with the configured vector metric. - Extend the test vector index with a reverse-score mode to simulate approximate misranking, and add single/batch rerank coverage. - Document the new `ivf.refine_factor` search option. ## Testing - `mvn -pl paimon-core -am -Dtest=VectorSearchBuilderTest -DfailIfNoTests=false -DskipITs -Dcheckstyle.skip -Drat.skip=true -Dspotless.check.skip=true test` - `mvn -pl paimon-spark/paimon-spark-common -am -Dtest=SparkVectorReadImplTest -DfailIfNoTests=false -DskipITs -Dcheckstyle.skip -Drat.skip=true -Dspotless.check.skip=true test` - `mvn -pl paimon-core,paimon-spark/paimon-spark-common -am -DskipITs -DfailIfNoTests=false -Drat.skip=true spotless:check` - `git diff --check` ## Notes No table format or index file format change is required; rerank uses row ids returned by the vector index to read original vectors from the Paimon table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
