JingsongLi opened a new pull request, #8581: URL: https://github.com/apache/paimon/pull/8581
## Summary Add Spark SQL single-vector search support for primary-key tables backed by the bucket-local vector indexes introduced in #8579. Primary-key searches keep their snapshot-scoped Core execution path, while data-evolution tables continue to use Spark-dispatched vector index evaluation. ## Changes - Route configured primary-key vector fields from `SparkVectorSearchBuilderImpl` to the Core `PrimaryKeyVectorRead`, even when `spark.paimon.vector-search.distribute.enabled` is enabled. - Introduce `ScoreRecordReader` so Spark metadata handling depends on the score-reader contract instead of the concrete `IndexedSplitRecordReader` implementation. - Let primary-key physical-position readers expose search scores through the same reader contract used by row-id-based vector results. - Skip Spark's metadata-only `PaimonLocalScan` optimization for primary-key vector fields, because their results are represented by snapshot-scoped physical splits rather than global row IDs. - Add Spark SQL coverage for score-only projection, global Top-K across buckets, partition pruning, DEDUPLICATE updates and deletes, and PARTIAL_UPDATE lookup completion. - Preserve the existing metadata-only fast path for data-evolution vector indexes. ## Behavior - `vector_search` on a configured primary-key vector field returns physical table rows and `__paimon_search_score` from the same captured snapshot. - Partition predicates are applied before bucket-level Top-K; non-partition predicates remain unsupported for primary-key vector search. - DEDUPLICATE and PARTIAL_UPDATE changes become searchable through the vector index state published with completed Level-1 data, while deletion vectors filter stale ANN candidates. - Hybrid search, lateral/batch vector search, and Spark-distributed bucket ANN execution are outside this PR. ## Testing - [x] `PrimaryKeyVectorSearchTest` — 5 Spark SQL scenarios covering routing, score metadata, multi-bucket Top-K, partition pruning, DEDUPLICATE update/delete, and PARTIAL_UPDATE. - [x] `VectorSearchOptionsTest` — verifies the existing data-evolution query-option and metadata-only path. - [x] `PrimaryKeyVectorPositionReaderTest` and `IndexedSplitRecordReaderTest` — 12 Core score-reader tests. - [x] Spotless checks for Spark Common and Spark UT. ## Notes Builds on #8579. There are no configuration or storage-format changes in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
