shyjsarah opened a new pull request, #8879: URL: https://github.com/apache/paimon/pull/8879
## Purpose Distribute Spark lateral `vector_search` over compatible vector-index split groups so multi-partition searches can use executor parallelism instead of searching every index split serially in one task. ## Changes - group vector-index splits by estimated search cost - execute local Top-K search independently for each split group - merge and deduplicate partial results into a deterministic global Top-K per query - preserve the outer row when a split group returns no result - pin the snapshot before materializing distributed results - fall back to the existing local path for unsupported plans, including deletion vectors, raw-data fallback, and refine/rerank - add an opt-in Spark connector option; the feature remains disabled by default: ```text spark.paimon.vector-search.lateral-join.distributed.enabled=true ``` The maximum number of split groups can be controlled with: ```text spark.paimon.vector-search.lateral-join.distributed.max-split-groups=16 ``` This does not change IVF-PQ `nprobe` and does not introduce a cross-partition index. ## Tests - `VectorSearchBuilderTest` - `LateralVectorSearchExecutionTest` - `TableValuedFunctionsTest` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
