TheR1sing3un opened a new pull request, #9935: URL: https://github.com/apache/paimon/pull/9935
### Purpose Share the final row lookup across batch vector query results. Previously, `search_vectors(...).to_arrow()` planned and read a separate lookup for every query, including overlapping hits. Read the union of final row IDs once through the existing table reader, then reconstruct each query's result from its own membership. Keep per-query limits, post-filters, projection schemas, input query order, and the execution's pinned snapshot. The internal row ID used for reconstruction is removed unless requested. Single-query batches keep their existing lookup path. Candidate search and scoring are unchanged; the shared table contains only the projected final hits. In a local warm-cache lookup benchmark (20,000 rows, 32 dimensions, K=10, id/content projection, five measured runs), 32 identical queries decreased from 47.89 ms to 2.02 ms, and 32 independent queries from 49.84 ms to 11.54 ms. Both changed from 32 logical lookups to one, with equal output tables. These measurements cover final lookup only, not complete query latency or remote storage. ### Tests - 246 tests passed across batch lookup, snapshot consistency, batch raw scans, streaming refinement, vector filters, and multimodal tables; three unrelated distributed-data tests were deselected. - New cases cover serial and parallel split reads, overlapping and independent results, per-query limits, pre/post-filters, empty and missing results, projections and hidden row IDs, MAP-key projection, BLOB payloads, concurrent updates/deletes, historical snapshots and retained tags, query reuse, and a real native vector index. - Flake8, Python 3.6 syntax parsing, and `git diff --check` passed. - Rust main planner validation remains with CI. The locally available 0.3.0 wheel fell back to Python planning and does not establish native planner coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
