wangzhigang1999 opened a new issue, #11253:
URL: https://github.com/apache/paimon/issues/11253
### Search before asking
- [x] I searched the existing issues for this problem.
### Motivation
After appending embeddings to a Data Evolution table, a user may want to
find the ten nearest rows before building an index. With the Python reader and
`global-index.search-mode=full`, the following query scans the unindexed rows.
By default, `to_arrow()` runs the search in the calling Python process, even
when the table is on OSS.
```python
hits = table.search(q, column="embedding", options={"metric":
"l2"}).limit(10).to_arrow()
```
On master `d90a902d`, the reader collects vectors and row IDs from all
planned splits into one Arrow table. The scorer then iterates over batches, but
the whole table stays in memory until scoring finishes. For an L2 Top-10 search
on GIST1M (one million 960-dimensional vectors), peak process RSS reached 4,865
MiB on local Parquet and 5,158 MiB on OSS. [Raw search on
master](https://github.com/apache/paimon/blob/d90a902d0a4f11d8c8512b8f3ccd6d6a4e92e89a/paimon-python/pypaimon/execution/python/read/search/vector_search_read.py#L352-L365)
### Solution
Score Arrow batches as they arrive and keep one Top-K heap across the scan.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]