wangzhigang1999 opened a new issue, #11253:
URL: https://github.com/apache/paimon/issues/11253

   ### Search before asking
   
   - [x] I searched the existing issues for this problem.
   
   ### Motivation
   
   After appending embeddings to a Data Evolution table, a user may want to 
find the ten nearest rows before building an index. With the Python reader and 
`global-index.search-mode=full`, the following query scans the unindexed rows. 
By default, `to_arrow()` runs the search in the calling Python process, even 
when the table is on OSS.
   
   ```python
   hits = table.search(q, column="embedding", options={"metric": 
"l2"}).limit(10).to_arrow()
   ```
   
   On master `d90a902d`, the reader collects vectors and row IDs from all 
planned splits into one Arrow table. The scorer then iterates over batches, but 
the whole table stays in memory until scoring finishes. For an L2 Top-10 search 
on GIST1M (one million 960-dimensional vectors), peak process RSS reached 4,865 
MiB on local Parquet and 5,158 MiB on OSS. [Raw search on 
master](https://github.com/apache/paimon/blob/d90a902d0a4f11d8c8512b8f3ccd6d6a4e92e89a/paimon-python/pypaimon/execution/python/read/search/vector_search_read.py#L352-L365)
   
   ### Solution
   
   Score Arrow batches as they arrive and keep one Top-K heap across the scan.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to