zjw1111 opened a new issue, #361: URL: https://github.com/apache/paimon-cpp/issues/361
### Search before asking - [x] I searched existing issues and found no issue specifically tracking vector and full-text search in File Index. #173 tracks general File Index writing. ### Motivation Paimon C++ has File Index infrastructure for per-data-file indexes, but does not yet provide an end-to-end File Index search path for vector similarity or full-text queries. Applications should be able to build these indexes with their data files and use them to select matching file-local rows without requiring a Global Index. ### Solution Add Vector Search and Full Text Search support to File Index: - Build and serialize vector and full-text index artifacts through the existing File Index write path and container format, and open them through File Index readers. - Add file-local search visits to `FileIndexReader` and route search requests from the read context to the appropriate index reader. - Apply returned file-local row positions to the data-file reader. Vector Search should return a TopK result independently for each data file and keep scores aligned with selected rows; expose scores through an explicitly projected `_INDEX_SCORE` column. - Support Full Text Search row selection. Score output for Full Text Search can be handled separately. - Add focused tests for write/read round trips, null or empty indexes, multiple data files and batches, score alignment, and columns with more than one File Index type. The initial implementation can target append-only/raw-file reads. Predicate combination, deletion-vector and row-range pre-filtering, Merge-on-Read/Data Evolution, and cross-file global TopK should be handled in follow-up work so that filtering does not silently change TopK semantics. ### Anything else? This builds on the File Index writing work in #173; it is specifically about vector/full-text index implementations and their search read path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
