morningman opened a new issue, #66322: URL: https://github.com/apache/doris/issues/66322
## Goal Track the work required to add production-ready, read-only Lance support to Apache Doris. Lance is a columnar data format designed for analytics and AI workloads. Doris should be able to discover and query Lance datasets through a Lance Catalog, read a dataset directly through file TVFs, push compatible predicates down to Lance, and perform vector similarity search. ## Initial implementation The foundational implementation is #65730, targeting `branch-4.1`. It introduces the Lance reader through File Scanner V2 and covers the FE catalog, planner, TVF, vector-search, BE reader, type conversion, tests, and test environment. Documentation is tracked in [apache/doris-website#4024](https://github.com/apache/doris-website/pull/4024). ## Scope and progress ### Catalog and metadata - [ ] Support a filesystem Lance Catalog backed by a local path, `file://`, or `s3://`. (#65730) - [ ] Support Lance REST Namespace catalogs, including Bearer Token, API Key, custom headers, and temporary storage credentials. (#65730) - [ ] Map multilevel Lance Namespaces to Doris databases and support metadata operations such as `SHOW DATABASES`, `SHOW TABLES`, and `DESC`. (#65730) - [ ] Preserve snapshot consistency by pinning the dataset version during planning. (#65730) ### Query execution - [ ] Add the Lance reader to File Scanner V2 and scan Lance Fragments in parallel. (#65730) - [ ] Support column pruning and the documented Lance/Arrow-to-Doris type mappings. (#65730) - [ ] Push semantically compatible scalar predicates down to Lance while retaining unsupported predicates in Doris. (#65730) - [ ] Expose pushed predicates in `EXPLAIN` for observability. (#65730) ### Direct dataset access - [ ] Read Lance datasets directly through the `s3()` TVF. (#65730) - [ ] Read Lance datasets directly through the `local()` TVF. (#65730) ### Vector search - [ ] Add the `vector_search()` TVF for Top-K similarity search. (#65730) - [ ] Support indexed and flat search with `l2`, `cosine`, and `dot` distance metrics. (#65730) - [ ] Return the computed distance through the `_distance` column. (#65730) ### Quality and documentation - [ ] Add BE and FE unit tests for the reader, type conversion, predicates, catalog behavior, and FE/BE contracts. (#65730) - [ ] Add regression coverage for catalog queries, predicate pushdown, file TVFs, and vector search. (#65730) - [ ] Publish the Lance Catalog user documentation. ([apache/doris-website#4024](https://github.com/apache/doris-website/pull/4024)) ## Current limitations and follow-up opportunities The initial implementation is read-only. The following capabilities are not included in #65730 and should be tracked by dedicated issues or PRs before being checked here: - [ ] Write support, including create, insert, update, and delete operations. - [ ] Time travel and explicit version selection. - [ ] REST Namespace managed versioning support in the BE reader. - [ ] Full-text search and hybrid search. - [ ] Additional file TVFs and broader path handling, including glob patterns and multiple datasets per TVF invocation. - [ ] Additional Lance/Arrow types currently documented as unsupported, including Blob v2, Arrow JSON Extension, BFloat16 Extension, `null`, and `duration`. ## Completion criteria - The implementation, regression tests, and user documentation are merged. - Supported behavior and known limitations are documented. - Follow-up work discovered during review or adoption is linked from this issue. ## Maintenance - Add new Lance-related issues and PRs under the closest section. - Check an item only after its implementation is merged and verified. - Keep known limitations explicit until they are implemented or intentionally declared out of scope. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
