Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5212293290
Thanks for starting the upstream discussion. The Rust-core findings are useful for future distributed-index work, and lance-format/lance-c#55 can remain as that future track. However, this does not change the confirmed scope for Doris 4.2, and Q3 is no longer open: **lance-c extensions are not a prerequisite for #66497 in 4.2**. Features that depend on those extensions must be deferred. Therefore, Wave 1 should not be described as blocking the 4.2 implementation or as de-risking its current schedule; it is outside the release's critical path. For #66497, the next design revision should be simplified around the existing branch-4.1 baseline and the pinned lance-c 0.1.2 one-shot APIs: - `lance_dataset_create_vector_index` and `lance_dataset_create_scalar_index`; - full rebuild/replacement through the existing `replace=true` path; - `lance_dataset_index_count` and `lance_dataset_index_list_json` for the currently available metadata; - `lance_dataset_drop_index`; - execution of the heavy direct-Dataset call on a selected BE for Directory Catalog. The 4.2 design should explicitly exclude fragment-scoped distributed builds, shared-model distribution, BE-to-FE uncommitted segment transfer, coordinator segment commit, incremental `BUILD INDEX`, physical merge, native progress, cooperative cancellation, and per-segment coverage/statistics extensions. It should also avoid promising reliable `CANCEL BUILD INDEX` semantics when the underlying one-shot native call cannot be interrupted. The REST path is separate: lance-c additions do not expand REST Catalog capabilities. REST index lifecycle operations must continue to use the pinned Namespace request/response models and the server's implemented capabilities. In particular, newer lance-c APIs would not provide REST `OR REPLACE`, incremental build, cancellation, or additional request parameters. One future-facing note for lance-c#55: `pb::IndexMetadata` bytes are not directly accepted by the pinned Java SDK. `Index.Builder` takes the metadata as individual fields and exposes no public `parseFrom(byte[])` entry point, so the cross-SDK design still needs an explicit Java decoder/adapter and a version-compatibility contract. This does not need to be solved for #66497 because that distributed path is out of scope for 4.2. Please make the next revision a 4.2 design based only on the existing APIs, covering BE selection/execution, supported index types and properties, `replace=true` semantics, privileges, concurrency, versioning, failure behavior, metadata refresh, local-path visibility, and the restricted Namespace 0.7.7 REST capability matrix. Query-side work should remain limited to validating index consumption through the existing `vector_search()` and scalar predicate paths. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
