u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5208309962
Thank you @Gabriel39 for the patient and precise guidance across both review rounds — each round has made this design substantially stronger. Below is the v3 full revision, addressing every round-2 point directly and incorporating several self-identified corrections. # Revised Design Proposal v3 — Asynchronous Lance Index Lifecycle > Draft for apache/doris#66497, prepared 2026-08-07. Incorporates Gabriel39's round-2 review (2026-08-06 13:15 UTC) + self-identified errata. This revision has passed three independent adversarial reviews (research-consistency, review-coverage, source-level fact-check); every load-bearing claim carries pinned-source evidence, and the distributed build path is verified by an end-to-end PoC against lance-core 9.1.0-beta.3 (fragment-scoped uncommitted build → coordinator commit → k-NN consumption). Post as a full standalone revision. This revision incorporates the second-round review from @Gabriel39: REST is now an explicit capability matrix, SHOW/CANCEL BUILD INDEX get a concrete external-job integration design, reconciliation requires job-specific provenance, DROP uses the same durable job record, and the distributed segment contract is specified. It also contains several self-identified corrections to v2. /cc @Gabriel39 @zhangstar333 ### Major changes from v2 1. **REST capability matrix** replaces uniform-behavior language: a per-operation/per-property support table; unsupported combinations are rejected at analysis time where statically known, otherwise surfaced as a clear capability error. 2. **Job-specific provenance**: every mutation job persists a client-generated `operation_id` before dispatch. Reconciliation requires job-specific evidence — for Directory, the expected segment UUID set plus the `operation_id` recorded in the Lance transaction properties at commit time; for REST, the server-returned `transaction_id` (see the honest boundary in §10). A matching postcondition alone is never accepted as success — including for DROP. 3. **Concurrency semantics stated in both layers**: the Lance format documentation defines same-name concurrent CreateIndex as last-committer-wins; the pinned implementation enforces it via a retryable commit conflict + rebase on the first attempt. Either way, a name/definition match never proves attribution. 4. **DROP is durable**: it uses the same LanceIndexJob record (it usually transitions quickly), closing the post-commit failover gap — and its reconciliation follows the same provenance rules. 5. **Distributed segment contract**: a pinned lance-c/Java-SDK compatibility requirement, a worker-result validation checklist, full-final-segment-set commits for incremental BUILD, an explicit shared IVF model strategy, a defined BE→FE segment-metadata schema, and no credentials in the edit log. 6. **SHOW/CANCEL BUILD INDEX integration design** for external jobs: registry, proc source, catalog qualification, filtering, privileges, output schema, and state mapping (current implementations verified internal-only on both master and branch-4.1). 7. **Metadata**: SHOW INDEX.Index_type reports the physical type (IVF_PQ); REST SHOW fields are capability-dependent. 8. **Type matrix narrowed**: vector element types are FLOAT16/FLOAT32 only (see corrections). 9. **Execution model locked**: Directory = BE via extended lance-c; no external build service. The FE-side build remains an experimental, disabled-by-default development fallback using the Java SDK's existing uncommitted-build primitives. 10. **Provenance-carrying commits use the general Transaction path**: the one-shot `Dataset.createIndex` / `commitExistingIndexSegments` / `dropIndex` APIs do not carry transaction properties in the pinned SDK; commits that must record `operation_id` go through `Transaction.Builder.transactionProperties` + `operation.CreateIndex` + `CommitBuilder` (verified end-to-end through JNI in the pinned sources). ### Corrections to v2 (self-identified) - **Concurrent same-name CreateIndex — both layers, precisely.** The [Lance format transaction documentation](https://lance.org/format/table/transaction/) states: two concurrent CreateIndex operations are allowed; with the same name, "the second operation will win and replace the first" (`docs/src/format/table/transaction.md:172-173`). The pinned implementation reaches that semantics via conflict + rebase: `check_create_index_txn` returns a retryable conflict for a same-name regular index on the first attempt (`rust/lance/src/io/commit/conflict_resolver.rs:577-617`), and the Java `CommitBuilder.maxRetries` defaults to 0, so without retries the caller observes the conflict error. v2 saw only the implementation layer; the design must assume the doc layer for external writers (retried/ rebased commits can replace Doris-created indexes) — hence provenance in §10. - **UINT8 element type**: the Rust core treats UInt8 vectors as binary vectors and allows only Hamming distance (`rust/lance/src/index/vector/utils.rs:193,204-211`). v2's matrix (UINT8 with L2/COSINE/DOT) was inconsistent. UINT8 is deferred until a binary-vector/HAMMING decision is made. INT8 is deferred: it is advertised by the pinned lance-c creation surface and accepted by the Rust core (L2/COSINE/DOT allowed), but it is an undocumented branch there and lacks end-to-end verification. FLOAT64 remains deferred — not advertised by the pinned lance-c creation surface (`lance.h:926` restricts to float32/float16/uint8/int8), although the Rust core accepts it. - **REST monitoring channel**: index progress is tracked through the Namespace list/stats operations — the spec states "Index creation is handled asynchronously. Use the ListTableIndices and DescribeTableIndexStats operations to monitor index creation progress" (spec.yaml:1633-1634). The optional `transaction_id` is a generic long-running-operation handle, not the index progress channel. - **FTS deferral rationale**: the BE reader already has a full-text search path (`be/src/format_v2/table/lance_reader.cpp:521` on branch-4.1). Lance INVERTED is deferred because Doris has no SQL-level FTS semantics for it yet — not because the query path is absent. - **TVF per-segment coverage dropped from Phase 1 (declared downgrade)**: v2 promised per-segment row/fragment counts in the physical-segment TVF. No pinned API exposes them: the lance-c list JSON is hard-coded to `{name, uuid, columns, type, dataset_version}` per entry (lance-c `src/index.rs`), and the Namespace stats model exposes only row counts (spec.yaml:3206-3234). This becomes lance-c extension item ⑥; the TVF ships without those columns until then. Reviewer approval for this deferral is requested explicitly. - **Index-name case behavior (research correction)**: an earlier internal research note claimed Lance lowercases index names. The pinned sources show Lance index names are **case-sensitive, exactly matched, and stored as-is** (`rust/lance/src/index/create.rs:202`, `rust/lance/src/index/api.rs:211`; no normalization in builder/Python/Java layers). The §8 design — Doris-layer case-insensitive resolution over Lance's exact-match storage — stands precisely because `idx`/`IDX` coexistence is a real external state. ## 1. Goals and Phase-1 boundary Phase 1 provides a production-oriented lifecycle for user-visible Lance indexes: - create IVF_PQ vector indexes and BTREE/BITMAP scalar indexes (Directory full; REST restricted to the request model per §5); - incrementally index uncovered fragments and atomically replace an index by name (Directory); - drop an index by name (Directory and REST); - inspect logical definitions, aggregate coverage, and the logical index count (Directory full; REST capability-dependent); - durable status, cancellation, concurrency limits, provenance-based reconciliation, and metadata refresh; - REST behavior follows the capability matrix in §5; everything outside it is rejected explicitly. Lance dataset manifests or the Namespace service remain the authoritative index metadata. Doris persists only its own job state and reconciliation evidence. Historical-version tables remain read-only for index mutation; inspection uses the selected snapshot, mutation targets only the latest writable state. An FE-side synchronous build is not part of the production implementation. A development fallback (experimental, disabled by default, limited by dataset-size/fragment/concurrency settings) MAY use the pinned Java SDK's existing uncommitted primitives (`IndexOptions.withFragmentIds` / `withIndexUUID`, javadoc-verified uncommitted + fragment-scoped). The production objection is compute placement in the control plane, not missing APIs. ## 2. SQL surface Unchanged from v2 except the property-key question (Q2): - vector: `CREATE INDEX ... USING ANN PROPERTIES ("index_type"="IVF_PQ", "metric_type"="l2", "num_partitions"="256", "num_sub_vectors"="16")` (key naming is Q2 below; example uses the proposed key); - scalar: `USING BTREE` / `USING BITMAP`; - `CREATE OR REPLACE INDEX` = full rebuild + atomic name replacement; - `BUILD INDEX` = incremental coverage of uncovered fragments; - `SHOW INDEX FROM t`; `SHOW BUILD INDEX FROM db WHERE ...`; `CANCEL BUILD INDEX ON ctl.db.tbl (job_id)`; `DROP INDEX [IF EXISTS]`. Statement semantics (delta from v2): - Create, replace, build, **and drop** each return after a durable job is accepted. Drop typically completes within one commit cycle but is equally durable. - REST IF NOT EXISTS reconciliation compares only the fields observable through the Namespace stats model (name, columns, index type, distance type); properties the REST path cannot express are rejected at analysis for REST tables, so no hidden definition skew is possible (§5). - All other semantics (OR REPLACE convergent-but-not-idempotent, single column, analysis-time rejection of unknown types/properties) unchanged. ## 3. Neutral command model and catalog dispatch Unchanged from v2: `ParsedIndexSpec` → resolve catalog/database/table → `InternalIndexProvider` or `LanceIndexProvider` → provider validation → internal schema change or Lance external-index job. Lance definitions never enter Doris internal `catalog.Index` serialization. ## 4. Durable job framework and SHOW/CANCEL BUILD INDEX integration ### Job record (persisted via edit log) - job id, operation kind (CREATE / REPLACE / BUILD / **DROP**), catalog/database/table identity, dataset URI or Namespace table id; - logical index name and normalized definition; - **`operation_id` (client-generated UUID), persisted before any dispatch**; - starting dataset version; expected/accepted segment UUID set; old segment UUIDs (replace); - executor identity, worker task ids, or Namespace transaction id (persisted when returned); - state, progress, timestamps, sanitized failure information; - **no credentials** — workers obtain or refresh catalog credentials when a task or retry is dispatched. ### States Unchanged from v2: `PENDING → PREPARING → BUILDING → COMMITTING → REFRESHING → FINISHED`, plus `FAILED / CANCELLING / CANCELLED / COMMITTED_REFRESH_PENDING / OUTCOME_UNKNOWN`. DROP traverses the same machine (typically PENDING → COMMITTING → REFRESHING → FINISHED in one cycle). FE failover reconstructs unfinished jobs from the edit log and resumes polling, reconciliation, commit, or refresh; it does not blindly restart native training. ### Resource admission Unchanged from v2, restated here because it gates the production path: jobs are admitted through explicit per-catalog and cluster-wide concurrency limits; BE execution is associated with workload-management accounting where the executor supports it. ### Integration with the existing job commands The grammar already accepts the required shapes on both master and branch-4.1: `SHOW BUILD INDEX ((FROM|IN) db)? wildWhere? sortClause? limitClause?` and `CANCEL BUILD INDEX ON <multipart table> (job_ids)` (DorisParser.g4: master fe-sql-parser :392-393/:644-646; branch-4.1 fe-core :390-391/:640-642). The current implementations are internal-only on both branches: `CancelBuildIndexCommand.validate` calls `Util.prohibitExternalCatalog` (:80) and resolves an `OlapTable` + `SchemaChangeHandler.IndexChangeJob`; `ShowBuildIndexCommand` builds the internal proc path `/jobs/<dbId>/build_index` (:188-191) with internal-specific columns (`PartitionName`, `AlterInvertedIndexes`, TITLE_NAMES :68-74). The design: - **Registry**: a catalog-agnostic external-index job registry in FE, edit-log persisted and rebuilt on replay, sibling to the internal `IndexChangeJob` registry. Lance jobs register here; the internal path is untouched. - **Proc/SHOW source**: `SHOW BUILD INDEX` unions internal and external jobs. A new `JobType` column (`INTERNAL` / `EXTERNAL_LANCE`) distinguishes rows. Internal-only columns stay empty for external rows (Lance jobs are table-level, not partition-level). External rows populate the generic columns: JobId, DbName (catalog-qualified), TableName, IndexName, State, Progress, CreateTime, FinishTime, Msg. - **Catalog qualification**: CANCEL's multipart table name already parses `catalog.db.tbl` (grammar + `LogicalPlanBuilder.visitCancelBuildIndex` are catalog-aware). SHOW's FROM clause currently takes a single `identifier` only; it is extended to accept a catalog-qualified `catalog.db` (preferred, consistent with the table qualification) — exact grammar detail in the PR. - **Filtering and privileges**: external rows are visible with table-level `SHOW` on the resolved external table; CANCEL requires table-level `ALTER`. - **State mapping**: external-index job states are displayed verbatim (`PENDING … OUTCOME_UNKNOWN`); WHERE filtering on `State` uses these same job-state values; documentation maps them to the internal lifecycle. - **Cancellation semantics**: unchanged from v2 (persisted before workers are signaled; fences COMMIT; generation-checked results). For REST-tracked operations there is no Namespace cancel operation: cancellation fences the Doris-side commit/refresh and stops polling; a later external completion is handled by the provenance rules in §10. - **Conflict reporting**: automatic conflict retry decisions are never made by parsing native exception strings; typed conflict results are used where the provider exposes them (unchanged from v2). ## 5. Execution model ### Directory Catalog: BE pipeline + segment contract Production execution requires extending lance-c (prerequisite, §12). FE orchestrates; BE workers build; FE performs the single atomic logical-index commit through the pinned Java SDK. **Commit path (verified against pinned sources):** provenance-carrying commits go through the general transaction path — `Transaction.Builder.transactionProperties(Map)` (Transaction.java:176-179) carrying the job's `operation_id`, with `operation.CreateIndex`, executed via `CommitBuilder` (JNI wiring: lance-jni/src/transaction.rs:830-846). The one-shot `Dataset.createIndex(IndexOptions)`, `commitExistingIndexSegments`, and `dropIndex` paths do **not** carry transaction properties (they build the commit internally with `None` properties; create/commit-existing: rust/lance/src/index.rs:1538-1547, drop: rust/lance/src/index.rs:1177-1194) and are therefore not used for provenance commits. **Segment contract:** - **Compatibility**: the pinned lance-c and the pinned Java SDK must be verified against the same Lance core index-metadata format. The commit-side API surface is verified in the pinned sources: `Index.Builder` (Index.java:187-258), `commitExistingIndexSegments` (Dataset.java:1176), `IndexOptions.withFragmentIds`/`withIndexUUID` (IndexOptions.java:135-157), and the commit path's own validation (non-empty set, unique UUIDs, non-overlapping fragment coverage; rust/lance/src/index.rs:100-131). **The full loop is additionally verified end to end by a PoC on lance-core 9.1.0-beta.3**: fragment-scoped uncommitted builds over disjoint fragment subsets → single coordinator commit → dataset version increment by exactly one → k-NN query consumes the resulting multi-segment logical index. An end-to-end Doris test will prove the same for a BE-built (lance-c) segment committed via the Java SDK. - **BE→FE segment metadata schema**: per segment, workers report: `uuid`, field IDs, `name`, `datasetVersion`, `indexVersion`, `indexType`, fragment coverage, and the `indexDetails` protobuf bytes (`createdAt`/`baseId` optional; `createdAt` is filled by the commit side). This is the exact field set the commit path requires — validated statically (JNI `index_metadata_to_segment`, lance-jni/src/blocking_dataset.rs:1218-1243, rejects segments missing fragment coverage or index details; `segment.fields` is validated against the indexed column's field id, rust/lance/src/index.rs:1469-1475) and empirically (PoC: a Builder-reconstructed Index missing any of these fails before commit). Bare UUIDs are not sufficient. - **Segment identity**: every segment carries its **own** UUID — `commitExistingIndexSegments` rejects duplicate segment UUIDs (PoC-verified); the "shared UUID" wording in `withIndexUUID`'s javadoc belongs to the separate merge flow. FE pre-assigns per-worker UUIDs (via `withIndexUUID` where the executor supports it) so the expected UUID set is known before dispatch; this set backs the provenance rule in §10. - **Worker result acceptance**: before accepting a worker result, FE validates per segment: field IDs, physical index type and parameters, starting dataset version, exact assigned fragment coverage (disjoint across workers and equal to the assignment), UUID uniqueness, and artifact location beneath the dataset index directory. - **Full final segment set**: CREATE/REPLACE commit the complete segment set for the starting snapshot. Incremental BUILD commits the **complete final logical segment set** — existing compatible segments plus newly built ones — not only the new segments (PoC-verified; note that a follow-up commit on an already-committed name uses replace semantics in the pinned SDK). - **IVF model strategy (Phase 1): shared model.** A dedicated training task derives IVF centroids and PQ codebooks once from a sample of the starting snapshot; all build workers use the shared model — concretely, the pinned Java SDK supports exactly this split (centroids trained once via `VectorTrainer.trainIvfCentroids`, distributed to builders through `IvfBuildParams.setCentroids`; PoC-verified across workers); on the BE path this depends on lance-c extension ① accepting caller-supplied centroids/codebooks (§12). This keeps segments mutually compatible for a future physical merge and yields uniform recall across segments. Independent per-segment models (valid for fan-out query) are rejected for Phase 1 because they preclude later merge. Physical segment merge itself is not performed in Phase 1. - **Credentials**: never persisted in the job record; obtained/refreshed at task dispatch (§4). - **Local/file restriction (restored from v2)**: a local/file dataset is mutable only when the deployment guarantees that the path is visible with identical contents to the selected executor and to every FE that may reconcile or finalize the operation; otherwise mutation is rejected. Local filesystem support is intended primarily for single-node development and tests; it must not silently depend on the current master FE's local disk. **FE experimental fallback**: unchanged gating (experimental, disabled by default, size/fragment/concurrency limits). Implementation uses the Java SDK's existing uncommitted primitives; no lance-c dependency. ### REST Namespace Catalog: capability matrix Capability detection is behavioral: the pinned Java interface represents unsupported operations as default methods throwing a typed `UnsupportedOperationException` bound to spec error code 0 (`LanceNamespace.java:402-448`, `errors/ErrorCode.java:22`); there is no capability-advertisement API. Unsupported combinations fail during analysis where statically known, otherwise with a clear capability error. Doris never falls back to direct Dataset mutation with vended credentials. All REST mutations use the durable job record. | Operation | REST Phase-1 (pinned Namespace 0.7.7) | |---|---| | `SHOW INDEX` | **Yes** — via `ListTableIndices` + `DescribeTableIndexStats`. Fields limited to the stats model: name, UUID, columns, status; index type, distance, indexed/unindexed row counts, physical index count (spec.yaml:3149-3234). Fragment counts and creation properties are unavailable → corresponding SHOW fields are empty. | | `CREATE INDEX` | **Restricted** — only what `CreateTableIndexRequest` carries: column, index_type, name, distance_type (FTS fields unused; spec.yaml:3049-3108). IVF build properties (`num_partitions`, `num_sub_vectors`) and BTREE `zone_size` are **rejected** for REST tables. | | `CREATE OR REPLACE INDEX` | **Rejected** — the request model has no replace semantics and the spec defines `TableIndexAlreadyExists` (error code 7); a non-atomic drop+create pair is not offered under OR REPLACE. | | `BUILD INDEX` | **Rejected** — no incremental/uncovered-fragment operation exists in the Namespace API. | | `DROP INDEX` | **Yes** — via `DropTableIndex` (LanceNamespace.java:445); still uses the durable job record. | | `lance_index_segments` TVF | **Rejected** — the stats model exposes no per-segment metadata. | Monitoring and provenance: progress is polled via `ListTableIndices` / `DescribeTableIndexStats` (the spec-sanctioned channel). If the server returns a `transaction_id`, it is persisted as the REST job's only server-side evidence (§10). The pinned request model cannot carry a client operation ID or transaction properties; adding a namespace-level idempotency/provenance key is listed as a spec-evolution follow-up. (Alternative considered: make Namespace spec/API changes — replace semantics, build parameters, incremental build — an explicit prerequisite and offer uniform REST behavior. The matrix above is the Phase-1 proposal; the spec-evolution path can be a follow-up with the Lance community.) ## 6. Initial type and property matrix - **IVF_PQ**: SQL category ANN + `index_type=IVF_PQ`; exactly one fixed-size-list column; element types **FLOAT16, FLOAT32** (others deferred, see corrections); field and elements non-null; metrics L2/COSINE/DOT; `num_partitions`, `num_sub_vectors` optional positive integers (`num_sub_vectors` must divide the dimension); `num_bits` fixed to 8. - **BTREE**: one column; integral/floating numeric, DECIMAL, STRING, DATE/DATETIME/TIMESTAMPTZ per the existing Lance pushdown support; nullable columns gated by the NULL-predicate tests; optional `zone_size` positive integer (Lance default 4096, `DEFAULT_BTREE_BATCH_SIZE`, rust/lance-index/src/scalar/btree.rs:91). - **BITMAP**: one column; BOOLEAN, integral numeric, STRING, DATE; explicit NULL predicate tests; no user-visible properties. - **Deferred**: IVF_FLAT (end-to-end query verification), Lance INVERTED (needs Doris FTS SQL semantics; to be exposed as FTS), UINT8/INT8/FLOAT64 (see corrections). ## 7. Logical and physical metadata surfaces - **SHOW INDEX**: unchanged 13-column schema (ShowIndexCommand.java:52-66, identical on master and branch-4.1). One row per logical index/column: the pinned Lance list APIs return one entry per physical segment — multiple same-named entries for a multi-segment logical index (PoC-verified) — so SHOW INDEX **groups by logical name**. `Index_type` reports the **physical Lance type** (`IVF_PQ`, `BTREE`, `BITMAP`) rather than the SQL category. `Properties` carries bounded logical details only (physical type, metric, parameters, aggregate indexed/unindexed rows and fragments from the logical description) — never segment arrays. The logical index count is the number of user-visible logical index descriptions (i.e., distinct grouped names), excluding Lance system indexes, and is not derived from physical manifest entries; no separate count statement is added. - **`lance_index_segments` TVF**: catalog, database, table, logical index name, segment UUID, physical index type, dataset version, creation timestamp when available. **Per-segment row/fragment counts are deferred** (see corrections): the lance-c list JSON carries per-entry `{name, uuid, columns, type, dataset_version}` and no fragment coverage or row counts; this becomes lance-c extension item ⑥. - **REST**: SHOW fields are capability-dependent per §5; the TVF is rejected. ## 8. Authorization and identifier semantics Unchanged from v2: mutation + CANCEL require table-level ALTER; SHOW surfaces require table-level SHOW. Doris resolves user-supplied index names case-insensitively (quoted and unquoted identifiers alike) and preserves display case; new case-only duplicates are rejected; pre-existing external collisions surface an explicit ambiguity error on mutation. Note the layered reality (corrected research): Lance itself is case-sensitive and exact-matching (create.rs:202, api.rs:211), so Doris's case-insensitive resolution is a Doris-layer policy over exact-match storage — which is exactly why external case-only collisions are a real state that must be handled explicitly. ## 9. Versioning, concurrency, and visibility - Successful create/replace/build/drop commits a new dataset version atomically; queries pinned to older versions are unaffected; historical-version mutation is rejected. - **Concurrent same-name CreateIndex (external writers)**: the format documentation defines last-committer-wins; the pinned implementation reaches it via retryable conflict + rebase (see corrections). Doris serializes same-name jobs internally (case-insensitive) but does not control external writers, which may retry and therefore replace; reconciliation never treats a name/definition match as proof of attribution (§10). - **Conflicts** (per transaction.md:175-194): Overwrite/Restore/UpdateMemWalState conflict with index creation. Rewrite conflicts only with overlapping fragments and no stable row ids and no fragment reuse index; DataReplacement conflicts only with overlapping fragments when the replaced column is being indexed — these are surfaced as retryable. Concurrent append is compatible; appended fragments are reported as unindexed and covered by a later BUILD. - Doris does not hold table metadata locks for the lifetime of a build. - Conflict-retry decisions are never made by parsing native exception strings (typed conflict results only). - Queries over a partially indexed table must combine indexed results with scans of unindexed fragments; end-to-end tests must prove result equivalence with index use disabled. ## 10. Failure recovery, provenance, and metadata refresh The three outcome classes from v2 are retained, with a strengthened and honestly-bounded success criterion. ### Confirmed pre-commit failure Unchanged: job FAILED; no post-DDL refresh; unreferenced artifacts reclaimed by Lance cleanup per retention. ### Confirmed commit — requires job-specific evidence - **Directory (create/replace/build)**: the logical index in the reopened latest snapshot is backed by exactly the job's expected segment UUID set — pre-assigned by FE before dispatch where the executor supports it (§5), so the expectation is deterministic rather than observed; the commit itself carries the job's `operation_id` in the Lance transaction properties via the general Transaction path (§5; runtime-verified readable back from the new version's transaction). - **Directory (drop)**: the pinned SDK has no `DropIndex` operation class, and the one-shot `Dataset.dropIndex` commits internally with `None` properties (rust/lance/src/index.rs:1177-1194), so it is not used for provenance commits. A provenance-carrying drop goes through the same general Transaction path, expressed as `operation.CreateIndex` with an empty new-index list and `removedIndices` set to the index metadata loaded for the name before dispatch (`withRemovedIndices`, operation/CreateIndex.java:92-94) — mirroring the Rust one-shot's internal operation — with `transactionProperties` carrying the job's `operation_id`. That recorded `operation_id` is the job-specific evidence; the exact removed segment UUID set is further corroboration. The postcondition "name is absent" is corroboration only — an external writer can independently drop the same name (false positive) or recreate it after our drop (false negative). Without transaction-properties evidence, a Directory drop follows the unknown-outcome rule below. - **REST**: the pinned Namespace API cannot carry a client operation ID in index requests. The only job-specific evidence is the server-returned `transaction_id` (persisted when returned), matched via the generic transaction API (`DescribeTransaction`, LanceNamespace.java:754) where the server supports it. A matching logical definition observed through list/stats is corroboration only. On a confirmed commit, Doris invalidates/refreshes metadata and emits the external-DDL refresh edit log; a refresh failure transitions the job to COMMITTED_REFRESH_PENDING with retry (unchanged). ### Unknown or post-commit failure Reconcile by reopening the latest snapshot / querying the Namespace state, as in v2 — **but** if no job-specific evidence is available (expected: FE failover before persisting the returned transaction_id; a drop without transaction-properties evidence; any REST operation on a server that returns no transaction_id), the job remains **OUTCOME_UNKNOWN** with a diagnostic. It is never marked successful from a matching postcondition alone — this is required precisely because same-name CreateIndex is last-committer-wins for external writers. A reconciled IF NOT EXISTS / IF EXISTS no-op still invalidates/refreshes Doris metadata when newer external state is observed (unchanged). ## 11. Test plan v2 coverage, plus: - capability matrix: every rejected REST combination fails with the designed error (analysis-time vs capability error); - REST IF NOT EXISTS compares only stats-observable fields (name, columns, type, distance); - provenance: Directory commit carries `operation_id` in transaction properties (read back from the new version's transaction); commit-with-lost-response reconciled via UUID set; drop commits are expressed as `operation.CreateIndex` with `removedIndices` over the general Transaction path (verified through JNI) and reconciled only via transaction-properties evidence; matching-postcondition-without-evidence stays OUTCOME_UNKNOWN (including REST with no transaction_id returned); - external same-name replacement during a Doris job (retried create replacing ours) is detected and attributed correctly; - DROP failover: drop commits externally, FE fails before refresh → resumed from the durable record; - incremental BUILD commits the full final segment set (existing + new), validated against duplicate-UUID / overlapping-coverage rejection; - BE→FE segment metadata schema: segments missing fragment coverage or index details are rejected before commit (mirroring the JNI conversion requirements); - shared-model build: segments built by different workers are merge-compatible and fan-out query recall is uniform; - SHOW/CANCEL BUILD INDEX: union display, JobType column, catalog qualification, privilege filtering, external-index job states, cancellation fencing; - resource admission: per-catalog and cluster-wide concurrency limits enforced; - local/file visibility restrictions (mutation rejected when the path is not uniformly visible); - job record contains no credentials (edit-log content test); - REST monitoring via list/stats polling; transaction_id persisted when present. ## 12. Confirmed design decisions and implementation decomposition 1. **Execution**: Directory = BE via extended lance-c, production-gated until the required APIs land; REST = Namespace capability matrix; no external build service (deployment/auth/ops dependency for a core feature is not justified); FE sync build = experimental fallback only. 2. **lance-c extensions (prerequisite)**: ① fragment-scoped uncommitted index builds, including caller-supplied shared IVF centroids / PQ codebooks (so the §5 shared-model strategy reaches BE workers); ② segment metadata serialization/transfer (BE→FE per §5 schema); ③ progress reporting; ④ cooperative cancellation; ⑤ segment merge / commit-existing; ⑥ per-segment coverage/stats exposure (feeds the TVF deferral). Upstream engagement starts now — lance-c is actively landing related distributed APIs (v0.1.3 added segment-selection for distributed search). 3. **Segment contract**: pinned compatibility, BE→FE metadata schema, validation checklist, full-final-set commits, shared IVF model, no persisted credentials. 4. **Commit path**: provenance commits use the general Transaction path (`Transaction.Builder.transactionProperties` + `operation.CreateIndex` + `CommitBuilder`); one-shot convenience APIs are not used where provenance is required. 5. **Provenance**: `operation_id` persisted pre-dispatch; success requires job-specific evidence; otherwise OUTCOME_UNKNOWN. REST boundary per §10 is stated explicitly. 6. **DROP durable**; SHOW/CANCEL BUILD INDEX integration per §4; resource admission per §4. 7. **SQL**: USING ANN + `index_type=IVF_PQ`; OR REPLACE = full rebuild; BUILD = incremental; SHOW Index_type = physical type. 8. **Types**: IVF_PQ (FLOAT16/FLOAT32) + BTREE + BITMAP only. 9. **REST**: capability matrix; mutation never bypasses the service. Work items (dependency-ordered; deliverable as separate PRs): 1. **lance-c extensions** (upstream; tracked prerequisite subtask, then pin bump); 2. **FE neutral command model** (ParsedIndexSpec + providers; internal behavior unchanged); 3. **Durable job framework + SHOW/CANCEL integration** (registry, edit log, proc union, privileges, resource admission); 4. **BE build pipeline** (training task, build workers, result validation, credential handling); 5. **Commit, provenance, reconciliation** (general Transaction path, segment contract enforcement, refresh retry); 6. **Metadata surfaces** (SHOW INDEX physical type, TVF, REST capability matrix); 7. **Tests and documentation** per §11 + lifecycle guarantees/limitations doc. ## Questions for reviewers - **Q1 — target branch**: #65730 landed on **branch-4.1** only; master currently has no Lance catalog/FE code, and the branches have diverged structurally (the SQL grammar lives in `fe/fe-sql-parser` on master vs `fe/fe-core` on branch-4.1; both verified). Should #66497 implementation target branch-4.1 only, master-first with a 4.1 backport, or dual-track? - **Q2 — metric property key**: since vector creation reuses the Doris category surface (`USING ANN`), I propose adopting the internal ANN property key `metric_type` (verified required by `AnnIndexPropertiesChecker`). One caveat: internal ANN currently accepts only `l2_distance` and `inner_product` — COSINE would be a newly introduced value for this key, and Lance value names differ (`l2`/`cosine`/`dot`). Options: (a) `metric_type` with Lance value names (`l2`/`cosine`/`dot`), documented as Lance-table-specific values; (b) `metric_type` with Doris value names mapped to Lance (`l2_distance`→L2, `inner_product`→DOT, plus new `cosine`→COSINE); (c) fully Lance-native key `metric`. My preference is (a) for key consistency with internal ANN while keeping Lance-native values. Build parameter names stay Lance-native (`num_partitions`, `num_sub_vectors`, `num_bits`) in all options. - **Q3 — lance-c packaging**: confirm the lance-c extension is tracked as an explicit prerequisite subtask (upstream issue + PRs, then a pin bump in Doris), with #66497 remaining the parent lifecycle issue until the native and Doris work is integrated end to end. Related: the TVF per-segment coverage columns (see corrections) ride on extension item ⑥ — please confirm the Phase-1 TVF ships without them. ## References - v2 proposal and round-1/round-2 review threads on this issue - #65730, #66340 - Lance transaction semantics: https://lance.org/format/table/transaction/ (pinned: `docs/src/format/table/transaction.md:162-194`); implementation conflict path: `rust/lance/src/io/commit/conflict_resolver.rs:577-617` (pinned v9.1.0-beta.3) - Lance distributed indexing: https://lance.org/guide/distributed_indexing/ - Lance Namespace operations + spec: https://lance.org/format/namespace/operations/ (pinned: spec.yaml:1633-1634, 3049-3234) - Pinned Lance Java SDK v9.1.0-beta.3: `Dataset.java` (:590/:607/:1106/:1176), `IndexOptions.java` (:135-157), `Index.java` (:187-258), `Transaction.java` (:176-179), `operation/CreateIndex.java` (:92-94), `CommitBuilder.java` (:75,:273); JNI: `lance-jni/src/transaction.rs:830-846`, `lance-jni/src/blocking_dataset.rs:1218-1243` - Pinned lance-c v0.1.2 (`include/lance/lance.h`); upstream lance-c v0.1.3 segment-selection APIs - Doris (master + branch-4.1): `CancelBuildIndexCommand.java` (:80), `ShowBuildIndexCommand.java` (:68-74,:188-191), `ShowIndexCommand.java` (:52-66), `DorisParser.g4` (master fe-sql-parser :392-393/:644-646; 4.1 fe-core :390-391/:640-642), `AnnIndexPropertiesChecker.java` (:42-46) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
