u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5301314544
# Design Proposal v5.1 — Minimal One-Shot Lance Index Lifecycle for Doris 4.2 > Prepared for publication: 2026-08-15 (Asia/Shanghai)<br> > Target: Apache Doris `branch-4.1` / Doris 4.2<br> > Issue: [apache/doris#66497](https://github.com/apache/doris/issues/66497)<br> > Status: v5.1 final publication candidate; community confirmation requested below<br> > Review boundary last checked: 2026-08-15, through > [the final 4.2 scope clarification](https://github.com/apache/doris/issues/66497#issuecomment-5290050744) ## 0. Authority and revision summary This standalone document defines the proposed capability and safety contract for Lance index lifecycle management in Doris 4.2. If accepted, it supersedes v5. The v5 proposal and its errata remain issue-history provenance only: they provide no implementation guidance, compatibility contract, or roadmap. Requirements exist only where stated here; the final 4.2 scope clarification controls if review-history statements conflict. ### Context for this revision During earlier review rounds, v5 incorporated the correctness and implementation constraints under discussion, and I interpreted some of them as Doris 4.2 delivery requirements. The latest review clarified a narrower final 4.2 boundary. v5.1 preserves the accepted safety invariants, explicitly defers the broader mechanisms, and supersedes v5 for implementation. Relative to v5, this revision retains the accepted correctness rules and four contained requirements—credential rotation, unresolved-work quotas, possible-live worker accounting, and a small versioned schema contract—while removing broader reconciliation, identity, metadata, REST, and distributed or incremental architecture from the 4.2 gates. The contract below restates every retained requirement without depending on an earlier proposal. The baseline is Doris commit [`e3289c1a5df7558cb8e63d80379d4edebf9c498c`](https://github.com/apache/doris/commit/e3289c1a5df7558cb8e63d80379d4edebf9c498c), with FE `lance-core` `9.1.0-beta.3` and BE `lance-c` `0.1.2` backed by Rust Lance `4.0.1`. No 4.2 guarantee depends on upgrading or extending those APIs. Normative terms in this document use **MUST**, **MUST NOT**, **SHOULD**, and **MAY** in their ordinary requirements sense. A section explicitly marked future work is non-normative for 4.2. > **Review focus: Does this document accurately reflect the final Doris 4.2 > implementation boundary?** The requested confirmation appears in Section 11. ## 1. Scope and release boundary ### 1.1 Core safety contract Doris 4.2 executes one one-shot Lance mutation without retrying after dispatch may have occurred, fabricating an outcome, or reusing its logical name while the result is uncertain. Such a result remains `UNKNOWN` until an authorized operator accepts the late-commit risk through audited `FORCE_RELEASE`. ### 1.2 Required in 4.2 The 4.2 deliverable includes: - Directory Catalog `CREATE INDEX` for the initially verified vector and scalar index types; - `CREATE OR REPLACE INDEX` as a full same-name rebuild; - `DROP INDEX` by logical name; - `SHOW INDEX` for logical index metadata; - one bounded physical inspection surface for index UUID and dataset version; - basic `IF NOT EXISTS` and `IF EXISTS` behavior based on authoritative Lance metadata; - table-level `SHOW` and `ALTER` authorization; - an asynchronous durable job created before dispatch; - at-most-once execution on one selected BE through a hard resource-isolated worker process; - the compact lifecycle `PENDING -> RUNNING -> COMMITTED | NOT_COMMITTED | UNKNOWN`; - independent metadata-refresh status; - a same-name correctness fence retained by `UNKNOWN`; - one explicit audited `FORCE_RELEASE` path; - a small versioned schema contract and dataset-version revalidation before native invocation; - bounded unresolved-job admission and hard possible-live worker accounting; - credential rotation while a job is active or `UNKNOWN`; - focused crash, OOM, panic, failover, replay, privilege, and end-to-end tests; - user documentation for syntax, lifecycle, failures, limitations, and operator recovery. Directory Lance manifests remain the authoritative index metadata. Doris persists mutation intent and control state, not a second copy of the current external index definition. ### 1.3 Explicitly deferred from 4.2 These exclusions are normative and cannot gate 4.2. Future work requires a separate issue and review and creates no roadmap here: - automatic `UNKNOWN` watchers or reconciliation daemons, including persisted backoff, jitter, observations, or metrics; - `ACKNOWLEDGED`, multiple force variants, `FORCE ... WITHOUT REFRESH`, a general immutable resolution-event framework, or permanent `UNKNOWN` tombstones and advanced archival/replay beyond the required durable job and fence; - provider-backed table/dataset incarnation registries or detection of external replacement by another dataset at the same URI; - normalization-version migration, mixed-version fence protocols, or a general cross-release Arrow/schema canonicalization framework; - broad catalog-wide external-DDL transaction machinery beyond protecting the stable target locator and same index name; - the duplicate logical `lance_indexes()` TVF, server-side exact logical-index count, `countRows()` in normal inspection, row/fragment coverage, or rich physical provenance diagnostics; - all REST inspection and mutation work; any independently proposed read-only REST surface belongs to a separate issue and MUST NOT gate this lifecycle; - distributed index construction, segment commit, fragment-parallel vector TopN, candidate merging, or a reusable query I/O recorder; - incremental `BUILD INDEX`, coverage repair, build progress, running cancellation, automatic mutation retry, idempotency protocols, or a continuously appended-table coverage-freshness SLA; - additional index types, composite indexes, nullable indexed fields, and richer physical metadata. ## 2. User-visible contract ### 2.1 SQL surface ```sql CREATE INDEX [IF NOT EXISTS] idx ON lance_ctl.db.tbl (embedding) USING ANN PROPERTIES ( "index_type" = "IVF_PQ", "metric" = "l2", "num_partitions" = "256", "num_sub_vectors" = "16" ); CREATE INDEX idx_btree ON lance_ctl.db.tbl (event_time) USING BTREE; CREATE INDEX idx_bitmap ON lance_ctl.db.tbl (category) USING BITMAP; CREATE OR REPLACE INDEX idx ON lance_ctl.db.tbl (embedding) USING ANN PROPERTIES ( "index_type" = "IVF_PQ", "metric" = "cosine", "num_partitions" = "256", "num_sub_vectors" = "16" ); SHOW INDEX FROM lance_ctl.db.tbl; SELECT * FROM lance_index_entries("table" = "lance_ctl.db.tbl") WHERE IndexName = "idx"; DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl; ``` `USING ANN` is a neutral DDL category. It does not add or imply Doris internal ANN query syntax for Lance. Lance vector queries continue to use `vector_search()`. `CREATE OR REPLACE` maps to `replace=true` and fully rebuilds the logical name. It converges by name but is not idempotent: another invocation may produce a different UUID and dataset version. It is mutually exclusive with `IF NOT EXISTS`; `USING`, target column, and required properties are validated before job admission. Only top-level CREATE, REPLACE, and DROP mutate Lance indexes in 4.2; `ALTER TABLE ... ADD/DROP INDEX` and `BUILD INDEX` remain unsupported. Internal tables are unchanged. ### 2.2 Submission and `IF` behavior Admission reads authoritative metadata before creating a durable job: - plain CREATE fails if the normalized logical name already exists; - CREATE IF NOT EXISTS returns an immediate no-op, without creating a job, only when the existing authoritative normalized name, indexed columns, physical family, and algorithm match. A property is compared only when the same bounded logical metadata already used by `SHOW INDEX` exposes a stable value; no additional canonicalization or metadata read is a prerequisite; - CREATE IF NOT EXISTS fails on an authoritative mismatch rather than treating a different same-name definition as success; - DROP fails if the authoritative name is absent; - DROP IF EXISTS returns an immediate no-op, without creating a job, when the authoritative name is absent; - REPLACE requires an existing or creatable unambiguous target name and always represents a new one-shot mutation when admitted. Authoritative preflight is not an external lock. A post-preflight race is classified from the worker result rules in Section 6; Doris never turns a generic post-invocation error into an IF no-op by inspecting message text. An admitted mutation returns its job ID after the job and fence are durable, not after native work finishes. A client that loses this response must inspect jobs rather than resubmit blindly. ### 2.3 Job inspection and operator release ```sql SHOW LANCE INDEX JOBS [FROM lance_ctl.db] [WHERE TableName = "tbl" AND State = "UNKNOWN"]; SHOW LANCE INDEX JOB <job_id>; RESOLVE LANCE INDEX JOB <job_id> AS FORCE_RELEASE COMMENT '<non-empty operational risk note>'; ``` `FORCE_RELEASE` is the only accepted `AS` literal; 4.2 defines neither an extensible resolution framework nor job cancellation. Job inspection exposes job ID, target, index name, operation, mutation and refresh states, possible-live status, bounded timestamps/message, and applicable FORCE audit fields. Detail may add admitted dataset/schema-contract versions and executor identity for replay diagnosis. Exact columns and coordination fields are implementation details. Credentials, secret-bearing paths, raw responses, unbounded text, and fabricated progress are never exposed. ### 2.4 Supported type and property matrix | Type | Indexed-column contract | User properties and checks | | --- | --- | --- | | `IVF_PQ` | Exactly one non-null fixed-size-list field with non-null `FLOAT16` or `FLOAT32` elements | `index_type=IVF_PQ`; `metric` is `l2`, `cosine`, or `dot`; positive required `num_partitions` and `num_sub_vectors`; subvectors divide the vector dimension; configured static bounds apply; `num_bits=8` is fixed | | `BTREE` | Exactly one non-null scalar field from the pinned predicate-pushdown set: integral, floating, decimal, string, date, or timestamp | No user build properties; C `params_json=NULL` | | `BITMAP` | Exactly one non-null boolean, integral, string, or date field | No user build properties; C `params_json=NULL` | Hamming distance, nullable/unsupported nested fields, `FLOAT64`, `UINT8`, or `INT8` vector elements, composite indexes, arbitrary scalar JSON, and unknown properties or values fail before job creation. FE enforces static syntax, type, and bounds; the worker repeats them against the admitted snapshot and performs provider-required snapshot-local validation before FFI. ## 3. Authoritative metadata contract ### 3.1 One pinned Directory snapshot Every SHOW, physical inspection, IF check, admission, and force-release refresh uses the fields needed for index management from one selected/latest Directory `Dataset` snapshot: - `Dataset.describeIndices()` without criteria supplies logical descriptions; - the same snapshot schema resolves indexed field IDs; - `Dataset.getIndexes()` supplies physical UUIDs and dataset versions. Normal inspection and FE admission MUST NOT call `Dataset.countRows()`. `Dataset.getIndexStatistics()` is also forbidden because the pinned version may migrate metadata and write a manifest. Provider-local build validation stays in the isolated worker and creates no Doris row-count surface or release gate. FE metadata reads have bounded results and deadlines. Timeout does not imply JNI cancellation; ownership remains until the native call returns. A separate read helper process is not a 4.2 requirement. ### 3.2 Logical `SHOW INDEX` `SHOW INDEX` keeps the existing MySQL-compatible 13-column schema and returns one row per logical index/column. For Lance: - `Key_name` is the exact display name; - `Column_name` comes from the same-snapshot field-ID resolution; - `Index_type` is the logical Lance algorithm such as `IVF_PQ`, `BTREE`, or `BITMAP`, not the SQL category `ANN`; - `Properties` is deterministic, valid, allowlisted JSON under a reviewed finite UTF-8 bound. There is no second logical TVF or server-side exact-count API; callers may count distinct `Key_name` values in the bounded SHOW result. ### 3.3 Minimal physical inspection `lance_index_entries("table"="ctl.db.tbl")` returns one row per physical entry: `CatalogName, DatabaseName, TableName, IndexName, IndexUuid, DatasetVersion`. The TVF exists only because UUID and dataset version do not fit the established SHOW schema; logical name, columns, type, and properties remain in SHOW. It exposes no fragments, coverage, provider/consistency state, opaque details, or transaction properties. The TVF requires one table argument, table `SHOW`, and an optional ordinary predicate over the bounded result. It is all-or-error and never silently truncated. ### 3.4 Pinned type vocabulary, system entries, and bounds The pinned Java SDK has different logical and physical vocabularies: - `describeIndices()` derives a concrete vector algorithm such as `IVF_PQ`, while `getIndexes()` may expose the umbrella manifest type `VECTOR`; - scalar values differ in case and separators, for example `BTREE` versus `BTree` and `LABEL_LIST` versus `LabelList`. Authoritative IF and internal joins use one normalization rule: - normalize case and ignore underscores for family comparison; - accept physical `VECTOR` only for supported vector algorithms; - accept physical `SCALAR` only for supported scalar algorithms; - never match an internal Doris index family to a Lance family. Logical SHOW displays the concrete `describeIndices()` value. The minimal physical TVF does not expose the umbrella type. The boundary filters system entries `__lance_frag_reuse` and `__lance_mem_wal`. Duplicate logical names or UUID ownership, unknown field IDs, malformed or oversized data, and required logical/physical mismatches fail closed with a typed bounded error. Directory inspection caps logical names, physical entries, external strings, fields per index, and aggregate field-name bytes. Exact values follow Doris configuration conventions. Because these post-JNI checks do not bound native allocation, focused FE stress/failure tests cover accepted and over-limit data. ## 4. Target identity, admission, and Doris-side guards ### 4.1 Stable target and same-name key The durable target/fence key is: ```text ( persisted catalog identity, provider = DIRECTORY, normalized stable dataset locator, persisted normalized logical-index-name bytes ) ``` Both display name and normalized bytes are persisted. Normalization v1 is the UTF-8 result of Java `toLowerCase(Locale.ROOT)`. Doris preserves display case, rejects new case-only duplicates, resolves a unique match to its stored name, and fails mutation on ambiguous external case-only collisions. 4.2 defines no normalization migration, mixed-version fence protocol, or provider-backed dataset-incarnation registry. Credential-bearing URLs are never identity; URI aliases are external writers. Replacing the dataset at the same URI while a job is active or `UNKNOWN` is unsupported. ### 4.2 Dataset-version and schema-contract revalidation Admission records the exact selected/latest dataset version and the following ordered contract: ```text schema_contract_version = 1 indexed_field = { field_id, normalized_name, normalized_type, nullable, fixed_size_list_dimension, // when applicable vector_element_type, // when applicable vector_element_nullable // when applicable } ``` `normalized_type` includes relevant parameters such as decimal precision/scale and timestamp unit/time-zone semantics. Unindexed fields are excluded. Before native invocation, the worker reopens the admitted dataset version, independently recomputes contract v1, and compares the ordered representation. An unavailable version, mismatch, or unsupported contract produces a complete pre-invocation `NOT_COMMITTED` result with typed `STALE_ADMISSION` or `UNSUPPORTED_SCHEMA_CONTRACT`; the worker never switches to latest. Java admission and Rust execution share golden contract fixtures. General Arrow canonicalization and contract migration are deferred. ### 4.3 Credentials and credential rotation Jobs never persist credentials. Dispatch, authoritative reads, refresh, and `FORCE_RELEASE` resolve current catalog credentials when executed. A credential-only catalog ALTER is not target-changing and MUST remain allowed for `PENDING`, `RUNNING`, or `UNKNOWN`; provider, stable-locator, and selected-version changes remain guarded. Operators can therefore rotate an expired token before inspection, refresh, or release. Known-expired credentials fail before invocation as `NOT_COMMITTED`. No new credential-TTL or safety-margin protocol is required. Section 8 defines secret handling. ### 4.4 Minimal DDL guard boundary Admission and Doris DDL share a target-aware guard only when changing or removing the persisted catalog identity, Directory provider, normalized locator, or selected-version semantics. Same-name admission uses Section 4.1's fence key. Credential-only ALTER, cache refresh, unrelated properties, and append are not target-changing. Indexed-field revalidation provides schema safety; 4.2 adds no general full-table or unrelated-object DDL transaction machinery. Target-aware routing precedes internal `catalog.Index` translation. Internal validation/serialization is unchanged, and Lance algorithms do not extend the persisted internal `IndexType` enum. ## 5. Directory execution and resource boundary ### 5.1 One-shot execution flow ```text SQL validation and authoritative IF check -> unresolved-job quota admission -> durable job + same-name fence -> select one BE and reserve a possible-live slot -> durable RUNNING(invocation identity, BE process epoch, deadline) -> dedicated BE supervisor -> resource-isolated worker process -> one pinned lance-c invocation -> complete typed result or UNKNOWN -> durable terminal state -> authoritative metadata refresh when required ``` FE metadata/DDL locks cover only bounded admission, guard checks, and durable transitions—not dispatch, native execution, metadata reads, or refresh. ### 5.2 Hard process isolation The pinned Rust runtime creates threads outside Doris `MemTracker` control and uses `panic=abort`; production mutation therefore never runs inside FE, BE, or an ordinary RPC handler. Before Lance/Rust initializes materially, the selected BE launches a same-build worker in a verified OS-enforced boundary providing: - a hard memory limit; - a hard PID/thread limit; - a wall-clock runtime limit; - reliable termination detection and child reaping. A deployment unable to verify the boundary rejects before invocation; there is no in-process or soft-RSS fallback. Launcher, containment, environment, diagnostic, and handshake details are not public contracts. The BE supervisor and queue are bounded. Pre-invocation busy rejection leaves the job `PENDING` for another admissible BE; durable `RUNNING` forbids any automatic redispatch. ### 5.3 Durable dispatch boundary and result envelope Before network I/O, FE records durable `RUNNING` with immutable invocation ID, selected BE process epoch, and deadline, then rechecks leadership, job revision, and invocation ID immediately before send. The worker/supervisor result envelope contains only what is needed to classify the one invocation: - the matching invocation identity and BE process epoch; - complete trusted pre-invocation-failure and native-return markers; - the saved `LanceErrorCode` read before the consuming error message; - a bounded sanitized message; - matching child-reap proof when available. Only a complete identity-matched result proves `COMMITTED` or `NOT_COMMITTED`; a missing marker never proves FFI was not entered. EOF, signal, timeout, OOM, BE loss, malformed/partial protocol, or identity mismatch after acceptance yields `UNKNOWN`. The adapter validates deterministic user errors before FFI and classifies only saved typed codes, never message text. It reads the code before the consuming message, checks it before treating a zero count as empty, and releases returned strings with `lance_free_string`. ### 5.4 Three independent controls | Control | Purpose | Release condition | | --- | --- | --- | | Same-name fence | Prevent a late old mutation from overwriting, removing, or reintroducing the name after newer work | Known terminal result after required refresh; `UNKNOWN` only after durable `FORCE_RELEASE` | | Unresolved-job quota | Bound durable job records, fences, and operator work | Known terminal state after required refresh, or durable `FORCE_RELEASE` for `UNKNOWN` | | Possible-live worker slot | Bound child processes that may still be running | Matching child reap, proof that the recorded BE process epoch no longer exists, or durable audited `FORCE_RELEASE` | Deadlines bound wait/runtime but do not prove termination or release a possible-live slot. Termination proof releases only that slot; it neither changes `UNKNOWN` nor releases the same-name fence. Admission enforces positive finite active-plus-`UNKNOWN` limits per persisted table/locator identity, per catalog, and globally. Checks precede durable job creation; a bound returns typed overload without a job or fence. Running concurrency does not replace unresolved-job quotas. ## 6. Compact durable lifecycle ### 6.1 Mutation states ```text PENDING -> RUNNING -> COMMITTED -> NOT_COMMITTED -> UNKNOWN ``` | State | Durable meaning | | --- | --- | | `PENDING` | The request and fence are durable; no execute send may have occurred | | `RUNNING` | The durable dispatch boundary has been crossed; the one-shot call may execute or may already have executed | | `COMMITTED` | A complete identity-matched typed success proves this job committed | | `NOT_COMMITTED` | A complete trusted result proves this job did not commit | | `UNKNOWN` | Doris cannot safely prove whether this job committed | All three outcomes are terminal. `UNKNOWN` has no automatic transition, and metadata never changes it to a known outcome. Pre-admission IF no-ops create no job. A typed post-dispatch DROP-not-found for `DROP IF EXISTS` may be `NOT_COMMITTED` with `CompletionReason=IF_CONDITION_NOOP`; `NOOP` is not a state. ### 6.2 Independent refresh state ```text NOT_REQUIRED | REQUIRED | RUNNING | DONE | FAILED ``` Mutation state and refresh state are stored independently: - native success sets `COMMITTED` and `REQUIRED` before refresh; - a proven no-commit result that also proves or observes external metadata advancement sets `NOT_COMMITTED` and `REQUIRED`; - a proven pre-invocation failure with no relevant metadata change may use `NOT_REQUIRED`; - refresh success/failure sets only `DONE`/`FAILED`, respectively, and never changes mutation outcome; - replay retries required refresh through the existing idempotent external-table refresh path and never replays the mutation. A known terminal job retains its fence until required refresh is `DONE`; failed refresh may retry through the existing idempotent path. Live SHOW may inform an operator but cannot attribute external state to an `UNKNOWN` job. Section 7 defines FORCE's final authoritative refresh. ### 6.3 Provider-result classification | Complete saved result | CREATE | REPLACE | DROP | Refresh obligation | | --- | --- | --- | --- | --- | | rejection before FFI, including version/schema/credential/resource revalidation | `NOT_COMMITTED` | `NOT_COMMITTED` | `NOT_COMMITTED` | required only if the authoritative revalidation observed relevant advancement/change | | `LANCE_OK` from the one native invocation | `COMMITTED` | `COMMITTED` | `COMMITTED` | required | | typed `LANCE_ERR_COMMIT_CONFLICT` | `NOT_COMMITTED` | `NOT_COMMITTED` | `NOT_COMMITTED` | required because the external dataset advanced | | typed `LANCE_ERR_NOT_FOUND` after invocation | `UNKNOWN` | `UNKNOWN` | `NOT_COMMITTED`, or IF no-op reason for DROP IF EXISTS | required for DROP; no attribution for CREATE/REPLACE | | typed `INVALID_ARGUMENT`, `NOT_SUPPORTED`, `INDEX`, `IO`, or `INTERNAL` after invocation | `UNKNOWN` | `UNKNOWN` | `UNKNOWN` | no outcome inference; operator may inspect current state | | no matching complete response, signal, OOM, panic, timeout, BE loss, or protocol ambiguity after send may have occurred | `UNKNOWN` | `UNKNOWN` | `UNKNOWN` | no outcome inference | Duplicate CREATE may return coarse `INDEX` or `INVALID_ARGUMENT`, not a stable already-exists/no-commit result. CREATE IF NOT EXISTS therefore resolves before dispatch; a later matching index is only corroboration and leaves the job `UNKNOWN`. ### 6.4 Fence rules - `PENDING` and `RUNNING` hold the same-name fence. - `COMMITTED` and `NOT_COMMITTED` release it only after required refresh is `DONE`, or immediately when refresh is `NOT_REQUIRED`. - An unresolved `UNKNOWN` retains it across FE failover, BE restart, timeout, termination proof, and metadata observations. - The unresolved job and fence remain durable and quota-owned until the durable `FORCE_RELEASE` transition; 4.2 adds no reduced or permanent tombstone. ## 7. Manual release, replay, and crash safety ### 7.1 Single `FORCE_RELEASE` protocol `FORCE_RELEASE` is allowed only for `UNKNOWN`: 1. Load the job without disclosing fields and require target-table `ALTER`, or global `ADMIN` if the target no longer resolves. 2. Require a non-empty sanitized UTF-8 note under a reviewed finite bound. 3. Keep the fence and possible-live ownership while performing one authoritative latest metadata read and external-table refresh with current credentials. 4. On refresh failure, return a typed incomplete-resolution error and retain the fence, quota ownership, possible-live slot, and `UNKNOWN` state. 5. On success, append one revision-checked record containing `ForceReleased=true`, actor, timestamp, note, and late-commit warning while atomically releasing the fence, unresolved quota, and possible-live slot. 6. Keep the mutation outcome `UNKNOWN`; FORCE never fabricates attribution. If FE crashes after refresh but before release, replay sees the fence held and may safely repeat the idempotent refresh. Concurrent FORCE requests may duplicate refresh, but only one expected-revision transition wins; later retries return the existing release record. Completed FORCE emits a SQL warning and audit entry that the old worker may still overwrite, remove, or reintroduce the name. It is the only authorized way to reuse a possible-live slot without termination proof. After FORCE, the job and audit data follow ordinary bounded retention; Section 1.3 excludes alternative resolution modes, permanent tombstones, and advanced archival. ### 7.2 Minimal durable job record The durable record contains only information required by the 4.2 invariants: - job identity, creator, revision, and bounded timestamps; - persisted target identity, normalized same-name fence key, and mutation intent; - admitted dataset version and schema-contract version/representation; - mutation outcome, independent refresh state, typed result, and bounded errors; - selected BE process epoch, immutable invocation identity, deadline, possible-live ownership, and any matching termination proof; - FORCE actor, time, bounded note, and late-commit warning when released. Edit-log/image layout, coordination counters, and public columns may vary without weakening replay, no-redispatch, stale-callback rejection, or audit. The record excludes secrets, raw/opaque provider data, unbounded values, and the deferred watcher or resolution-framework state; Section 8 covers all surfaces. ### 7.3 Replay rules | Replayed record | Required action | | --- | --- | | `PENDING` | Reconstruct job/fence/quota and allow the normal dispatcher to select one admissible BE | | `RUNNING` without a complete matching terminal result | Transition to `UNKNOWN`, reconstruct the fence/quota/possible-live ownership, and never redispatch | | `COMMITTED` or `NOT_COMMITTED` with refresh `REQUIRED/RUNNING/FAILED` | Resume only the idempotent refresh; never call lance-c again | | `UNKNOWN`, not force released | Reconstruct the same-name fence and unresolved quota; preserve possible-live ownership until proof or FORCE; schedule no watcher | | retained `UNKNOWN`, force released | Reconstruct the retained audit information with no fence, unresolved quota ownership, or possible-live slot; ordinary retention still applies | Callbacks validate leadership, job revision, invocation ID, and applicable BE epoch; stale callbacks cannot change state or reacquire a fence after FORCE. ### 7.4 Required crash-safety outcomes Required evidence covers these boundaries: - pre-dispatch loss replays `PENDING` with at most one eventual dispatch; - ambiguous post-dispatch loss, including a lost success, becomes fenced `UNKNOWN` and is never redispatched; - OOM, panic, signal, timeout, or BE loss neither kills FE/BE, fabricates an outcome, nor frees a possible-live slot on deadline; - failover independently replays saved outcome and required refresh while rejecting stale callbacks; - credential rotation during `UNKNOWN` enables later inspection, refresh, and FORCE with current credentials; - crashes on either side of FORCE preserve the held/released fence and slot; a late commit remains accepted operator risk, never re-attributed. ## 8. Authorization, confidentiality, and cleanup - CREATE, REPLACE, and DROP require table `ALTER`. - SHOW INDEX, physical inspection, and job list/detail require table `SHOW`. - FORCE requires table `ALTER` while the persisted target resolves, otherwise global `ADMIN`. - Background refresh runs as a system action only after the initiating mutation or resolution passed authorization. - A direct job lookup first loads the record without returning any field, then checks privilege against the persisted target. Unauthorized or missing jobs have the same non-disclosing response. - Orphaned jobs are visible only to `ADMIN`; other listings omit them and leak no target name, locator, message, executor, count, or force note. - Secrets are excluded from durable state, child argv/environment, retry buffers, logs, core, SQL results, error messages, and test artifacts. Failed native work may leave unreferenced index files. Doris never guesses and deletes physical paths. Lance cleanup/VACUUM and historical-version retention own physical reclamation. ## 9. Delivery and release evidence ### 9.1 Reviewable delivery slices 1. Directory authoritative `SHOW INDEX`, preserving internal behavior. 2. Minimal Directory physical UUID/dataset-version inspection. 3. Disabled FE mutation control: neutral target-aware routing, durable jobs, same-name fences, unresolved quotas, compact replay, job SQL, and fake-worker fault tests. 4. Disabled isolated one-shot worker with an IVF_PQ tracer bullet. 5. BTREE/BITMAP lifecycle, manual FORCE, focused end-to-end query consumption, and user documentation. These slices describe reviewable implementation and enablement order only; they prescribe neither PR count/mapping nor ownership/merge mechanics. Mutation remains disabled until all applicable evidence below passes. ### 9.2 G1 — SQL and metadata - parser, to-SQL, target-aware routing, unchanged internal-index behavior, and every CREATE, full REPLACE, DROP, and IF case; - supported and rejected type/property rules at admission and exact-snapshot revalidation before FFI; - one-snapshot logical SHOW and physical UUID/version inspection, including normalization, system-entry filtering, finite bounds, and fail-closed malformed or inconsistent metadata; - case-only collisions, table SHOW/ALTER, and orphan-job non-disclosure. ### 9.3 G2 — One-shot lifecycle correctness - the dispatch, result-classification, state, refresh, and fence contracts in Sections 5–7, including durability before send and no second dispatch; - every ambiguous post-dispatch result becomes `UNKNOWN`, metadata never fabricates attribution, and same-name admission remains fenced until FORCE; - independent refresh replay and FORCE refresh, revision, audit, and late-commit warning behavior. ### 9.4 G3 — Resource and quota safety - unavailable isolation rejects before invocation; FE and BE survive worker OOM, `panic=abort`, malformed protocol, and signals; - unresolved quotas remain bounded, and possible-live concurrency stays bounded per BE, per catalog, and cluster-wide under admission, deadline, reap, BE-epoch loss, and FORCE as defined in Section 5.4; - bounded metadata reads, credential rotation, and secret non-disclosure pass focused success, rejection, and native-failure fixtures. ### 9.5 G4 — Replay and failover - the representative crash-safety outcomes in Section 7.4, including FE failover, BE loss, OOM/panic, refresh replay, FORCE response loss, stale callbacks, and late commit after FORCE. ### 9.6 G5 — End-to-end usability and documentation - for each enabled index type, one focused fixed-snapshot end-to-end test creates the index through the isolated worker and proves that the existing Doris query path consumes it; - no new query architecture, exhaustive metric/predicate matrix, coverage/freshness guarantee, or reusable query-recorder infrastructure is a 4.2 release gate; - user documentation covers SQL, asynchronous jobs, UNKNOWN, FORCE warnings, refresh, privileges, quotas, worker settings, credential rotation, deployment prerequisites, and Section 10's accepted limitations. ### 9.7 Required safety configuration invariants The 4.2 contract does not fix setting names or expose a new configuration API. Mutation defaults disabled and cannot be enabled until the implementation has a verified hard worker memory/PID/runtime boundary, finite possible-live limits, positive per-table/catalog/global unresolved quotas, and bounded metadata/input processing. Exact settings and reviewed defaults follow Doris conventions and are implementation details. ## 10. Accepted operational limitations This section summarizes consequences of Section 1.3 and adds no exclusions: - a dispatched mutation may remain `UNKNOWN`; there is no automatic reconciliation or metadata-based attribution, and its name remains fenced until FORCE accepts the possible late commit without rewriting the outcome; - CREATE and REPLACE build one admitted snapshot on one worker, with no native progress, running cancellation, automatic retry, incremental repair, or continuously appended-table freshness SLA; - the branch-4.1 query path remains one whole-dataset split and one scanner per BE for global TopK, without distributed vector search or candidate merge; - external dataset replacement at the same URI, external writers, and URI aliases remain outside Doris serialization; - REST inspection and mutation remain outside this contract; - initial type, metric, nullability, schema, and property limits remain strict. ## 11. Review request To help close the Doris 4.2 scope, could reviewers please confirm whether this document accurately reflects the final Doris 4.2 implementation boundary? If it does, a simple **Yes** would be helpful. If not, please identify any specific conflict with **Required in 4.2** or **Explicitly Deferred**. Please review this comment as the complete proposal. No implementation requirement or follow-up design should be inferred from v5 or from the review history. ## Appendix A. Controlling review boundary and primary evidence ### Controlling review boundary - [review accepting the three round-5 correctness fixes and requesting four contained changes](https://github.com/apache/doris/issues/66497#issuecomment-5289747725) - [scope correction to the original index lifecycle](https://github.com/apache/doris/issues/66497#issuecomment-5289814826) - [final Doris 4.2 implementation boundary](https://github.com/apache/doris/issues/66497#issuecomment-5290050744) These links document how the final boundary was reached. They are not additional contracts; the final clarification governs on conflict. ### Doris baseline - [Parser index surface](https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4#L235-L240) - [`CreateIndexOp` validation and internal `Index` translation](https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateIndexOp.java#L73-L89) - [`ShowIndexCommand` baseline](https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowIndexCommand.java#L51-L132) - [External-table refresh/edit log](https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java#L123-L209) - [FE `lance-core` pin](https://github.com/apache/doris/blob/1d147d8ec65576d3edf4c9ca0b6a36078193e4d2/fe/pom.xml#L337) - [BE `lance-c` pin](https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/thirdparty/vars.sh#L577-L581) ### Pinned Lance evidence - [`lance-c` lifecycle ABI](https://github.com/lance-format/lance-c/blob/v0.1.2/include/lance/lance.h#L494-L538) - [`lance-c` error model](https://github.com/lance-format/lance-c/blob/v0.1.2/src/error.rs#L4-L143) - [Rust dependencies and `panic=abort`](https://github.com/lance-format/lance-c/blob/v0.1.2/Cargo.toml#L20-L63) - [Java Dataset index APIs](https://github.com/lance-format/lance/blob/e934cc2ceda2bd5f5aa37a953cc29f71d24bd5c0/java/src/main/java/org/lance/Dataset.java#L1366-L1438) - [Java logical `IndexDescription`](https://github.com/lance-format/lance/blob/e934cc2ceda2bd5f5aa37a953cc29f71d24bd5c0/java/src/main/java/org/lance/index/IndexDescription.java#L25-L102) - [Java physical `Index`](https://github.com/lance-format/lance/blob/e934cc2ceda2bd5f5aa37a953cc29f71d24bd5c0/java/src/main/java/org/lance/index/Index.java#L29-L131) - [`getIndexStatistics()` side effect](https://github.com/lance-format/lance/blob/e934cc2ceda2bd5f5aa37a953cc29f71d24bd5c0/rust/lance/src/index.rs#L1791-L1810) ## Appendix B. Non-normative implementation notes Likely implementation seams are a neutral parsed index specification, a target-aware external-provider branch before internal `catalog.Index` serialization, a master-owned minimal Lance job/fence manager, a dedicated BE supervisor, an isolated worker launcher, and the existing bounded FE authoritative metadata reader. Class names may change without changing this contract. The implementation should reuse Doris edit-log/image, external-table refresh, master forwarding, privilege, audit, result-set, and configuration conventions. It must not claim that generic `JobManager` or internal `IndexChangeJob` already provides the external one-shot CAS, no-redispatch rule, same-name fence, or possible-live ownership required here. Implementations may choose details such as cgroup v2 layout, inherited-FD and environment allowlists, protocol/ABI handshakes, edit-log field decomposition, job result columns, setting names/defaults, and additional fault-test permutations. Those choices are not 4.2 capabilities or release-contract APIs; they may vary as long as the normative safety invariants above still hold. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
