u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5225638962
# Revised Design Proposal v4.1 — Round-4 Delta for Lance Index Lifecycle > Delta for [apache/doris#66497](https://github.com/apache/doris/issues/66497). This is not a standalone restatement of the design. The [v4 proposal](https://github.com/apache/doris/issues/66497#issuecomment-5214045343) remains authoritative except for the paragraphs explicitly replaced or supplemented below. v4 + this v4.1 delta is the current design; the first implementation PR will carry the consolidated full text. Thanks for the precise round-4 review. The four primary issues and five additional details are addressed below without reopening the accepted 4.2 framework. Section 5.2 also marks one deliberately new v4.1 safety decision—the REST terminal-evidence admission gate—rather than presenting it as a round-4 requirement. ## 1. CANCEL — accepted only before dispatch This replaces v4 §5 cancellation semantics. `CANCEL BUILD INDEX` is accepted only while the job is `PENDING`. It transitions: ```text PENDING -> CANCELLING -> CANCELLED ``` Nothing is dispatched in those states. Once `RUNNING` is durably persisted, CANCEL is rejected with a typed error explaining that the one-shot native call is not interruptible in 4.2. The job then runs to its real outcome: success enters refresh and `FINISHED`; proven no-commit failure enters `FAILED`; ambiguity enters `OUTCOME_UNKNOWN`. An accepted pre-dispatch CANCEL ends in `CANCELLED`, not `FAILED`. All state transitions use one FE job-registry writer in edit-log order, equivalently compare-and-swap on the expected durable state. The dispatcher's `PENDING -> RUNNING` transition and dispatch decision form one critical section: - if CANCEL wins first, the dispatcher observes `CANCELLING` and abandons dispatch; - if dispatch wins first, `RUNNING` is persisted before the RPC send and CANCEL is rejected; - a replayed `CANCELLING` job converges to `CANCELLED` and was never dispatched. The network send cannot be atomic with the edit log. If FE fails after persisting `RUNNING` but before or after the send, recovery follows §2 and never blindly re-dispatches the job. REST jobs use the same pre-dispatch-only rule; Namespace 0.7.7 exposes no cancel operation. ## 2. Directory `RUNNING` failover — no resumable polling in 4.2 This narrows v4 §5's general failover wording. The 4.2 Directory job record deliberately carries no selected-BE identity, BE task id, or queryable dispatch generation. Neither the Doris task path nor lance-c 0.1.2 provides a durable job-status API for the one-shot call. Therefore, loss of FE/BE task communication while a Directory job is `RUNNING`, including FE failover, BE process loss, and lost RPC response, transitions the job to `OUTCOME_UNKNOWN`. It cannot resume polling and is never automatically re-dispatched. `RUNNING` is persisted before the RPC is sent. Consequently, replayed states have these meanings: - `PENDING`: never dispatched; normal scheduling is safe; - `CANCELLING`: never dispatched; complete to `CANCELLED`; - Directory `RUNNING`: dispatch may or may not have begun; move to `OUTCOME_UNKNOWN`; - `REFRESHING` / `COMMITTED_REFRESH_PENDING`: retry refresh only, never mutation. For REST, polling may resume only when the returned `transaction_id` was durably persisted. If FE failed before persisting it, a matching list/stats postcondition remains corroboration only and the job becomes `OUTCOME_UNKNOWN`. The admission slot of an `OUTCOME_UNKNOWN` job is held conservatively until operator acknowledgement/resolution or expiry. The unknown job no longer holds the same-name serialization fence, but the held catalog/cluster admission slot may still block re-issue when the configured limit is exhausted; the design does not claim otherwise. `OUTCOME_UNKNOWN` is never retried automatically. Operator resolution records a non-empty resolution note, refreshes authoritative metadata, and releases the slot. Unacknowledged unknown jobs expire after the configurable terminal-job retention period, default seven days; expiry also releases the slot. Expiry is not proof that an old executor stopped and may therefore cause temporary over-admission—an explicit 4.2 limitation of not adding a durable/queryable BE task protocol. The same lifecycle applies to DROP jobs. ## 3. Typed error/outcome classification This replaces v4 §6's rule that any returned BE/REST error is a confirmed pre-commit failure. - **Proven not dispatched or not committed -> `FAILED`**: FE analysis/provider validation failure; BE proof that the native call was not invoked; typed unsupported/invalid-argument/not-found/already-exists result with a pinned no-commit guarantee; typed commit conflict. No refresh. - **Accepted pre-dispatch cancellation -> `CANCELLED`**, per §1. - **Proven committed -> refresh and finish**: Directory one-shot call returned success; an uninterrupted REST job received provider-guaranteed terminal-success evidence and the operation postcondition matches; or a replayed REST job has a persisted transaction identity that proves success and the postcondition matches. Refresh failure enters `COMMITTED_REFRESH_PENDING` and retries refresh only. - **Ambiguous -> `OUTCOME_UNKNOWN` plus refresh-on-observation**: RPC timeout/disconnect, lost response, object-store I/O around the commit point, BE process/contact loss, REST timeout/5xx/service-internal failure, lost transaction identity, or any aggregate native error class without a no-commit guarantee. No error class becomes `FAILED` unless its provider contract guarantees no commit. In particular, lance-c's aggregate `LANCE_ERR_INDEX`, `LANCE_ERR_IO`, and `LANCE_ERR_INTERNAL` are not classified by parsing their messages. When in doubt, the job becomes `OUTCOME_UNKNOWN`. 4.2 does not automatically retry a one-shot mutation after typed commit conflict. The job becomes `FAILED` with a retryable diagnostic; an operator re-issue creates a new durable job. A reconciliation postcondition alone never proves attribution after a lost response. For Directory, a matching name/column/type at a newer version is corroboration only because the one-shot commit carries no Doris operation id. For REST, the same rule applies unless the persisted `transaction_id` supplies job-specific evidence. DROP name absence follows the same rule. Any reconciliation that observes newer external state invalidates/refreshes Doris metadata even when the job remains `OUTCOME_UNKNOWN`. ## 4. IVF_PQ required properties This corrects the property wording inherited from v3 and referenced by v4 §2. For Directory IVF_PQ creation: - `num_partitions` is a required positive integer; - `num_sub_vectors` is a required positive integer and must divide the vector dimension; - `num_bits` remains fixed to 8. The pinned lance-c 0.1.2 implementation calls `require_field` for both required values and rejects zero. Doris 4.2 does not introduce implicit data-dependent defaults. REST continues to reject these IVF build properties because Namespace 0.7.7 cannot express them; a REST provider uses only the restricted request contract already defined by v4 §4. ## 5. Additional details pinned for implementation ### 5.1 `SHOW INDEX.Properties` `Properties` is never populated from a historical creating-job record. Lance metadata or the Namespace service remains authoritative, because an external writer may have replaced the current UUID/version. Directory metadata is read on FE through the pinned Java SDK from the selected/current snapshot. `Properties` contains only exact fields returned by authoritative metadata, such as metric and effective PQ details where available. An unreadable field remains empty; `target_partition_size` must not be relabeled as the user-supplied `num_partitions`. REST may populate `distance_type` and row statistics where its stats model offers them; inexpressible build parameters remain empty. `Properties` stays bounded and never contains segment arrays or fragment-id lists. For `IF NOT EXISTS` / `IF EXISTS`, Directory compares name, columns, physical type, and every requested field that the current Java-SDK description exposes exactly. Unreadable fields do not participate, and the documented same-name limitation remains for those fields. REST compares only stats-observable fields and rejects inexpressible properties before dispatch. ### 5.2 REST polling Namespace 0.7.7 defines index `status` as a non-normative string. The round-4 requirement is to define the status mapping, polling interval, timeout, and CREATE/DROP completion conditions. **New v4.1 safety decision—not required by round 4:** Doris admits each REST mutation only when its provider contract supplies reliable terminal evidence known before dispatch through at least one of: 1. a provider-specific, code-defined and tested exact status allowlist with pending and terminal-success semantics, with every failure marker explicitly classified as either typed no-commit or ambiguous; 2. a guaranteed queryable `transaction_id` contract; or 3. a provider-guaranteed synchronous response whose successful return is the final mutation outcome. The capability is declared by the provider adapter/profile, not by an arbitrary user-configurable status string and not by learning after the mutation has been dispatched. This deliberately narrows v4 §4 to avoid admitting a mutation whose normal completion can never be classified reliably. The gate is evaluated per operation: - asynchronous CREATE requires an exact status contract or guaranteed queryable transaction identity; a generic Namespace 0.7.7 CREATE implementation with neither is rejected before dispatch with a typed capability error; - DROP may use a provider-guaranteed synchronous 2xx completion, an exact status contract, or a queryable transaction identity; - `SHOW INDEX` is read-only and is unaffected by this mutation gate. If a provider violates its declared contract after accepting a mutation—for example, a guaranteed transaction id is unexpectedly absent—the job becomes `OUTCOME_UNKNOWN`; Doris does not guess from the postcondition. Polling rules: - normal interval starts at five seconds; - consecutive transient polling failures back off to a maximum of sixty seconds; - overall timeout is twenty-four hours by default and configurable; - CREATE absence before first appearance is pending, never failure; - CREATE completes only on matching presence plus terminal success evidence from the admitted provider contract; - DROP completes on name absence plus terminal success evidence, or on a provider-guaranteed synchronous successful response followed by observation/refresh; - unrecognized status strings continue polling; - a recognized failure status becomes `FAILED` only when the provider guarantees no commit; - timeout becomes `OUTCOME_UNKNOWN`. After failover or a lost response without persisted transaction identity, presence/absence is corroboration only, as defined in §3. ### 5.3 Enforceable `file://` restriction Local-path mutation is disabled by default and allowed only by an explicit single-node/test deployment flag. Admission and dispatch both require the documented single-node topology with one registered FE and one registered BE sharing the same filesystem namespace. Any multi-node deployment rejects `file://` mutation with a typed error. Reads are unaffected. ### 5.4 Directory metadata-read placement Directory `SHOW INDEX`, `IF NOT EXISTS`, `IF EXISTS`, reconciliation, and operator resolution execute on FE through the pinned Java SDK—the read path already used by the branch-4.1 Lance catalog. BE is used only for the heavy one-shot mutation. No synchronous BE metadata RPC is introduced in 4.2. The acceptance suite explicitly creates/replaces/drops through BE lance-c and immediately reads the result through FE Java SDK, proving cross-SDK manifest interoperability and field mapping. Query-consumption coverage remains as specified by v4. ## 6. Test-plan delta The following cases are added to v4 §8: - CANCEL accepted in `PENDING`, rejected in `RUNNING`, and the CANCEL/dispatch CAS race; - replayed `CANCELLING -> CANCELLED` with proof that nothing was dispatched; - the crash window after durable `RUNNING` but before RPC send, yielding `OUTCOME_UNKNOWN` without re-dispatch; - Directory failover/BE loss and conservative unknown-slot accounting; - each typed outcome class, including aggregate native errors and transport ambiguity, with proof that no message parsing affects state; - required IVF_PQ property rejection for missing, zero, negative, non-integer, and non-divisor values; - per-operation REST admission for status, transaction, and synchronous-completion contracts; generic asynchronous CREATE rejection; unaffected SHOW; provider-contract violation after acceptance; - REST pre-appearance absence, exact status mapping, unknown status, transaction-id loss, CREATE/DROP completion, backoff, and timeout; - operator acknowledgement/resolution, resolution note, refresh-before-release, slot release, and unacknowledged expiry; - authoritative-only `Properties`, including external replacement proving that creating-job values never leak; - BE lance-c mutation followed by FE Java-SDK metadata reads; - single-node enabled/disabled `file://` mutation and mandatory multi-node rejection. All other v4 scope, SQL, type, authorization, query-verification, future-work, and implementation-decomposition decisions remain unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
