u70b3 opened a new pull request, #67754:
URL: https://github.com/apache/doris/pull/67754
> Target: branch-4.1. Stacks on #67201, #67235 and #67630 (slice 3C).
Implements slice 3E of design v5.1 (issue #66497): `RESOLVE LANCE INDEX JOB
<id> AS FORCE_RELEASE COMMENT '<note>'` — the operator escape hatch that
durably releases an UNKNOWN job — plus the retention GC that removes resolved
job records. There is no dispatcher yet (next slice): no real UNKNOWN job can
be produced by this stack, so FORCE has unit-test coverage only and the
regression suite is deliberately negative/static (see Tests).
## What lands
1. **RESOLVE SQL** (design 2.3/7.1): grammar (`RESOLVE` and `FORCE_RELEASE`
are new non-reserved keywords, COMMENT is mandatory), `ForwardWithSync` to
master. Flow: load job (missing and unauthorized share the same fixed 5103,
non-disclosing) → authorize against the persisted target (table ALTER when it
resolves, global ADMIN otherwise) → idempotent short-circuit (an already
released job returns OK + the late-commit warning, even with the pre-release
revision) → UNKNOWN-only state gate (5104) → note validation (non-empty, ≤1024
UTF-8 bytes, 5100) → for a live catalog: one authoritative latest-metadata read
and one external-table refresh with current credentials, both outside every
lock, then the durable release inside the admission critical section
(`captureLanceIndexTarget` → read/refresh → `withLanceIndexAdmission` recheck),
serialized against DROP CATALOG / identity ALTER exactly like admission. The
durable release is one ordinary job upsert with the five FORCE
audit fields set; the fence, the unresolved quota charge and the possible-live
slot flip off through the existing field semantics in `applyToMemory` — no new
teardown code.
2. **Failure semantics** (7.1 step 4): any read/refresh failure is the typed
5105 (`ERR_LANCE_INDEX_JOB_RESOLUTION_INCOMPLETE`); nothing is logged, the job
stays UNKNOWN holding fence/quota/slot, and the operator fixes the cause
(including the 4.3 credential rotation case) and retries the same statement.
Failure details pass through the catalog's sanitized root-cause chain only — no
locator, credential or dataset URI is echoed.
3. **Retention GC**: resolved records (force-released UNKNOWN and
refresh-DONE terminal alike) are audit-only; a master-only
`LanceIndexJobCleaner` daemon (interval
`lance_index_job_clean_interval_second`, default 1h) removes records resolved
longer than `lance_index_job_keep_max_second` (default 7d, aligned with
`history_job_keep_max_second`) through one batch edit-log record
(`OP_LANCE_INDEX_JOB_REMOVE = 501`) per round, so every FE serves the same SHOW
LANCE INDEX JOBS view. Unresolved records are never removed regardless of age
(fail-closed, including corrupt identity-less records); the retention clock is
the durable update time, which the force release bumps onto the force time.
4. **Config**: the two retention items are mutable, masterOnly,
positive-validated (same handler as the existing quotas).
## Two design readings that need reviewer confirmation
1. **RESOLVE is not behind `enable_lance_index_mutation`.** The gate
controls mutation admission; FORCE is the escape hatch and must stay usable
exactly when the gate is off — otherwise PENDING/UNKNOWN jobs could never be
released once the gate is closed, and the catalog DDL guard would freeze that
catalog forever. Section 2.3 does not list RESOLVE among the gated statements.
2. **Full-orphan jobs skip the authoritative read and the refresh.** Section
7.1 step 1 explicitly opens the global-ADMIN branch for a target that no longer
resolves, but steps 3-4 ("with current credentials") cannot execute when the
catalog itself is gone — there are no credentials and nothing to invalidate.
Requiring them mechanically would make an orphan job unreleasable,
contradicting step 1. So: catalog gone → release directly after ADMIN
authorization; catalog alive but db/table unresolvable (half-orphan) → skip the
read, refresh with `ignoreIfNotExists=true` as best-effort invalidation. A
non-null exception during target resolution is never treated as an orphan
verdict — it fails with 5105 so the fence is kept when "table gone" cannot be
told apart from "network down". (Slice 3C's guard makes orphans unreachable on
the normal path; they can only come from pre-guard journals/images.)
## Operational notes
- A FORCE retry is a success, not an error: the idempotent short-circuit
deliberately runs before the expected-revision check, so a retry carrying the
pre-release revision still observes the existing release record (unlike the
stale-callback convention of the dispatch paths).
- The release does not clear `possibleLiveOwned`/`terminationProof`; the
derived slot is released by `forceReleased=true` and the raw values stay for
audit.
- Success returns an OK packet with one warning row carrying the late-commit
warning, persisted verbatim as the job's `forceWarning`: the old worker may
still overwrite, remove, or reintroduce the index name; the mutation outcome
remains UNKNOWN. (SHOW WARNINGS is a stub in the new planner; the OK packet is
the only working warning channel.)
- Audit needs no extra code: the coordinator FE's standard audit log carries
the full SQL (note included), the durable force* fields replay everywhere, and
SHOW LANCE INDEX JOB(S) already renders the force columns.
- The GC daemon is registered in `startMasterOnlyDaemonThreads`
(MasterDaemon itself does no master check); both retention configs are re-read
every round, so ADMIN SET takes effect without restart.
## Tests
- FE UT (JUnit5, no FE startup): manager FORCE transition (success with
atomic fence/quota/slot release and exactly one journal record, idempotent
retry with stale revision, four non-UNKNOWN states rejected without a journal,
revision CAS, bounded-text rejection, same-name re-admission, corrupt
identity-less unblock, refresh-queue filter); retention GC (both resolved
families removed, unresolved never removed at any age, one batch record per
round, per-round cap, replay convergence/idempotence, image shrink, 501
JournalEntity round-trip, update-time clock base); config validator (defaults,
wiring, rejection, setMutableConfig end-to-end); RESOLVE parser (full forms,
`resolve`/`force_release` stay usable as identifiers/aliases, rejection
family); RESOLVE command (5103 byte-identical for missing vs unauthorized,
ALTER vs ADMIN authorization, proxy identity, note validation, 5104 gate,
idempotent replay, full/half-orphan paths, read/refresh failures keep
everything with zero manager c
alls, REST defensive 5101, FORWARD_WITH_SYNC). Full FE suite (`mvn test -pl
fe-common,fe-core -am`, no checkstyle skip): fe-common all green; fe-core 9392
tests, the only failures being 19F+2E in 14 classes (date/timezone literals,
huggingface network unreachable, native-lib link, etc.) reproduced identically
on the base commit — environmental, unrelated to this PR. The 97 tests in the
seven touched/new Lance classes are all green.
- Regression (`external_table_p0/lance/test_lance_index_resolve.groovy`,
nonConcurrent, per-run suffix, master-config capture/restore): syntax rejection
family, missing job 5103, unauthorized 5103 under a second identity, PENDING
job rejected with 5104, retention config SHOW/SET smoke (defaults,
zero/negative rejected), GC never touches an unresolved job. Needs the external
MinIO/REST docker env — runs in pipeline, not locally.
- Not covered here (needs the dispatcher slice): FORCE happy-path e2e,
same-name reuse e2e, quota reclaim e2e, expiry-GC e2e — there is no way to
produce a real UNKNOWN job in this stack.
## Deliberately not done
- No dispatch/worker (next slice); no gate flip (9.7 evidence gates).
- No "persisted locator + storage options" bypass read for half-orphans
(optional enhancement; the late-commit warning already covers the risk).
- No count-based GC cap (the time bound plus the global unresolved quota
already bounds the table).
- RESOLVE takes one job id per statement (the 2.3 form); no WHERE/batch form.
- SHOW output unchanged (the force columns already render since slice 3C).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
