u70b3 opened a new pull request, #67978:
URL: https://github.com/apache/doris/pull/67978
# Add Lance index job dispatcher with thrift dispatch boundary
> **Stacked on #67630** (admission + job inspection SQL). This branch
currently carries that PR's commits as well; only the last 8 commits are new
here, and the diff collapses to the dispatcher changes once #67630 merges.
Reviewing the last 8 commits directly is the easiest reading order.
## Problem
The durable Lance index job records created by the admission path (#67630)
currently stay `PENDING` forever: nothing drives them toward a terminal state
and no execution channel exists. This change adds the master-side driver and
the dispatch-boundary contracts so an admitted job can be dispatched at most
once, classified into its terminal state, and released — still entirely behind
the disabled-by-default `enable_lance_index_mutation` gate.
## What lands
- **Thrift dispatch boundary (contract only)**: `TLanceIndexJobDispatch` /
`TLanceIndexJobReport` plus `submit_lance_index_job` and
`reportLanceIndexJobResult`, four new enums with explicit field ids. No backend
consumption lands here (the isolated worker is a later release); the backend
handler stub answers a clean not-implemented error, so a cluster without
workers resolves a dispatched job as `NOT_COMMITTED` instead of `UNKNOWN`.
- **`LanceIndexJobDispatcher`** (master-only `MasterDaemon`, one round in
fixed order):
- *deadline sweep*: an expired `RUNNING` job converges to `UNKNOWN`
through `completeWithResult(NO_TRUSTED_RESULT)`; the possible-live slot, fence,
and quota all stay held (a deadline bounds the wait, never proves termination);
- *possible-live sweep*: a slot is released only when the recorded backend
process epoch no longer exists; a missing backend entry or heartbeat loss
proves nothing and keeps the slot held;
- *refresh driver*: terminal jobs with refresh `REQUIRED`/`FAILED` resume
through the existing idempotent external-table refresh path; `FAILED` retries
are throttled to one attempt per retry interval while a first `REQUIRED`
refresh is never delayed; `DONE` releases the fence and quota through the
existing apply-to-memory accounting;
- *PENDING dispatch*: durable `RUNNING` (invocation id, backend process
epoch, deadline) is journaled **before** any network I/O, then
leadership/revision/invocation-id are rechecked immediately before the single
send; per-round and per-backend in-flight caps bound the dispatch rate;
local-file datasets additionally require the separate operator assertion and a
single-FE, single-alive-backend topology.
- **Callback handler**: a typed result envelope is applied through
`completeWithResult` (the identity check and the result classification live in
the job manager from #67235); stale or malformed envelopes are logged and
dropped.
- **Config**: dispatch interval, execute deadline, per-round dispatch cap,
per-backend in-flight cap, refresh retry interval, and the local-file mutation
assertion — all mutable, master-only, positive-validated through the existing
validator.
- **Job manager**: three read-only sweep queries plus a force-release filter
on the refresh query. No new edit-log operation (the job upsert is reused), no
new write path, no lifecycle change.
## Design interpretations (please confirm)
1. **Dedicated thrift methods** instead of extending `TTaskType`/the
agent-task queue: this avoids wire-enum drift across branches and the
tablet-oriented semantics (and resend culture) of that track; FE-side
single-send plus the manager compare-and-set already provide at-most-once.
2. **PENDING retry is unbounded**: there is no dispatch-exhaustion terminal
state; a pre-send failure (no backend selectable, caps reached, local-file
assertion off) keeps the job `PENDING` and retries on the next round.
3. **Send failure classification**: a transport exception after the send may
have occurred converges the job to `UNKNOWN` immediately; only a clean error
status — provably not enqueued — resolves `NOT_COMMITTED` as a resource
rejection.
4. **The dispatcher does not read the mutation gate**: the gate guards
admission; a job that is already durable must be driven to its terminal state
regardless of the gate's current value.
5. **Local `file://` handling** enforces the topology rejection at dispatch
only; the pre/post-invocation version and identity checks belong to the worker
slice.
## Concurrency
The dispatcher holds no catalog or manager lock across any call; every
durable transition goes through the job manager under its own lock, and the
catalog → job-manager lock order is unchanged. The report handler runs on the
RPC thread and performs no I/O beyond the manager's edit-log write. The daemon
early-exits on lost mastership and on the checkpoint thread, and an idle round
writes no journal record.
## Validation
- Focused FE run: **481 tests across 16 classes, 0 failures / 0 errors / 0
skipped** (89 new: dispatcher 26, refresh driver 8, report handler 16, frontend
shim 3, thrift contract 7, manager queries +5, wiring +2, config validator
+4... alongside the existing suites), with checkstyle.
- Full FE UT (clean build, fe-common + fe-core): **9490 tests**; all 21
failing methods across 14 classes are the known environmental set
(timezone/network/native library), identical class-by-class to the merge base —
including `DiskReblanceWhenSchedulerIdle`, re-verified to fail with the same
assertion on the merge base itself. Zero new failures.
- Regression suite `test_lance_index_dispatch.groovy`: negative/static only
(config smoke, dispatcher-inert assertions); it needs the external docker
fixture and runs in the pipeline, not locally.
`test_lance_index_admission.groovy` was made deterministic by pinning the
dispatch interval for its window, because admitted jobs are now actually
dispatched once the gate is open.
- Backend compiles with the not-implemented stub; no backend behavior change.
## Deferred
- Backend consumption of the boundary: supervisor, hard-isolated worker
process, and the lance-c invocation (a later release).
- The mutation gate stays closed; opening it requires the verified worker
memory/PID/runtime boundary.
- End-to-end dispatch evidence and real `UNKNOWN` production (fake-worker
fault tests in UT cover the lifecycle contracts; G2/G4 e2e evidence belongs to
the worker slice).
- Master-side IDL sync PR, if needed, lands with the worker slice.
## Release note
None — fully gated behind the disabled-by-default
`enable_lance_index_mutation`; no user-visible behavior change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
